Document retrieval method and automatic question answering method

Abstract

Embodiments of the present description provide a document retrieval method and an automatic question answering method. The document retrieval method comprises: obtaining data for retrieval; retrieving at least one candidate document from among a plurality of documents in a knowledge base on the basis of the data for retrieval; selecting at least one reference document from among the at least one candidate document on the basis of the association relationship between the data for retrieval and the at least one candidate document; and on the basis of the at least one reference document, updating the data for retrieval, to obtain updated data for retrieval, and retrieving a target document from among the plurality of documents by means of the updated data for retrieval. A reference document is obtained by means of coarse ranking retrieval and fine ranking retrieval, and thus, the accuracy of the reference document is guaranteed; the reference document is used for updating data for retrieval, so that positive and negative feedback interactions are achieved in the retrieval pipeline, making the data for retrieval more accurate, effectively solving retrieval errors caused by expression diversity and indirectness, and improving the accuracy of document retrieval.

Description

Document retrieval methods and automatic question answering methods Technical Field

[0001] The embodiments of this specification relate to the field of computer technology, and in particular to a document retrieval method and an automatic question-answering method. Background Art

[0002] With the development of computer technology, automated document retrieval has gradually become a research focus. Document retrieval refers to the process of finding and returning relevant information in a large collection of documents based on the query conditions entered by the user.

[0003] Currently, retrieval processes are often plagued by two types of problems: representation diversity, where the query terms entered by the user differ from the representation of the same object in the knowledge base; and logical reasoning, where the user's query terms do not have a direct answer in the knowledge base and must be inferred indirectly through reasoning. These two issues lead to extremely poor document retrieval accuracy, and therefore a highly accurate document retrieval solution is urgently needed. Summary of the Invention

[0004] In view of this, embodiments of this specification provide a document retrieval method. One or more embodiments of this specification also relate to an automatic question-answering method, a document retrieval apparatus, an automatic question-answering apparatus, a computing device, a computer-readable storage medium, and a computer program to address technical deficiencies in the prior art.

[0005] According to a first aspect of the embodiments of this specification, a document retrieval method is provided, comprising:

[0006] Get the data to be retrieved;

[0007] Retrieving at least one candidate document from multiple documents in a knowledge base according to the data to be retrieved;

[0008] Filtering at least one reference document from the at least one candidate document according to an association relationship between the data to be retrieved and the at least one candidate document;

[0009] The data to be retrieved is updated according to at least one reference document to obtain updated data to be retrieved, and the updated data to be retrieved is used to retrieve a target document from a plurality of documents.

[0010] According to a second aspect of the embodiments of this specification, an automatic question-answering method is provided, comprising:

[0011] Get questions to be answered;

[0012] According to the question to be answered, at least one candidate document is retrieved from multiple documents in the knowledge base;

[0013] Filtering at least one reference document from the at least one candidate document based on an association relationship between the question to be answered and the at least one candidate document;

[0014] updating the question to be answered according to at least one reference document to obtain an updated question to be answered, and retrieving a target document from the plurality of documents using the updated question to be answered;

[0015] Generates the answer corresponding to the question to be answered based on the target document.

[0016] According to a third aspect of the embodiments of this specification, a document retrieval device is provided, comprising:

[0017] A first acquisition module is configured to acquire data to be retrieved;

[0018] A first retrieval module is configured to retrieve at least one candidate document from a plurality of documents in a knowledge base according to the data to be retrieved;

[0019] A first screening module is configured to screen out at least one reference document from at least one candidate document based on an association relationship between the data to be retrieved and the at least one candidate document;

[0020] The second retrieval module is configured to update the data to be retrieved according to at least one reference document to obtain updated data to be retrieved, and retrieve a target document from the multiple documents using the updated data to be retrieved.

[0021] According to a fourth aspect of the embodiments of this specification, an automatic question-answering device is provided, comprising:

[0022] A second acquisition module is configured to acquire questions to be answered;

[0023] A third retrieval module is configured to retrieve at least one candidate document from a plurality of documents in the knowledge base according to the question to be answered;

[0024] A second screening module is configured to screen at least one reference document from the at least one candidate document based on an association relationship between the question to be answered and the at least one candidate document;

[0025] a fourth retrieval module configured to update the question to be answered according to the at least one reference document to obtain an updated question to be answered, and retrieve a target document from the plurality of documents using the updated question to be answered;

[0026] The first generation module is configured to generate a response result corresponding to the question to be answered based on the target document.

[0027] According to a fifth aspect of the embodiments of this specification, there is provided a computing device, including:

[0028] memory and processor;

[0029] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method provided in the first aspect or the second aspect are implemented.

[0030] According to a sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions. When the instructions are executed by a processor, the steps of the method provided in the first or second aspect above are implemented.

[0031] According to a seventh aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the method provided in the first aspect or the second aspect above.

[0032] One embodiment of the present specification provides a document retrieval method, comprising: obtaining data to be retrieved; retrieving at least one candidate document from multiple documents in a knowledge base based on the data to be retrieved; screening at least one reference document from the at least one candidate document based on the association between the data to be retrieved and the at least one candidate document; updating the data to be retrieved based on the at least one reference document to obtain updated data to be retrieved, and retrieving a target document from multiple documents using the updated data to be retrieved. By roughly sorting and retrieving candidate documents from multiple documents, and then retrieving and refining the reference documents from the candidate documents, the accuracy of the reference documents is ensured. Furthermore, by updating the data to be retrieved using the reference documents, positive and negative feedback interactions in the retrieval process are implemented, making the data to be retrieved more accurate, effectively resolving retrieval errors caused by expression diversity and indirectness, and improving the accuracy of document retrieval. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] FIG1 is a schematic diagram of a traditional search link based on matching;

[0034] FIG2 is an architecture diagram of a document retrieval system provided by one embodiment of this specification;

[0035] FIG3 is an architecture diagram of another document retrieval system provided by one embodiment of this specification;

[0036] FIG4 is a flowchart of a document retrieval method provided by one embodiment of this specification;

[0037] FIG5 is a flow chart of an automatic question-answering method provided by one embodiment of this specification;

[0038] FIG6 is a flowchart of a processing process of an automatic question-answering method provided by one embodiment of this specification;

[0039] FIG7 is a schematic diagram of an automatic question-and-answer interface provided by one embodiment of this specification;

[0040] FIG8 is a schematic diagram of the structure of a document retrieval device provided by one embodiment of this specification;

[0041] FIG9 is a schematic diagram of the structure of an automatic question-answering device provided by one embodiment of this specification;

[0042] FIG10 is a structural block diagram of a computing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0043] The following description sets forth many specific details to facilitate a thorough understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the scope of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0044] The terms used in one or more embodiments of this specification are for the purpose of describing specific embodiments only and are not intended to limit one or more embodiments of this specification. The singular forms "a," "the," and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0045] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0046] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0047] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, typically containing hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. A large model can also be called a cornerstone model / foundation model. By pre-training a large model with large-scale unlabeled corpus, a pre-trained model with more than 100 million parameters is produced. This model can adapt to a wide range of downstream tasks and has good generalization capabilities, such as a large language model (LLM) and a multi-modal pre-training model.

[0048] In actual applications, large models only require a small number of samples to fine-tune the pre-trained model and can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0049] First, the terms involved in one or more embodiments of this specification are explained.

[0050] Multi-agent system: In the field of artificial intelligence, a multi-agent system is a system composed of multiple information processing and decision-making units in a shared environment, interacting within the shared environment to achieve the same or conflicting goals. The multi-agent system in the embodiments of this specification refers to a system that includes a rewriting agent, a rough sorting agent, a fine sorting agent, and a text generation agent.

[0051] Knowledge base based question answering system: Different from open domain question answering, the knowledge base based question answering system relies on the large amount of information stored in the knowledge base when answering user questions.

[0052] BERT: BERT (Bidirectional Encoder Representations from Transformers) is a pre-trained natural language processing model that uses bidirectional context information to generate word vectors and sentence representations.

[0053] Because the input length that large models can accept is currently still relatively limited, often far smaller than the content size of the knowledge base, and the computational time of large models increases rapidly with the input length, knowledge-based question-answering systems typically extract relevant information from the knowledge base through a retrieval process. Obviously, as a precursor to answer generation, the accuracy of the retrieval process determines the upper limit of question-answering effectiveness, and optimizing the retrieval process is crucial to the effectiveness of the question-answering system.

[0054] In practical applications, retrieval chains often suffer from two types of problems. The first is the problem of representational diversity, where the query terms entered by the user differ from the representation of the same object in the knowledge base, resulting in poor retrieval accuracy. For example, OSS stands for Object Storage Service. When a user asks, "How is OSS charged?" and the knowledge base only documents the pricing of the object storage service, the retrieval chain struggles to find the correct relevant documents for the user's question. The second is the problem of logical reasoning, where the user's query terms do not have a direct answer in the knowledge base and must be inferred indirectly through reasoning. Traditional retrieval chains based on text matching are unable to handle this. For example, if a user asks, "Will the design drafts selected in this competition be primarily for regions outside of Country A or for the market in Country A?" and the two relevant documents in the knowledge base are "Our company's shampoo is designed for consumers in Country A and has a very high market share" and "This competition will select the best packaging design for our company's shampoo products," the retrieval chain will be unable to answer the user's question.

[0055] Currently, document retrieval and question answering can be performed in three ways. The first approach, as shown in Figure 1, shows a schematic diagram of a traditional matching-based retrieval chain. This chain generally consists of three modules: a question rewriting module, a coarse recall module, and a fine ranking module. When a user asks a question to a question-answering system, retrieval begins. First, the rewriting module rewrites the user question and historical conversation data to obtain a rewritten user question that facilitates accurate querying. Next, the coarse recall module roughly recalls the top-K candidate documents with the highest relevance scores from the knowledge base based on the rewritten user question. Next, the fine ranking module scores and ranks these top-K candidate documents, identifies those with fine ranking scores above a threshold, and passes them to a text generation model. Finally, the text generation model answers the user question based on the documents with fine ranking scores above the threshold, generating a generated response. However, this matching-based knowledge question-answering retrieval chain only performs one retrieval per question-answering process. Because the coarse recall module and the fine ranking module each calculate the matching score between the user question and each candidate document, they cannot comprehensively analyze the content of different candidate documents to determine their relationship to the user question, making it difficult to address the issues of representation diversity and logical reasoning.

[0056] The second approach, based on a matching-based knowledge question-answering retrieval chain, uses knowledge base data to retrain the rewriting module, rough recall module, and fine sorting module. This pre-injects knowledge base knowledge into each module of the retrieval chain through offline training. This can enhance the retrieval chain's understanding of knowledge base information to a certain extent, enabling better retrieval in a single call. However, this approach requires training and deploying different retrieval chains for different knowledge bases, which is costly. Furthermore, as knowledge base data is updated, the retrieval chain must also be retrained.

[0057] The third method is to conduct interactive question-answering based on a memory tree. By constructing a memory tree, a tree is established for the content of the knowledge base, from a single document (leaf node) to the knowledge base; and the large model generates a summary for each node as the representation of the node. When the user asks a question, starting from the root node, the content of the next-level child nodes is read one by one, and it is inferred whether to enter this node or return until information related to the question is found or the maximum number of attempts is reached. Although the above solution can take advantage of the large model's ability to understand, generate, and plan, and comprehensively consider the knowledge base information during retrieval, the memory tree construction and search require calling the large model too many times, and the time complexity is too high.

[0058] In order to solve the above problems, the embodiments of this specification comprehensively consider the accuracy problem and time complexity problem of retrieval enhancement, and conduct a limited number of positive and negative feedback interactions on the retrieval link, so as to gradually optimize the data to be retrieved in combination with the document information of the knowledge base. Specifically, the data to be retrieved is obtained; based on the data to be retrieved, at least one candidate document is retrieved from multiple documents in the knowledge base; based on the association between the data to be retrieved and the at least one candidate document, at least one reference document is screened from the at least one candidate document; based on the at least one reference document, the data to be retrieved is updated to obtain the updated data to be retrieved, and the updated data to be retrieved is used to retrieve the target document from multiple documents. It effectively solves the problems of expression diversity and logical reasoning, and question-answering systems based on different knowledge bases can share a set of retrieval solutions, which greatly reduces the training and deployment costs.

[0059] In this specification, a document retrieval method is provided. This specification also involves an automatic question-answering method, a document retrieval device, an automatic question-answering device, a computing device, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0060] Referring to FIG. 2 , FIG. 2 shows an architecture diagram of a document retrieval system provided by an embodiment of this specification. The document retrieval system may include a client 100 and a server 200 ;

[0061] The client 100 is used to send the data to be retrieved to the server 200;

[0062] The server 200 is configured to retrieve at least one candidate document from multiple documents in a knowledge base based on the data to be retrieved; select at least one reference document from the at least one candidate document based on an association between the data to be retrieved and the at least one candidate document; update the data to be retrieved based on the at least one reference document to obtain updated data to be retrieved, and retrieve a target document from the multiple documents using the updated data to be retrieved; and send the target document to the client 100.

[0063] The client 100 is also used to receive the target document sent by the server 200.

[0064] By applying the solution of the embodiment of this specification, candidate documents are obtained by rough sorting and retrieving from multiple documents, and reference documents are further obtained by fine sorting and retrieving from the candidate documents, thereby ensuring the accuracy of the reference documents. In addition, the reference documents are used to update the data to be retrieved, thereby realizing positive and negative feedback interaction on the retrieval link, making the data to be retrieved more accurate, effectively solving the retrieval errors caused by expression diversity and indirectness, and improving the accuracy of document retrieval. Among them, the retrieval link of retrieving candidate documents from multiple documents based on the data to be retrieved and then screening out the target document from the candidate documents can be understood as positive feedback interaction. Since the retrieval link of updating the data to be retrieved using the reference documents screened out from the candidate documents is opposite to the direction of the retrieval link of the positive feedback interaction, it can be understood as a negative feedback interaction. Positive and negative feedback interaction refers to a feedback interaction link including a positive retrieval link and a negative retrieval link.

[0065] Referring to Figure 3, Figure 3 shows an architecture diagram of another document retrieval system provided in accordance with one embodiment of this specification. The document retrieval system may include multiple clients 100 and a server 200. The clients 100 may include end-side devices, and the server 200 may include cloud-side devices. Multiple clients 100 may establish communication connections via the server 200. In a document retrieval scenario, the server 200 is used to provide document retrieval services between the multiple clients 100. The multiple clients 100 may act as either senders or receivers, communicating via the server 200.

[0066] Users can interact with the server 200 through the client 100 to receive data sent by other clients 100, or send data to other clients 100. In the document retrieval scenario, users can publish data streams to the server 200 through the client 100. The server 200 generates a target document based on the data stream and pushes the target document to other clients with which communication has been established.

[0067] The client 100 and the server 200 are connected via a network. The network provides a medium for the communication link between the client 100 and the server 200. The network can include various connection types, such as wired or wireless communication links or fiber optic cables. The data transmitted by the client 100 may need to be encoded, transcoded, compressed, or other processing before being released to the server 200.

[0068] The client 100 can be a browser, an APP (Application), or a web application such as an H5 (HyperText Markup Language 5, version 5) application, a light application (also known as a mini-program, a lightweight application), or a cloud application. The client 100 can be developed based on the software development kit (SDK) of the corresponding service provided by the server 200, such as a real-time communication (RTC) SDK. The client 100 can be deployed in an electronic device and needs to rely on the device to run or certain APPs in the device to run. For example, the electronic device can have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, tablet computer, personal computer, etc. Various other types of applications can also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0069] The server 200 may include servers that provide various services, such as servers that provide communication services to multiple clients, servers that provide background training to support models used on clients, and servers that process data sent by clients. It should be noted that the server 200 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. The server can also be a server for a distributed system, or a server that is integrated with a blockchain. The server can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0070] It is worth noting that the document retrieval methods provided in the embodiments of this specification are generally executed by the server. However, in other embodiments of this specification, the client may also have similar functions to the server and thus execute the document retrieval methods provided in the embodiments of this specification. In other embodiments, the document retrieval methods provided in the embodiments of this specification may also be executed jointly by the client and the server.

[0071] Referring to FIG4 , FIG4 shows a flowchart of a document retrieval method provided by an embodiment of this specification, which specifically includes the following steps:

[0072] Step 402: Obtain the data to be retrieved.

[0073] Since the amount of information stored in the knowledge base is much larger than the input length of the text generation agent, it is crucial to accurately retrieve knowledge documents related to the data to be retrieved from the knowledge base. In one or more embodiments of this specification, the data to be retrieved can be obtained, and document retrieval can be performed based on the data to be retrieved to obtain the target document corresponding to the data to be retrieved.

[0074] Specifically, the data to be retrieved is used to describe document retrieval requirements and retrieve target documents from the knowledge base. The data to be retrieved can be in various formats, such as voice data, text data, video data, and so on. The data to be retrieved can also be in different languages, such as English data, Chinese data, and so on. The data to be retrieved can also be data from different scenarios, such as product retrieval data in e-commerce scenarios, information retrieval data in conference scenarios, and so on.

[0075] In actual applications, there are many ways to obtain the data to be retrieved, and the specific method to be selected depends on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the data to be retrieved sent by the user can be received. In another possible implementation of this specification, the data to be retrieved can be extracted from other data acquisition devices or databases. In yet another possible implementation of this specification, the question to be retrieved can be obtained, and the data to be retrieved can be generated based on the question to be retrieved. Among them, the question to be retrieved refers to the question that the user wants to obtain the answer or information through retrieval.

[0076] It should be noted that there are multiple ways to generate data to be retrieved based on the question to be retrieved, and the specific method to be selected depends on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the question to be retrieved can be directly used as the data to be retrieved. In another possible implementation of this specification, historical conversation data of the question to be retrieved can be obtained, and the question to be retrieved and the historical conversation data can be integrated to obtain the data to be retrieved. In another possible implementation of this specification, knowledge base description information can be obtained, and the question to be retrieved and the knowledge base description information can be integrated to obtain the data to be retrieved, or the question to be retrieved, historical conversation data and knowledge base description information can be integrated to obtain the data to be retrieved.

[0077] In an optional embodiment of the present specification, taking an automatic question-answering scenario as an example, since the question to be retrieved may not fully express the user's search requirements, the data to be retrieved can be constructed based on historical conversation data and knowledge base description information. That is, the above-mentioned acquisition of the data to be retrieved can include the following steps:

[0078] Obtain the questions to be retrieved, historical conversation data of the questions to be retrieved, and knowledge base description information;

[0079] Construct the data to be retrieved based on the questions to be retrieved, historical conversation data and knowledge base description information.

[0080] Specifically, historical conversation data refers to historical conversation records related to the question being retrieved. Knowledge base description information is used to describe the knowledge base, which stores structured or unstructured knowledge to promote knowledge sharing, collaboration, and reuse. Knowledge base description information includes, but is not limited to, the knowledge base content domain and knowledge base content type. The specific selection depends on the actual situation and is not limited in this specification.

[0081] It should be noted that the method of "obtaining the questions to be retrieved, the historical conversation data of the questions to be retrieved, and the description information of the knowledge base" can refer to the implementation method of "obtaining the data to be retrieved" mentioned above, and will not be described in detail in the embodiments of this specification. There are many ways to determine the knowledge base corresponding to the questions to be retrieved. In a first possible implementation method, the user can specify the knowledge base; in a second possible implementation method, the type of the questions to be retrieved can be identified, and the knowledge base can be determined according to the type of the questions to be retrieved. In a third possible implementation method, all knowledge bases can be used as the knowledge base corresponding to the questions to be retrieved.

[0082] By applying the solution of the embodiments of this specification, the questions to be retrieved, the historical conversation data of the questions to be retrieved, and the knowledge base description information are obtained; based on the questions to be retrieved, the historical conversation data and the knowledge base description information, the data to be retrieved is constructed, so that the data to be retrieved is more in line with the current knowledge base context, the accuracy of the data to be retrieved is improved, and the accuracy of document retrieval is further improved.

[0083] In practical applications, when constructing the data to be retrieved based on the question to be retrieved, historical conversation data and knowledge base description information, the question to be retrieved can be rewritten based on the historical conversation data and knowledge base description information to obtain the data to be retrieved.

[0084] In one possible implementation of this specification, keywords can be extracted from historical conversation data and knowledge base description information, and the keywords can be integrated into the question to be searched to obtain the data to be searched. When integrating keywords into the question to be searched, the keywords can be directly inserted into the question to be searched, such as by placing the keywords at the beginning or end of the question to be searched. Alternatively, a fusion template can be obtained and the keywords can be integrated into the question to be searched based on the fusion template, wherein the fusion template is set according to the actual situation, such as "When searching the question to be searched, search based on the following keywords."

[0085] In another possible implementation of this specification, a pre-trained language model may be used to generate the data to be retrieved. That is, the above-mentioned construction of the data to be retrieved based on the question to be retrieved, the historical conversation data, and the knowledge base description information may include the following steps:

[0086] The questions to be retrieved, historical conversation data, and knowledge base description information are input into the pre-trained language model to obtain the data to be retrieved.

[0087] Specifically, the pre-trained language model is good at generating long sequences of text. The pre-trained language model used to generate data to be retrieved can be understood as a rewriting agent. The rewriting agent can be a large model or a rewriting model trained by multiple sample texts (including sample retrieval questions, sample historical conversation data, and sample knowledge base description information) and sample retrieval labels corresponding to the sample texts.

[0088] It should be noted that after the query to be retrieved, historical conversation data, and knowledge base description information are input into the pre-trained language model, the pre-trained language model can first encode the query to be retrieved, historical conversation data, and knowledge base description information into vector form. Then, the pre-trained language model can use its self-attention mechanism (such as Transformer) to globally understand and model the vector form of the query to be retrieved, historical conversation data, and knowledge base description information. It can dynamically adjust the importance weight of each word or phrase based on the context to obtain the attention processing result, achieving a deep fusion between the query to be retrieved, historical conversation data, and knowledge base description information. Finally, the pre-trained language model can decode the attention processing result to obtain the query to be retrieved data.

[0089] For example, assuming that the question to be retrieved is "Can an external hard drive be connected?", and the historical conversation data is "How much memory does model A support?", the data to be retrieved generated by the rewritten intelligent agent is "Can model A be connected to an external hard drive?".

[0090] By applying the solution of the embodiments of this specification, the questions to be retrieved, historical conversation data and knowledge base description information are input into a pre-trained language model to obtain the data to be retrieved. The pre-trained language model can refer to the knowledge base description information and / or historical conversation data to rewrite the questions to be retrieved into an expression that is more comprehensive and more in line with the current knowledge base context, thereby improving the accuracy of the data to be retrieved and further improving the accuracy of document retrieval.

[0091] Step 404: According to the data to be retrieved, at least one candidate document is retrieved from multiple documents in the knowledge base.

[0092] In one or more embodiments of the present specification, after obtaining the data to be retrieved, at least one candidate document may be retrieved from multiple documents in the knowledge base based on the data to be retrieved.

[0093] Specifically, the candidate document refers to a document related to the data to be retrieved among multiple documents.

[0094] In actual applications, there are many ways to retrieve at least one candidate document from multiple documents in the knowledge base based on the data to be retrieved. The specific selection is based on the actual situation, and the embodiments of this specification do not impose any limitation on this.

[0095] In a possible implementation of the present specification, a search engine may be invoked to retrieve at least one candidate document related to the data to be retrieved from multiple documents in a knowledge base.

[0096] In another possible implementation of the present specification, at least one candidate document may be retrieved from multiple documents in a knowledge base by means of feature matching. That is, the above-mentioned process of retrieving at least one candidate document from multiple documents in a knowledge base based on the data to be retrieved may include the following steps:

[0097] Obtain document features corresponding to multiple documents;

[0098] Perform feature extraction on the data to be retrieved to obtain the features to be retrieved;

[0099] Matching the features to be retrieved with the document features, and determining the matching information between the features to be retrieved and the features of each document;

[0100] At least one candidate document is selected from the multiple documents based on the matching information.

[0101] Specifically, document features refer to the embedding vectors of documents in the knowledge base. Retrieval features refer to the embedding vectors of the data to be retrieved. Matching information describes the degree of match between the retrieval features and each document feature, such as a similarity value.

[0102] In practical applications, there are multiple ways to obtain document features corresponding to multiple documents, and the specific method to be selected depends on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, document features pre-calculated in an offline environment can be read from other data acquisition devices or databases. In another possible implementation of this specification, after obtaining the data to be retrieved, feature extraction can be performed on multiple documents to obtain document features corresponding to the multiple documents.

[0103] It should be noted that there are many ways to extract features from the data to be retrieved and obtain the features to be retrieved, and the specific selection should be made according to the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, the word embedding model (Word2Vec) of deep learning can be used to extract features from the image description text to obtain text features. In another possible implementation of this specification, the data to be retrieved can be input into a coarse sorting agent to obtain the features to be retrieved. Among them, the coarse sorting agent can adopt a model such as BERT with a small number of parameters that focuses on understanding the context in a given text. The coarse sorting agent is used to determine the coarse sorting matching information between the data to be retrieved and each document.

[0104] Furthermore, after determining the features to be retrieved and the text features, matching information between the features to be retrieved and the features of each document can be determined, and documents whose matching information is greater than a matching threshold, or the top K documents with matching information, can be selected as candidate documents. There are various ways to determine matching information. In one possible implementation of this specification, the cosine similarity between the features to be retrieved and the features of each document can be calculated to obtain matching information. In another possible implementation of this specification, the Euclidean distance between the features to be retrieved and the features of each document can be calculated to obtain matching information.

[0105] By applying the solution of the embodiments of this specification, document features corresponding to multiple documents are obtained; feature extraction is performed on the data to be retrieved to obtain the features to be retrieved; the features to be retrieved and the document features are matched to determine the matching information between the features to be retrieved and the features of each document; based on the matching information, at least one candidate document is screened out from multiple documents, and the candidate documents are obtained by rough sorting and retrieving from multiple documents, thereby ensuring the relevance of the candidate documents to the question to be retrieved.

[0106] Step 406: Filter out at least one reference document from the at least one candidate document based on the association relationship between the data to be retrieved and the at least one candidate document.

[0107] In one or more embodiments of the present specification, data to be retrieved is obtained; after at least one candidate document is retrieved from multiple documents in a knowledge base based on the data to be retrieved, since the number of candidate documents is usually large, the at least one candidate document can also be sorted and retrieved based on the association between the data to be retrieved and the at least one candidate document to screen out at least one reference document.

[0108] Specifically, the reference document refers to a document in the candidate document whose relationship index is greater than the first threshold and less than the second threshold. The first threshold and the second threshold are set according to actual conditions, and the embodiments of this specification do not impose any restrictions on this.

[0109] In actual applications, when screening out at least one reference document from at least one candidate document based on the association relationship between the data to be retrieved and at least one candidate document, the relationship index between the data to be retrieved and at least one candidate document can be first determined based on the association relationship between the data to be retrieved and at least one candidate document, and then at least one reference document can be screened out from at least one candidate document based on the relationship index.

[0110] It should be noted that there are many ways to determine the relationship index between the data to be retrieved and at least one candidate document. In one possible implementation of this specification, the relationship index between the data to be retrieved and at least one candidate document can be generated using Euclidean distance or cosine similarity algorithm.

[0111] In another possible implementation of the present specification, a relationship determination model may be used to generate a relationship index corresponding to each candidate document. That is, the above-mentioned screening of at least one reference document from at least one candidate document based on the association relationship between the data to be retrieved and at least one candidate document may include the following steps:

[0112] For a first candidate document, inputting the data to be retrieved and the first candidate document into a relationship determination model to obtain a relationship index corresponding to the first candidate document, wherein the relationship index is used to describe the degree of association between the first candidate document and the data to be retrieved, and the first candidate document is any one of the at least one candidate document;

[0113] The candidate documents whose relationship index is greater than a first threshold and less than a second threshold are determined as reference documents.

[0114] Specifically, the relationship determination model is used to determine the precise ranking relationship index between the data to be retrieved and each candidate document. The relationship determination model can be called a precise ranking agent, which can adopt a model such as BERT with a small number of parameters.

[0115] It should be noted that the relationship determination model calculates the relationship index between the data to be retrieved and each candidate document. The higher the relationship index, the higher the adaptability of the candidate document to the data to be retrieved. In the embodiment of this specification, a first threshold and a second threshold that is larger than the first threshold are set. Among them, the second threshold is used to determine the target document that can be transmitted to the pre-trained language model for generating a reply result; the first threshold is used to exclude irrelevant documents from the candidate documents; for reference documents whose relationship index is between the first threshold and the second threshold, it is considered that these documents are related to the data to be retrieved, but are not sufficient to provide the knowledge required to generate a reply result. Therefore, the retrieval link processing can be called again for the reference document.

[0116] Using the solution of the embodiments of this specification, for a first candidate document, the data to be retrieved and the first candidate document are input into a relationship determination model to obtain a relationship index corresponding to the first candidate document. Candidate documents whose relationship index is greater than a first threshold and less than a second threshold are determined as reference documents. This is achieved by performing a rough search from multiple documents to obtain candidate documents, and then performing a fine search from the candidate documents to obtain reference documents, thereby ensuring the accuracy of the reference documents.

[0117] In an optional embodiment of the present specification, after inputting the data to be retrieved and the first candidate document into the relationship determination model and obtaining the relationship index corresponding to the first candidate document, the following steps may also be included:

[0118] Determine the candidate documents whose relationship index is greater than or equal to a second threshold as target documents;

[0119] The data to be retrieved and the target document are input into the pre-trained language model to generate the response result.

[0120] It should be noted that since the relationship index of the target document is greater than or equal to the second threshold, it indicates that the target document is strongly correlated with the data to be retrieved. The target document and the data to be retrieved can be input into a pre-trained language model to generate a response result. The pre-trained language model used to generate the response result can be understood as a text generation agent. The text generation agent can be a large model or a generative model trained from multiple sample texts (including sample retrieval data and sample documents) and the response labels corresponding to the sample texts.

[0121] After determining the target document, the target document can be fed into a pre-trained language model. First, the pre-trained language model encodes the target document and the data to be retrieved into vectors. Then, the pre-trained language model uses its self-attention mechanism to gain a deep understanding of the target document and the data to be retrieved, measuring the relevance between the two. Finally, the pre-trained language model uses this relevance to find a response from the target document.

[0122] In practical applications, since the pre-trained language model used to generate the data to be retrieved is only used to perform rewriting tasks, while the pre-trained language model used to generate the reply results may have to handle multiple types of generation tasks, the pre-trained language model used to generate the data to be retrieved can adopt a model with a smaller parameter scale than the pre-trained language model used to generate the reply results.

[0123] Using the solution of the embodiments of this specification, candidate documents with a relationship index greater than or equal to a second threshold are identified as target documents. The data to be retrieved and the target document are input into a pre-trained language model to generate a response result. Because the target document is strongly correlated with the data to be retrieved, the accuracy of the response result is guaranteed.

[0124] In an optional embodiment of the present specification, for the first candidate document, after inputting the data to be retrieved and the first candidate document into the relationship determination model and obtaining the relationship index corresponding to the first candidate document, the following steps may also be included:

[0125] When the relationship indicators corresponding to at least one candidate document are all smaller than the first threshold, the data to be retrieved is input into the pre-trained language model to generate a reply result.

[0126] It should be noted that after using the relationship determination model to obtain the relationship indicators corresponding to each candidate document, if the relationship indicators of each candidate document are all less than the first threshold, it means that each candidate document is irrelevant and cannot be used to generate a reply result. At this time, the data to be retrieved can be directly input into the pre-trained language model. Since the pre-trained language model has been fully learned on large-scale unlabeled data, its parameters already contain a large amount of world knowledge and language rules. Therefore, after receiving the data to be retrieved, the pre-trained language model can reason based on its internal knowledge structure and its own understanding ability to generate a reply result.

[0127] It should be noted that when the pre-trained language model generates a response result, it may be impossible to generate a result. At this time, the pre-trained language model can directly output a pre-set response template, such as "This question cannot be answered at the moment."

[0128] By applying the solution of the embodiments of this specification, when the relationship indicators corresponding to at least one candidate document are all less than the first threshold, the data to be retrieved is input into the pre-trained language model to generate a reply result. Even if the target document is not retrieved, the reply result is still generated, thereby improving the user experience.

[0129] Step 408: updating the data to be retrieved according to at least one reference document to obtain updated data to be retrieved, and using the updated data to be retrieved to retrieve a target document from a plurality of documents.

[0130] In one or more embodiments of the present specification, data to be retrieved is obtained; based on the data to be retrieved, at least one candidate document is retrieved from multiple documents in a knowledge base; after at least one reference document is screened out from the at least one candidate document based on the association relationship between the data to be retrieved and the at least one candidate document, the retrieval link can be further called again to perform document retrieval, that is, the data to be retrieved is updated based on the at least one reference document to obtain updated data to be retrieved, and the updated data to be retrieved is used to retrieve a target document from multiple documents.

[0131] It should be noted that after obtaining at least one reference document, the at least one reference document can be used to update the data to be retrieved, thereby obtaining updated data to be retrieved that is more consistent with the knowledge base content. Furthermore, using the updated data to be retrieved, the document retrieval method described above is returned to execution until a preset stopping condition is reached, thereby terminating the document retrieval. The preset stopping condition includes, but is not limited to, obtaining a document with a relationship index greater than a second threshold and the number of iterations reaching a preset number. The specific selection is based on actual circumstances and is not limited in this embodiment of the present specification.

[0132] For example, assuming the data to be retrieved is "How much does OSS cost?" and the reference document is "Object Storage Service OSS is a massive, secure, low-cost, and highly reliable cloud storage service," by providing the reference document to the rewriting agent, the rewriting agent can understand that "OSS" in the knowledge base stands for "Object Storage Service," thereby generating updated data to be retrieved: "How much does Object Storage Service OSS cost?" Using the reference document to update the data to be retrieved can help the coarse-ranking agent and the fine-ranking agent more effectively capture the relevance of the data to be retrieved with the knowledge base document containing the following: "The unit price of the object storage service is RMB / GB / month, but the calculation method for pay-as-you-go is actual resource usage × hourly price, ...", thereby accurately completing the document retrieval task.

[0133] By applying the solution of the embodiments of this specification, candidate documents are obtained by rough sorting and retrieving from multiple documents, and reference documents are further obtained by fine sorting and retrieving from the candidate documents, thereby ensuring the accuracy of the reference documents. In addition, the reference documents are used to update the data to be retrieved, realizing positive and negative feedback interaction on the retrieval link, making the data to be retrieved more accurate, effectively solving the retrieval errors caused by expression diversity and indirectness, and improving the accuracy of document retrieval.

[0134] In an optional embodiment of the present specification, after updating the data to be retrieved based on at least one reference document to obtain the updated data to be retrieved, and retrieving the target document from multiple documents using the updated data to be retrieved, the following steps may also be included:

[0135] Generate response results based on the target document and the data to be retrieved.

[0136] Specifically, the answer result can be directly extracted from the target document, or it can be inferred from the target document based on the data to be retrieved. For example, assuming the data to be retrieved is "When is XXXX's birth date?", the answer result is "February 2, 2000".

[0137] It should be noted that there are multiple ways to generate a response result based on the target document and the data to be retrieved, and the specific method to be selected depends on the actual situation. The embodiments of this specification do not impose any restrictions on this. In one possible implementation of this specification, a search engine can be used to query the target document to obtain a response result corresponding to the data to be retrieved. In another possible implementation of this specification, the data to be retrieved and the target document can be input into a pre-trained language model to obtain a response result.

[0138] In practical applications, since the length of input that the pre-trained language model can accept is still relatively limited, before inputting the data to be retrieved and the target document into the pre-trained language model, the text length of the target document and the data to be retrieved after being combined can be determined. If the text length is less than or equal to the length threshold, the data to be retrieved and the target document can be directly input into the pre-trained language model; if the text length is greater than the length threshold, the target document can be screened until the text length is less than or equal to the length threshold, and the data to be retrieved and the target document can be input into the pre-trained language model.

[0139] By applying the solution of the embodiment of this specification, a reply result is generated according to the target document and the data to be retrieved. Since the target document is a document that is strongly related to the data to be retrieved, the accuracy of the reply result is guaranteed.

[0140] In one possible implementation of this specification, updating the data to be retrieved based on at least one reference document to obtain updated data to be retrieved, and retrieving the target document from multiple documents using the updated data to be retrieved may include the following steps:

[0141] Inputting the data to be retrieved and at least one reference document into the pre-trained language model to obtain updated data to be retrieved;

[0142] Retrieving at least one updated candidate document from the plurality of documents according to the updated data to be retrieved;

[0143] In a case where the at least one updated candidate document includes the target candidate document, the target candidate document is determined as the target document, wherein the relationship index of the target candidate document is greater than a second threshold.

[0144] In actual applications, the method of "inputting the data to be retrieved and at least one reference document into the pre-trained language model to obtain the updated data to be retrieved" can refer to the above-mentioned implementation method of "inputting the questions to be retrieved, historical conversation data and knowledge base description information into the pre-trained language model to obtain the data to be retrieved"; the method of "retrieving at least one updated candidate document from multiple documents based on the updated data to be retrieved" can refer to the above-mentioned implementation method of "retrieving at least one candidate document from multiple documents in the knowledge base based on the data to be retrieved", and the embodiments of this specification will not be repeated here.

[0145] It should be noted that after retrieving at least one updated candidate document from multiple documents, the relationship index of the at least one updated candidate document can be determined. If the at least one updated candidate document includes a target candidate document whose relationship index is greater than the second threshold, it means that the target document for generating the reply result has been found. At this time, the target candidate document can be directly determined as the target document.

[0146] By applying the solution of the embodiments of this specification, the data to be retrieved and at least one reference document are input into a pre-trained language model to obtain updated data to be retrieved; based on the updated data to be retrieved, at least one updated candidate document is retrieved from multiple documents; when the at least one updated candidate document includes a target candidate document, the target candidate document is determined as the target document, and the data to be retrieved is updated using the reference document, thereby realizing positive and negative feedback interaction on the retrieval link, making the data to be retrieved more accurate, effectively solving retrieval errors caused by expression diversity and indirectness, and improving the accuracy of document retrieval.

[0147] In an optional embodiment of the present specification, after obtaining at least one updated candidate document from multiple documents based on the updated data to be retrieved, the following steps may also be included:

[0148] In the event that the target candidate document is not included in the at least one updated candidate document, the process returns to the step of filtering out at least one reference document from the at least one candidate document based on the association relationship between the data to be retrieved and the at least one candidate document, until the target candidate document is included in the at least one updated candidate document, thereby obtaining the target document.

[0149] It should be noted that after retrieving at least one updated candidate document from multiple documents, the relationship index of the at least one updated candidate document can be determined. If the at least one updated candidate document does not include the target candidate document whose relationship index is greater than the second threshold, it means that the target document for generating the reply result has not been found. At this time, the process can return to the step of filtering out at least one reference document from at least one candidate document based on the association relationship between the data to be retrieved and the at least one candidate document, and use the reference document found for the second time to update the updated data to be retrieved for the third time, until the at least one updated candidate document includes the target candidate document to obtain the target document.

[0150] By applying the solution of the embodiment of this specification, when the target candidate document is not included in the at least one updated candidate document, the step of screening out at least one reference document from the at least one candidate document based on the association relationship between the data to be retrieved and the at least one candidate document is returned to, until the target candidate document is included in the at least one updated candidate document, and the target document is obtained. By continuously updating the reference documents and continuously using the updated reference documents to update the data to be retrieved, the accuracy of the target document is guaranteed.

[0151] In one possible implementation of the present specification, during the process of iteratively calling a search link to perform document retrieval, it may be possible that the target document cannot be found. Therefore, in an embodiment of the present specification, a preset number of iterations is introduced to avoid redundant resource waste caused by endless retrieval. That is, in the case where the at least one updated candidate document does not include the target candidate document, after returning to the step of screening at least one reference document from the at least one candidate document based on the association between the data to be retrieved and the at least one candidate document, the following steps may also be included:

[0152] When the number of iterations reaches a preset number of iterations and the target candidate document is not included in the at least one updated candidate document, a reply result is generated based on the at least one reference document and the data to be retrieved.

[0153] Specifically, the preset number of iterations is set according to actual conditions, and the embodiments of this specification do not impose any limitation on this.

[0154] It should be noted that if the number of iterations reaches the preset number of iterations and the target candidate document is not included in at least one updated candidate document, it means that the target document that is strongly related to the data to be retrieved has not been found. Since the reference document is also a document that has an association relationship with the data to be retrieved, a reply result can be generated based on at least one reference document and the data to be retrieved.

[0155] In actual applications, the method of "generating a reply result based on at least one reference document and data to be retrieved" can refer to the above-mentioned implementation method of "generating a reply result based on the target document and data to be retrieved", and the embodiments of this specification will not be repeated.

[0156] For example, taking the example of inputting the data to be retrieved and at least one reference document into a pre-trained language model to obtain a reply result, the pre-trained language model can internally determine whether there is a document in at least one reference document that can be used to generate a reply result. If so, it can generate a reply result corresponding to the data to be retrieved based on the at least one reference document; if not, it can use its own language understanding ability to generate a reply result corresponding to the data to be retrieved.

[0157] By applying the solution of the embodiment of this specification, when the number of iterations reaches a preset number of iterations and the target candidate document is not included in at least one updated candidate document, a reply result is generated based on at least one reference document and the data to be retrieved, thereby improving the flexibility of the reply result.

[0158] The following further illustrates the document retrieval method provided in this specification using the application of the document retrieval method in an automatic question-answering scenario as an example, in conjunction with FIG5 . FIG5 shows a flowchart of an automatic question-answering method provided in one embodiment of this specification, specifically comprising the following steps:

[0159] Step 502: Obtain questions to be answered.

[0160] Step 504: According to the question to be answered, retrieve at least one candidate document from multiple documents in the knowledge base.

[0161] Step 506: Filter out at least one reference document from the at least one candidate document based on the association relationship between the question to be answered and the at least one candidate document.

[0162] Step 508: Update the question to be answered according to at least one reference document to obtain an updated question to be answered, and use the updated question to be answered to retrieve a target document from multiple documents.

[0163] Step 510: Generate a response result corresponding to the question to be answered based on the target document.

[0164] It should be noted that the implementation method of steps 502 to 508 can refer to the implementation method of the above-mentioned steps 402 to 408; the implementation method of step 510 can refer to the implementation method of the above-mentioned "generating a reply result based on the target document and the data to be retrieved", and the embodiments of this specification will not be repeated.

[0165] By applying the solution of the embodiments of this specification, candidate documents are obtained by rough sorting and retrieving from multiple documents, and reference documents are further obtained by fine sorting and retrieving from the candidate documents, thereby ensuring the accuracy of the reference documents. In addition, the reference documents are used to update the questions to be answered, thereby realizing positive and negative feedback interaction on the retrieval link, making the questions to be answered more accurate, effectively solving the retrieval errors caused by the diversity and indirectness of the expressions of the questions to be answered, and improving the accuracy of the answer results.

[0166] Referring to FIG. 6 , FIG. 6 shows a flowchart of a processing process of an automatic question-answering method provided by one embodiment of this specification. The automatic question-answering method is applied to a multi-agent system, wherein the multi-agent system includes a rewriting agent, a rough sorting agent, a fine sorting agent, and a text generation agent. The processing flow of each agent is described below:

[0167] Rewriting agent: After automatic question answering begins, the rewriting agent is used to refer to historical conversation data and knowledge base description information to rewrite the query into more comprehensive query data that is more in line with the current knowledge base context;

[0168] Coarse sorting agent: The coarse sorting agent is used to extract features of the data to be retrieved and obtain the features to be retrieved;

[0169] It should be noted that after the features to be retrieved are obtained by the coarse sorting agent, matching information between the features to be retrieved and the features of each document can be determined based on the similarity relationship between the features to be retrieved and the features of each document. At least one candidate document can be selected from the multiple documents based on the matching information. The at least one candidate document and the data to be retrieved can be passed to the fine sorting agent.

[0170] Precision ranking agent: Precision ranking agent is used to determine the relationship indicators between the data to be retrieved and each candidate document;

[0171] It should be noted that after the refined ranking agent obtains the relationship index corresponding to each candidate document, it can be determined whether there is a candidate document whose relationship index is greater than or equal to the second threshold. The following three situations may occur:

[0172] The first method is to determine the candidate document whose relationship index is greater than or equal to the second threshold as the target document if there is a candidate document whose relationship index is greater than or equal to the second threshold among the relationship indexes corresponding to the candidate documents (if yes), and pass the target document and the data to be retrieved to the text generation agent;

[0173] The second method is to pass the data to be retrieved to the text generation agent if the relationship index corresponding to each candidate document is less than the first threshold (if not),

[0174] The third type is that if the relationship indicators corresponding to each candidate document are all less than the second threshold, and there is a candidate document whose relationship indicator is greater than the first threshold (if not), then the candidate document whose relationship indicator is greater than the first threshold and less than the second threshold is determined as the reference document, and the reference document is input into the rewriting intelligent agent to obtain the updated data to be retrieved, and the updated data to be retrieved is used for iterative retrieval until the preset stop condition is reached.

[0175] Furthermore, after reaching the preset stop condition, two situations may occur. In the first possible situation, if at least one candidate document includes the target document, the target document and the data to be retrieved are passed to the text generation agent. In the second possible situation, if at least one candidate document does not include the target document, the reference document and the data to be retrieved are passed to the text generation agent.

[0176] Text Generation Agent: Since the data passed to the text generation agent may appear in various situations, the text generation agent has the following three processing methods:

[0177] The first one is to generate a response result based on the target document and the data to be retrieved;

[0178] The second method is to generate a response result based on the reference document and the data to be retrieved;

[0179] The third method is to generate a response result based on the data to be retrieved.

[0180] Applying the solution of the embodiments of this specification, the embodiments of this specification propose a retrieval enhancement solution based on multi-agent interaction, which uses the model's understanding, generation, and planning capabilities to construct a retrieval link containing multiple agents, allowing the agents to conduct multiple rounds of positive and negative feedback interactions with the knowledge base, and gradually obtain the precise target documents required to generate answer results, thereby effectively improving the recall rate of the retrieval link; at the same time, the embodiments of this specification can use a combination of large and small models to construct agents, thereby having significant advantages in time complexity, further improving the effect of the question-answering system based on the model and knowledge base.

[0181] Referring to Figure 7, Figure 7 shows a schematic diagram of an automatic question-and-answer interface provided by one embodiment of this specification. The automatic question-and-answer interface is divided into a request input interface and a result display interface. The request input interface includes a request input box, an "OK" control, and a "Cancel" control. The result display interface includes a result display box.

[0182] The user enters an automatic question-and-answer request through the request input box displayed on the client. The automatic question-and-answer request carries the question to be answered. The user clicks the "OK" control. The server receives the question to be answered sent by the client and retrieves at least one candidate document from multiple documents in the knowledge base based on the question to be answered. Based on the relationship between the question to be answered and the at least one candidate document, the server selects at least one reference document from the at least one candidate document. The server updates the question to be answered based on the at least one reference document to obtain an updated question to be answered. The server then uses the updated question to retrieve a target document from multiple documents. Based on the target document, the server generates a response corresponding to the question to be answered and sends the response to the client. The client displays the response in the result display box.

[0183] In actual applications, users can operate controls by clicking, double-clicking, touching, hovering the mouse, sliding, long pressing, voice control, or shaking, etc. The specific selection is based on the actual situation, and the embodiments of this specification do not impose any restrictions on this.

[0184] Corresponding to the above-mentioned document retrieval method embodiment, this specification also provides a document retrieval device embodiment. FIG8 shows a schematic diagram of the structure of a document retrieval device provided in one embodiment of this specification. As shown in FIG8 , the device includes:

[0185] A first acquisition module 802 is configured to acquire data to be retrieved;

[0186] A first retrieval module 804 is configured to retrieve at least one candidate document from a plurality of documents in a knowledge base according to the data to be retrieved;

[0187] A first screening module 806 is configured to screen at least one reference document from at least one candidate document based on an association relationship between the data to be retrieved and the at least one candidate document;

[0188] The second retrieval module 808 is configured to update the data to be retrieved according to at least one reference document to obtain updated data to be retrieved, and retrieve a target document from multiple documents using the updated data to be retrieved.

[0189] Optionally, the first screening module 806 is further configured to input the data to be retrieved and the first candidate document into a relationship determination model for the first candidate document, and obtain a relationship index corresponding to the first candidate document, wherein the relationship index is used to describe the degree of association between the first candidate document and the data to be retrieved, and the first candidate document is any one of the at least one candidate document; and determine the candidate document whose relationship index is greater than the first threshold and less than the second threshold as a reference document.

[0190] Optionally, the device also includes: a second generation module, configured to determine candidate documents whose relationship indicators are greater than or equal to a second threshold as target documents; input the data to be retrieved and the target document into a pre-trained language model to generate a reply result.

[0191] Optionally, the device further includes: a third generation module configured to input the data to be retrieved into a pre-trained language model to generate a reply result when the relationship indicators corresponding to at least one candidate document are all smaller than a first threshold.

[0192] Optionally, the device further includes: a fourth generating module configured to generate a reply result based on the target document and the data to be retrieved.

[0193] Optionally, the second retrieval module 808 is further configured to input the data to be retrieved and at least one reference document into a pre-trained language model to obtain updated data to be retrieved; retrieve at least one updated candidate document from multiple documents based on the updated data to be retrieved; and determine the target candidate document as the target document when the at least one updated candidate document includes a target candidate document, wherein the relationship index of the target candidate document is greater than a second threshold.

[0194] Optionally, the device also includes: an execution module, configured to return to execute the step of filtering out at least one reference document from at least one candidate document based on the association relationship between the data to be retrieved and at least one candidate document, when the target candidate document is not included in the at least one updated candidate document, until the target candidate document is included in the at least one updated candidate document, thereby obtaining the target document.

[0195] Optionally, the device also includes: a fifth generation module, configured to generate a reply result based on at least one reference document and the data to be retrieved when the number of iterations reaches a preset number of iterations and the target candidate document is not included in at least one updated candidate document.

[0196] Optionally, the first retrieval module 804 is further configured to obtain document features corresponding to multiple documents respectively; perform feature extraction on the data to be retrieved to obtain the features to be retrieved; match the features to be retrieved with the document features to respectively determine the matching information between the features to be retrieved and the features of each document; and screen out at least one candidate document from the multiple documents based on the matching information.

[0197] Optionally, the first acquisition module 802 is further configured to acquire the question to be retrieved, historical conversation data of the question to be retrieved, and knowledge base description information; and construct the data to be retrieved based on the question to be retrieved, the historical conversation data, and the knowledge base description information.

[0198] Optionally, the first acquisition module 802 is further configured to input the question to be retrieved, the historical conversation data and the knowledge base description information into a pre-trained language model to obtain the data to be retrieved.

[0199] By applying the solution of the embodiments of this specification, candidate documents are obtained by rough sorting and retrieving from multiple documents, and reference documents are further obtained by fine sorting and retrieving from the candidate documents, thereby ensuring the accuracy of the reference documents. In addition, the reference documents are used to update the data to be retrieved, realizing positive and negative feedback interaction on the retrieval link, making the data to be retrieved more accurate, effectively solving the retrieval errors caused by expression diversity and indirectness, and improving the accuracy of document retrieval.

[0200] The above is a schematic diagram of a document retrieval device according to this embodiment. It should be noted that the technical solution of the document retrieval device and the technical solution of the document retrieval method described above are based on the same concept. For details not described in detail in the technical solution of the document retrieval device, please refer to the description of the technical solution of the document retrieval method described above.

[0201] Corresponding to the above-mentioned automatic question-answering method embodiment, this specification also provides an automatic question-answering device embodiment. FIG9 shows a schematic structural diagram of an automatic question-answering device provided in one embodiment of this specification. As shown in FIG9 , the device includes:

[0202] The second acquisition module 902 is configured to obtain questions to be answered;

[0203] The third retrieval module 904 is configured to retrieve at least one candidate document from a plurality of documents in the knowledge base according to the question to be answered;

[0204] A second screening module 906 is configured to screen at least one reference document from the at least one candidate document based on the association relationship between the question to be answered and the at least one candidate document;

[0205] A fourth retrieval module 908 is configured to update the question to be answered based on at least one reference document to obtain an updated question to be answered, and retrieve a target document from the plurality of documents using the updated question to be answered;

[0206] The first generating module 910 is configured to generate a response result corresponding to the question to be answered based on the target document.

[0207] By applying the solution of the embodiments of this specification, candidate documents are obtained by rough sorting and retrieving from multiple documents, and reference documents are further obtained by fine sorting and retrieving from the candidate documents, thereby ensuring the accuracy of the reference documents. In addition, the reference documents are used to update the questions to be answered, thereby realizing positive and negative feedback interaction on the retrieval link, making the questions to be answered more accurate, effectively solving the retrieval errors caused by the diversity and indirectness of the expressions of the questions to be answered, and improving the accuracy of the answer results.

[0208] The above is a schematic diagram of an automatic question-answering device according to this embodiment. It should be noted that the technical solution of this automatic question-answering device and the technical solution of the automatic question-answering method described above are based on the same concept. For details not described in detail in the technical solution of the automatic question-answering device, please refer to the description of the technical solution of the automatic question-answering method described above.

[0209] Figure 10 shows a block diagram of a computing device according to one embodiment of the present disclosure. Components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and a database 1050 is used to store data.

[0210] The computing device 1000 also includes an access device 1040 that enables the computing device 1000 to communicate via one or more networks 1060. Examples of such networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 1040 may include one or more of any type of network interface (e.g., a Network Interface Card (NIC)) whether wired or wireless, such as an IEEE 802.11 Wireless Local Area Networks (WLAN) wireless interface, a World Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and the like.

[0211] In one embodiment of the present specification, the aforementioned components of the computing device 1000 and other components not shown in FIG10 may also be connected to each other, for example, via a bus. It should be understood that the computing device structure block diagram shown in FIG10 is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art may add or replace other components as needed.

[0212] Computing device 1000 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook computer, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 1000 may also be a mobile or stationary server.

[0213] The processor 1020 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the above-mentioned document retrieval method or automatic question-answering method.

[0214] The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solutions of the document retrieval method and the automatic question-answering method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the document retrieval method or the automatic question-answering method described above.

[0215] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the above-mentioned document retrieval method or automatic question-answering method.

[0216] The above is a schematic diagram of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solutions of the document retrieval method and the automatic question-answering method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the document retrieval method or the automatic question-answering method described above.

[0217] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute the steps of the above-mentioned document retrieval method or automatic question-answering method.

[0218] The above is a schematic diagram of a computer program according to this embodiment. It should be noted that the technical solution of this computer program is based on the same concept as the technical solutions of the document retrieval method and the automatic question-answering method described above. For details not described in detail in the technical solution of the computer program, please refer to the description of the technical solutions of the document retrieval method or the automatic question-answering method described above.

[0219] The foregoing description of this specification describes specific embodiments. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0220] The computer instructions include computer program code, which may be in source code form, object code form, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal, and software distribution medium. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electric carrier signals and telecommunication signals.

[0221] It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0222] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0223] The preferred embodiments disclosed above are intended only to help illustrate this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and variations can be made based on the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that those skilled in the art can better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A document retrieval method, comprising: Obtaining data to be retrieved; Retrieving at least one candidate document from multiple documents in a knowledge base according to the data to be retrieved; Filtering out at least one reference document from the at least one candidate document according to the association relationship between the data to be retrieved and the at least one candidate document; Updating the data to be retrieved according to the at least one reference document to obtain updated data to be retrieved, and using the updated data to be retrieved to retrieve a target document from the multiple documents.

2. The method according to claim 1, wherein The filtering out at least one reference document from the at least one candidate document according to the association relationship between the data to be retrieved and the at least one candidate document includes: For a first candidate document, inputting the data to be retrieved and the first candidate document into a relationship determination model to obtain a relationship index corresponding to the first candidate document, where the relationship index is used to describe the degree of association between the first candidate document and the data to be retrieved, and the first candidate document is any one of the at least one candidate document; Determining a candidate document with a relationship index greater than a first threshold and less than a second threshold as a reference document.

3. The method according to claim 2, wherein After inputting the data to be retrieved and the first candidate document into the relationship determination model to obtain the relationship index corresponding to the first candidate document, it further includes: Determining a candidate document with a relationship index greater than or equal to the second threshold as a target document; Inputting the data to be retrieved and the target document into a pre-trained language model to generate a response result.

4. The method according to claim 2, wherein, After inputting the data to be retrieved and the first candidate document into the relationship determination model to obtain the relationship index corresponding to the first candidate document for the first candidate document, it further includes: In the case where the relationship indices corresponding to the at least one candidate document are all less than the first threshold, inputting the data to be retrieved into a pre-trained language model to generate a response result.

5. The method according to claim 1, wherein After updating the data to be retrieved according to the at least one reference document to obtain updated data to be retrieved, and using the updated data to be retrieved to retrieve a target document from the multiple documents, it further includes: Generating a response result according to the target document and the data to be retrieved.

6. The method according to claim 1, wherein The updating the data to be retrieved according to the at least one reference document to obtain updated data to be retrieved, and using the updated data to be retrieved to retrieve a target document from the multiple documents includes: Inputting the data to be retrieved and the at least one reference document into a pre-trained language model to obtain updated data to be retrieved; Retrieving updated at least one candidate document from the multiple documents according to the updated data to be retrieved; In the case where the updated at least one candidate document includes a target candidate document, determining the target candidate document as the target document, where the relationship index of the target candidate document is greater than the second threshold.

7. The method according to claim 6, wherein, After retrieving the updated at least one candidate document from the multiple documents according to the updated data to be retrieved, it further includes: In the case that the target candidate document is not included in the at least one updated candidate document, return to perform the step of screening at least one reference document from the at least one candidate document according to the association relationship between the data to be retrieved and the at least one candidate document, until the target candidate document is included in the at least one updated candidate document, and obtain the target document.

8. The method according to claim 7, wherein, After the step of, in the case that the target candidate document is not included in the at least one updated candidate document, returning to perform the step of screening at least one reference document from the at least one candidate document according to the association relationship between the data to be retrieved and the at least one candidate document, further includes: In the case that the number of iterations reaches the preset number of iterations and the target candidate document is not included in the at least one updated candidate document, generate a reply result according to the at least one reference document and the data to be retrieved.

9. The method according to claim 1, wherein The retrieving at least one candidate document from multiple documents in the knowledge base according to the data to be retrieved includes: Obtain the document features respectively corresponding to the multiple documents; Perform feature extraction on the data to be retrieved to obtain the to-be-retrieved features; Match the to-be-retrieved features with the document features, and respectively determine the matching information between the to-be-retrieved features and each document feature; According to the matching information, screen out at least one candidate document from the multiple documents.

10. The method according to claim 1, wherein, The obtaining the data to be retrieved includes: Obtain the question to be retrieved, the historical conversation data of the question to be retrieved, and the knowledge base description information; Construct the data to be retrieved according to the question to be retrieved, the historical conversation data, and the knowledge base description information.

11. The method according to claim 10, wherein The constructing the data to be retrieved according to the question to be retrieved, the historical conversation data, and the knowledge base description information includes: Input the question to be retrieved, the historical conversation data, and the knowledge base description information into a pre-trained language model to obtain the data to be retrieved.

12. An automatic question answering method, including: Obtain the question to be answered; Retrieve at least one candidate document from multiple documents in the knowledge base according to the question to be answered; Screen out at least one reference document from the at least one candidate document according to the association relationship between the question to be answered and the at least one candidate document; Update the question to be answered according to the at least one reference document to obtain the updated question to be answered, and use the updated question to be answered to retrieve the target document from the multiple documents; Generate a reply result corresponding to the question to be answered according to the target document.

13. A computing device, including: A memory and a processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the document retrieval method according to any one of claims 1 to 11 or the automatic question answering method according to claim 12 are implemented.

14. A computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of the document retrieval method according to any one of claims 1 to 11 or the automatic question-answering method according to claim 12 are implemented.

Images