An information retrieval method based on a question-answer library, a question-answer system, and a computing device

Abstract

The present disclosure discloses a question and answer library-based information retrieval method, a question and answer system and a computing device, wherein the question and answer library contains at least one question and at least one answer corresponding to each question, and the question and answer library-based information retrieval method comprises: in response to a user query, determining a recalled question and a candidate answer subset corresponding to the recalled question from the question and answer library as at least one question and answer pair, wherein the candidate answer subset is from an answer set; inputting the user query and the at least one question and answer pair into a preset matching model for processing to obtain a matching degree of the user query and each question and answer pair; and sorting the question and answer pairs according to the matching degree as a retrieval result. According to the scheme of the present disclosure, the accuracy and relevance of the question and answer can be significantly improved, and the quality of the question and answer can be improved.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer network, and in particular, to an information retrieval scheme based on a question-answer library. BACKGROUND

[0002] To overcome the shortcomings of traditional search engines, question-answer system technology emerges as the times require. The question-answer system can accurately identify the query semantic intention expressed by the user's natural language and match the most relevant answer to the query semantic intention as the query result. Taking community question-answering as an example, more and more community question-answering platforms provide an online question and answer platform for Internet users to help people quickly obtain high-quality answers to daily or professional problems.

[0003] With the increasing popularity of community question-answering, problems in the platform have gradually emerged, one of which is that the quality of answers is uneven, and low-quality answers greatly affect the user experience on the platform; in addition, the diversity of natural language question expressions makes it difficult to match similar questions and obtain the best answer under similar questions.

[0004] Therefore, how to quickly and effectively match the closest and high-quality answers from the question-answer resources is a key problem of community question-answering platforms. SUMMARY

[0005] The present disclosure provides a question-answer library-based information retrieval method, a question-answer system and a computing device to try to solve or at least alleviate at least one of the above problems.

[0006] According to one aspect of the present disclosure, a question-answer library-based information retrieval method is provided, the question-answer library containing at least one question and at least one answer corresponding to each question as an answer set of each question, the method comprising: determining, in response to a user query, a recall question and a candidate answer subset corresponding to the recall question from the question-answer library as at least one question-answer pair recalled, wherein the candidate answer subset is from the answer set; inputting the user query and the at least one question-answer pair into a preset matching model for processing to obtain a matching degree of the user query and each question-answer pair; and sorting the question-answer pairs according to the matching degree as a retrieval result.

[0007] Optionally, the method according to the present disclosure further comprises the steps of: generating a preset matching model. Specifically, taking the user query, the question and the answer as training data, and labeling the training data based on a preset task to obtain labeled data; constructing a training model for generating the preset matching model, and setting initial parameters of the model; inputting the training data after splicing into the training model for processing to obtain a predicted matching degree and a predicted category; and using the predicted matching degree, the predicted category and the labeled data to calculate a loss value, and adjusting the model parameters of the training model based on the loss value until the training is completed to obtain the preset matching model.

[0008] Optionally, in the method according to the present disclosure, the training model comprises a language representation model, a text matching component and an entity recognition component; and when the training is completed, the trained language representation model and the trained text matching component are taken as the preset matching model.

[0009] Optionally, the method according to the present disclosure further comprises the steps of: for each question in the question and answer library, selecting at least one answer with a high matching degree with the question as a candidate answer subset of the question. Specifically, for each question in the question and answer library, splicing the question with each corresponding answer to obtain each spliced clause; inputting each spliced clause into the preset matching model to obtain the matching degree of the corresponding question and answer; when the matching degree of the question and the answer is higher than a threshold, taking the answer as the candidate answer of the question.

[0010] Optionally, in the method according to the present disclosure, the text matching component comprises a fully connected layer and a classifier, and the entity recognition component comprises a conditional random field network layer.

[0011] Optionally, the method according to the present disclosure further comprises the steps of: determining a first number of questions with a vector similarity greater than a preset value by calculating the vector similarity between the user query and each question in the question and answer library; searching the questions in the question and answer library using the user query to determine a second number of questions; and performing a deduplication process on the first number of questions and the second number of questions to determine the recalled questions.

[0012] According to another aspect of the present disclosure, a question and answer system is provided, comprising: a question and answer storage device adapted to store at least one question and at least one corresponding answer of each question as a question and answer library, wherein each question contains a corresponding answer set; an offline device adapted to train a preset matching model, and further adapted to determine a candidate answer subset of each question from the question and answer library using the preset matching model, wherein the candidate answer subset is from the answer set; and an online device adapted to determine at least one question and its corresponding candidate answer subset from the question and answer library as a search result based on the preset matching model in response to a user query.

[0013] Optionally, in the system according to this disclosure, the offline device includes: a training unit adapted to construct a training model to generate a preset matching model; and an answer selection unit adapted to use the preset matching model to select at least one answer with a high degree of matching with the question as a subset of candidate answers for the question.

[0014] Optionally, in the system according to this disclosure, the online device includes: a recall unit, adapted to respond to a user query by determining, from a question-and-answer database, a question to be recalled and a subset of candidate answers corresponding to the recalled question, as at least one question-and-answer pair to be recalled; and a sorting unit, adapted to input the user query and the recalled question-and-answer pairs into a preset matching model for processing to obtain the matching degree between the user query and each question-and-answer pair, and to sort the question-and-answer pairs according to the matching degree as the retrieval result.

[0015] According to another aspect of this disclosure, a computing device is provided, comprising: one or more processor memories; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing any of the methods described above.

[0016] According to another aspect of this disclosure, a computer-readable storage medium is provided for storing one or more programs, the one or more programs including instructions that, when executed by a computing device, cause the computing device to perform any of the methods described above.

[0017] In summary, according to the scheme disclosed herein, a training model based on joint learning is constructed, and the model is trained simultaneously by jointly performing text matching and entity recognition tasks. On the one hand, the information of user query, question, and answer is integrated into the semantic vector, enabling the model to learn more semantic information; on the other hand, the entity consistency of question-answer matching is improved. Furthermore, by using offline answer selection (determining a subset of candidate answers), online question recall, and online question-answer ranking, development and maintenance costs can be significantly reduced. Attached Figure Description

[0018] To achieve the foregoing and related objectives, certain illustrative aspects are described herein in conjunction with the following description and accompanying drawings. These aspects indicate various ways in which the principles disclosed herein may be practiced, and all aspects and their equivalents are intended to fall within the scope of the claimed subject matter. The foregoing and other objectives, features, and advantages of this disclosure will become more apparent from the following detailed description, taken in conjunction with the accompanying drawings. Throughout this disclosure, the same reference numerals generally refer to the same parts or elements.

[0019] Figure 1 A schematic diagram of a computing device 100 according to some embodiments of the present disclosure is shown;

[0020] Figure 2 FIG. 1 shows a flowchart of a method for information retrieval based on a question-answer corpus according to some embodiments of the present disclosure;

[0021] Figure 3 FIG. 2 shows a schematic diagram of training a model according to some embodiments of the present disclosure;

[0022] Figure 4 FIG. 3 shows a schematic diagram of a question-answer system according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0023] Exemplary embodiments of the present disclosure will be described herein below with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the present disclosure to those skilled in the art.

[0024] Applicants have found in their research that the construction of a high-quality community question-answer system needs to consider the following two factors: first, the similarity between the user query and the question in the question-answer corpus. The higher the similarity, the more relevant the answer under the question to the user query. Second, the quality of the answer list under the question in the question-answer corpus. High-quality answers should be provided to the user first.

[0025] Around the above two points, researchers have proposed various ways, among which "answer ranking" is an effective way to help users quickly pick out high-quality answers from a series of answers. Related research mainly focuses on the lexical and syntactic relationships between the user query and the question or between the user query and the answer, while ignoring the important role of semantic information in answer ranking and the matching degree of the user query, the question, and the answer.

[0026] In addition, there are currently two main approaches to the above problems: one is based on the matching of the user query and the answer, which directly retrieves the most semantically relevant answer from the candidate answers in the question-answer corpus; the other is a two-stage matching and ranking approach, which first matches the user query with the question, ranks the questions, and then ranks the answer list under the most relevant question to obtain the answer. For complex problems, they can be broken down into simple and independent sub-problems to be solved separately, and then the results are combined to obtain the results of the original complex problem. This may seem reasonable, but in reality, many problems cannot be broken down into independent sub-problems, and even if they can be broken down, the sub-problems are interconnected through some shared factors or shared representations. Treating real-world problems as independent single tasks will ignore the rich contextual information between problems.

[0027] Therefore, the present disclosure proposes an information retrieval scheme based on a question-answer library. The question-answer library contains at least one question and at least one answer corresponding to each question as an answer set of each question. The information retrieval based on the question-answer library is decomposed into two sub-tasks for joint learning. First, a joint learning model is trained based on the annotation of data of each sub-task type; second, the model is used to select answers for questions and answers; finally, multiple recalls are performed online, and user queries and question-answer pairs are matched and sorted. This method fully utilizes the knowledge expression and reasoning ability of the joint learning model, and can obtain more accurate and semantically relevant question answers.

[0028] The information retrieval scheme based on the question-answer library of the embodiments of the present disclosure can be executed in one or more computing devices. Figure 1 is a block diagram of an example computing device 100.

[0029] In a basic configuration 102, the computing device 100 typically includes a system memory 106 and one or more processors 104. A memory bus 108 can be used for communicating between the processor 104 and the system memory 106.

[0030] Depending on the desired configuration, the processor 104 can be any type of processing unit including, but not limited to, a microprocessor (μP), a microcontroller (μC), a digital signal processor (DSP), or any combination thereof. The processor 104 can include one or more levels of cache memory 110 and 112, a processor core 114, and registers 116. An example processor core 114 can include an arithmetic logic unit (ALU), a floating point unit (FPU), a digital signal processing core (DSP Core), or any combination thereof. An example memory controller 118 can be used with the processor 104 or, in some implementations, the memory controller 118 can be an internal part of the processor 104.

[0031] Depending on the desired configuration, the system memory 106 can be of any type including but not limited to volatile memory (such as RAM), non-volatile memory (such as ROM, flash memory, etc.) or any combination thereof. System memory 106 can include an operating system 120, one or more applications 122, and program data 124. In some embodiments, application 122 is arranged to operate with the operating system 120 on the program data 124. Operating system 120, for example, can be a Linux, Windows, etc., including a program of instructions for controlling a basic

[0032] When the computing device 100 is in operation, the processor 104 is configured to read instructions from the memory 106 and execute these instructions to perform the operations of the system. The application 122 is arranged to operate with the operating system 120 and the underlying hardware of the computing device 100 to implement various user desired functions. When the user launches the application 122, the application 122 is loaded into the memory 106 and the processor 104 reads and executes the instructions of the application 122 from the memory 106.

[0033] The computing device 100 also includes a storage device 132 that can include a removable storage 136 (e.g., a CD, DVD, memory stick, removable hard disk, etc.) and a non-removable storage 138 (e.g., a hard disk drive (HDD), etc.), which are connected through a storage interface bus 134.

[0034] The computing device 100 can also include a storage interface bus 134. The storage interface bus 134 enables communication between the storage devices 132 (e.g., the removable storage 136 and the non-removable storage 138) and the basic configuration 102 via the bus / interface controller 130. The operating system 120, the application 122, and the program data 124 are stored on the removable storage 136 and / or the non-removable storage 138 as well as in the system memory 106, and loaded into the system memory 106 at the time of booting the computing device 100 or when the application 122 is loaded into the memory 106 by the user. The system memory 106 can also include a temporary storage area for data that is loaded by the computing device 100.

[0035] The computing device 100 can also include an interface bus 140 for facilitating communication from various interface devices (e.g., output devices 142, peripheral interfaces 144, and communication devices 146) to the basic configuration 102 via the bus / interface controller 130. Example output devices 142 include a graphics processing unit 148 and an audio processing unit 150, which can be configured to facilitate communication to various external devices such as a display or speakers via one or more A / V ports 152. Example peripheral interfaces 144 include a serial interface controller 154 or a parallel interface controller 156, which can be configured to facilitate communication to various external devices such as input devices (e.g., keyboard, mouse, pen, voice input device, touch input device) or other peripheral devices (e.g., printer, scanner, etc.) via one or more I / O ports 158. An example

[0036] Network communication links can be one example of a communication media. Communication media can typically be embodied by computer readable instructions, data structures, program modules, and the like, in a modulated data signal, such as a carrier wave or other transport mechanism, and can include any information delivery media. A "modulated data signal" can be a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, communication media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, radio frequency (RF), microwave, infrared (IR) and other wireless media. The term computer readable media as used herein can include both storage media and communication media. In some embodiments, computer readable media stores one or more programs implementing some of the methods described herein.

[0037] The computing device 100 can be implemented as a portion of a small- sized portable (or mobile) electronic device such as a cell phone, a personal digital assistant (PDA), a personal media player device, a wireless web appliance, a personal headset, an application-specific device, or a hybrid device that can include any of the above functions. The computing device 100 can also be implemented as a personal computer including both desktop and notebook configurations. The computing device 100 can also be implemented as a server including the configurations described above.

[0038] In an embodiment according to this disclosure, computing device 100 is configured to execute a question-and-answer database-based information retrieval method. The application 122 of computing device 100 includes multiple program instructions for executing the question-and-answer database-based information retrieval method 200 according to this disclosure, and program data 124 may also store relevant data of a network model used to execute the method 200, including but not limited to training data and hyperparameter information.

[0039] Figure 2 A flowchart illustrating an information retrieval method 200 based on a question-answer database according to some embodiments of the present disclosure is shown. Figure 2 As shown, method 200 begins with step S210.

[0040] According to the embodiments of this disclosure, in method 200, before performing step S210, the method further includes the step of generating a preset matching model. The preset matching model is used in subsequent steps to obtain semantic information of the user query, question, and answer texts to determine the degree of matching between the three. Typically, the user query is the content to be searched input by the user in real time, which can be characters, words, sentences, or images, voice, etc. The question and answer are information pre-stored in a question-and-answer database. For ease of description below, one question and one corresponding answer in the question-and-answer database are denoted as "question-answer".

[0041] The following section will first explain the specific steps for training and generating a preset matching model.

[0042] The first step is to use user queries, questions, and answers as training data. Specifically, one user query, one question, and one answer are used as one training sample, and all these training samples constitute the training data.

[0043] Subsequently, based on the preset tasks, the training data are labeled to obtain labeled data. According to the embodiments of this disclosure, the preset tasks (i.e., the two sub-tasks mentioned above) include a text matching task and an entity recognition task.

[0044] The text matching task calculates the similarity between user queries and question-answer pairs, thereby classifying the degree of matching between user queries and question-answer pairs. The entity recognition task identifies entity words in the text, enabling the semantic vector of the text to learn entity information, making it more sensitive to semantic entity words, and also aiding the text matching task. Based on the above preset tasks, the generated labeled data includes labeled matching scores and labeled categories.

[0045] Specifically, based on the text matching task, the matching degree between the user query, question and answer is labeled as the matching degree. Respectively, 0 and 1. 0 indicates that the semantic of the user query and the question is different, or the answer cannot answer the user query well, 1 indicates that the semantic of the user query and the question is the same, and the answer can answer the user query well. For the text matching task, the training data can be labeled in the following format: user query||question||answer||labeled matching degree, three examples are shown as follows.

[0046] The Audi A4 is said to be the strongest control in the 30W inland table; the BMW 320 has higher overall configuration, better brand response, and higher price. The power of the two cars is similar, and it is difficult to choose which one is suitable for the car. || Which one is better, Attes and BMW 3 series? || Attes and BMW 3 series belong to the series that is easy to open and handle. The difference is mainly in price, power, brand, and interior. First of all, the price, if the budget is 25, choose BMW 320, if the budget is 20, choose Attes 2.5. The power of BMW 3 series is superior, the brand is also superior, and the sound insulation of 3 series is superior. In fact, the interior of 320 is not good, not as good as Attes, Attes at least uses comfortable leather, and 3 series is full of sintered plastic with lychee pattern. As for comfort, I think the two cars are half a kilogram, and the rear seat comfort is general. The difference in later cost is large, Attes is much cheaper and saves much more oil, and the starting noise is one level.

[0047] ||0

[0048] Which one is better, Q5 and XC60? || How to choose Audi Q5 and Volvo XC60? || Each has its advantages and disadvantages, XC60 T5 low configuration is about 330,000, and the cost performance is very high, Q5 L45 low configuration is about 400,000, first of all, the three major parts, the engine power is close, and the durability is good. The transmission of XC60 is Aisin 8AT, and the Q5L is much worse, using a seven-speed wet double isolation. The quality of the chassis is not much different, Q5L is more comfortable, and XC60 is harder, but most of the chassis is white aluminum alloy material. Space Q5L is more spacious, and the power is stronger, configuration, material, safety XC60 is better, and the car smell is smaller. If you have money, buy Volvo, if you have enough money, buy Audi. ||1

[0049] I have a question for everyone: what maintenance does a 2017 Volkswagen Sagitar 1.4T need after 30,000 kilometers? Does anyone know? || What maintenance is needed for a 2017 Sagitar 1.4T after 30,000 kilometers? || After a car has traveled 30,000 kilometers, the following maintenance should be performed: 1. Replacement. After 30,000 kilometers, the car will accumulate a lot of dust and oil, and its performance will be significantly affected. Therefore, it is best to replace it to ensure its working performance. Most manuals will clearly state that the air filter must be replaced after 30,000 kilometers. 2. Replacement. This is very similar to the air filter. After 30,000 kilometers, a lot of impurities will be generated, greatly reducing lubrication and heat dissipation. Therefore, regular replacement is also very necessary. 3. Fuel system cleaning. This generally includes cleaning the combustion chamber fuel system and cleaning the engine interior. Fuel system cleaning mainly removes carbon deposits, which are deposits formed by the mixture of lubricating oil and gasoline vapors. Of course, this isn't always necessary; it depends on the car's actual condition. If the car was already addressed during the last maintenance, then this step can be skipped. ||1

[0050] Based on the entity recognition task, the entity category to which each element in each text in the training data belongs is labeled as the label category. The labeling work needs to label the entities to be recognized according to specific business needs or specific corpus conditions. Taking the automotive industry as an example, entity categories include, but are not limited to: SERIES, PRICE, and COMP. In one embodiment, the BIO labeling method is used to perform ternary labeling on the training data, labeling each element as "BX", "IX", or "O". Here, "BX" indicates that the segment (i.e., entity) in which this element is located belongs to type X and this element is at the beginning of this segment; "IX" indicates that the segment in which this element is located belongs to type X and this element is not at the beginning of this segment (e.g., in the middle or at the end); and "O" indicates that it does not belong to any type.

[0051] For the training data, each sentence is treated as a text, and each text is separated by a blank line. For each text, each element and labeled category is separated by "\t" (or one or more spaces), as shown in the example below.

[0052] Among them, text 1 is: "What is the engine model of the 2020 Highlander?", and text 2 is: "Recommend a car that can be bought for around 300,000 yuan".

[0053] 2O

[0054] 0O

[0055] Model O

[0056] Han B-SERIES

[0057] Lan I-SERIES

[0058] Dai I-SERIES

[0059] O's

[0060] B-COMP

[0061] I-COMP

[0062] I-COMP

[0063] O's

[0064] O's

[0065] O's

[0066] O's

[0067] O's

[0068] 3B-PRICE

[0069] 0I-PRICE

[0070] I-PRICE

[0071] O's

[0072] O's

[0073] O's

[0074] O's

[0075] O's

[0076] O's

[0077] O's

[0078] O's

[0079] In the second step, a training model for generating a preset matching model is constructed, and initial parameters of the model are set.

[0080] According to an embodiment, the training model comprises a language representation model, a text matching component and an entity recognition component. Figure 3 A schematic diagram of the training model according to some embodiments of the present disclosure is shown. As Figure 3 The text matching component and the entity recognition component are respectively coupled to the language representation model to complete the aforementioned subtasks.

[0081] In one embodiment, the language representation model uses the BERT model, but is not limited to this. It should be understood that the fundamental purpose of using a language representation model in this method 200 is to obtain accurate semantic information that can express the text, thereby ensuring the accuracy of the retrieval results. Therefore, other language representation models, such as RoBERTa, ALBERT, ELECTRA, etc., can be used when encoding the text.

[0082] The third step is to concatenate the training data and input it into the training model for processing to obtain the predicted matching degree and predicted category.

[0083] Specifically, the training data is input into the language representation model for processing, resulting in a first semantic vector and a second semantic vector. For example... Figure 3 As shown, the BERT model input consists of sentence pairs, where one sentence is the user query and the other is a concatenation of the question and answer. The input to the language representation model is denoted as: CLS{user query}SEP{question}SEP{answer}SEP. In one embodiment, the answer portion is truncated when the total input length exceeds 512.

[0084] Language representation models encode the input text, converting the concatenation of user queries, questions, and answers into a fixed-dimensional semantic vector. For example... Figure 3 As shown, the output of the language representation model includes a first semantic vector V1 and a second semantic vector V2. The first semantic vector V1 is the output of the last Transformer corresponding to the classification token ([CLS]), and the second semantic vector V2 is the output of the last Transformer corresponding to other tokens.

[0085] In this way, the input of the language representation model integrates the three parts of text: user query, question, and answer. The encoding vector can learn the semantic information of the three parts at the same time, which can avoid the transmission error caused by two-stage matching and sorting.

[0086] Next, the first semantic vector V1 is input into the text matching component for processing to output the predicted matching score. In one embodiment, the text matching component includes a fully connected layer and a classifier (e.g., Softmax). The first semantic vector V1 is input into the fully connected layer, followed by a Softmax layer to calculate the predicted matching score.

[0087] Meanwhile, the second semantic vector V2 is input into an entity recognition component for processing to output a predicted class. In one embodiment, the entity recognition component includes a Conditional Random Fields (CRF) network layer. The second semantic vector V2 is input into the CRF network layer for decoding to output a class label with the largest probability value, i.e., the predicted class. According to one embodiment of the present disclosure, the form of the predicted class corresponds to the form of the annotated class annotated in the training phase (i.e., BIO triple annotation).

[0088] In the fourth step, the loss value is calculated using the predicted matching degree, the predicted class, and the annotated data, and the model parameters of the training model are adjusted based on the loss value until the training is completed, thereby obtaining a preset matching model.

[0089] According to some embodiments of the present disclosure, the loss value is calculated in the following manner.

[0090] On one hand, the first loss of the predicted matching degree and the annotated matching degree is calculated by a cross-entropy loss function, denoted as L relevance In one embodiment, the first loss is calculated in the following formula:

[0091]

[0092] In the formula, n is the number of samples, y i is the annotated matching degree (0 or 1), y′ i is the predicted matching degree.

[0093] On the other hand, the second loss of the predicted class and the annotated class is calculated by a cross-entropy loss function, denoted as L ner Since both are calculated by a cross-entropy loss function, the specific calculation of the second loss can refer to the related content of the first loss, which will not be repeated here.

[0094] Finally, the loss value is obtained based on the first loss and the second loss.

[0095] According to one embodiment, the dynamic weighting method is used to sum the loss values, and the loss value L can be represented by the following formula:

[0096] L = ω relevance *L relevance + ω ner *L ner ,

[0097] In the formula, ω relevance and ω ner are the weights of the first loss and the second loss, respectively, which are adjusted according to the difficulty and learning effect of different task learning.

[0098] The model parameters of the trained model are adjusted based on the loss value. In one embodiment, when the loss value exceeds a preset loss threshold, the parameters of the text matching component, entity recognition component, and BERT model are updated using backpropagation and gradient descent algorithms until training is complete. At the end of training, the trained language representation model and the trained text matching component are used as the preset matching model.

[0099] According to the embodiments of this disclosure, a training model is constructed to jointly learn two sub-tasks: text matching and entity recognition. The text matching sub-task matches from a semantic rather than keyword perspective, allowing the model's output encoding vector to learn more semantic information and thus improving the model's generalization ability. Simultaneously, the entity recognition sub-task emphasizes the focus on key entity components in the sentence, helping to improve the entity consistency of question-answer matching, thereby significantly improving the accuracy and relevance of question answering.

[0100] In step S210, in response to a user query, the question to be recalled and the subset of candidate answers corresponding to the question to be recalled are determined from the question-answer database as at least one question-answer pair to be recalled.

[0101] According to one embodiment of this disclosure, the recall phase employs vector recall and keyword recall methods.

[0102] On one hand, by calculating the vector similarity between the user query and each question in the question-answering database, a first number of questions with similarity scores greater than a preset value are identified. For example, using a sentence vector model, the user query and the question are encoded into semantic vectors respectively; then, the similarity between these two semantic vectors (i.e., the user query vector and the question vector) is calculated using vector similarity, and the K1 questions with the highest similarity scores are selected as the results of vector recall.

[0103] On the other hand, user queries are used to retrieve questions from the question-and-answer database to identify a second set of questions. For example, using open-source search engines like Elasticsearch, user queries are used to retrieve questions, and the top K2 questions are taken as the keyword recall results. Alternatively, keyword recall can extract keywords from both the user query and the question-and-answer database, and then calculate metrics such as co-occurrence, relevance, and similarity between the keywords in the user query and the question-and-answer database to obtain the keyword recall results. This disclosure does not impose excessive restrictions on the specific execution process of vector recall and keyword recall.

[0104] Then, the first and second number of questions are deduplicated to determine the questions to be recalled, which is the final result of the recall phase.

[0105] In addition, the candidate answer subset corresponding to the recall question is from the answer set of each recall question. According to still some embodiments of the present disclosure, the candidate answer subset can be determined before the recall stage, so as to improve the speed of online recall.

[0106] In some embodiments, for each question in the question-answer library, at least one answer with high matching degree with the question is selected as the candidate answer subset of the question. Specifically, the candidate answer subset of each question can be determined by the aforementioned preset matching model. For each question, the question is concatenated with each answer corresponding to the question respectively, and each concatenated clause is obtained. Each concatenated clause is input into the preset matching model, and the matching degree of the corresponding question and answer is obtained. When the matching degree of the question and the answer is higher than a threshold, the answer is taken as the candidate answer of the question.

[0107] Since the semantic space of the user query and the question is the same, when determining the candidate answer subset, the input of the preset matching model is question-answer, and the output result is the matching degree of the question and the answer. The higher the matching degree of the question and the answer is, the better the answer is. Only the answer higher than the threshold is retained, so that the subsequent question-answer sorting only considers the high-quality answers under each question.

[0108] The recall stage adopts a multi-path recall strategy, simultaneously recalls question-answer pairs with high similarity from the semantic and keyword angles, and can improve the recall rate and the processing speed of the sorting stage. In addition, the candidate answer subset determined in advance can reduce the data volume of the sorting stage and reduce the online running time on the one hand, and the task performed offline can decouple the system and improve the performance of the system.

[0109] In step S220, the user query and the recalled question-answer pair are input into the preset matching model for processing, so as to obtain the matching degree of the user query and each question-answer pair.

[0110] As described above, the question-answer pair used in the matching stage is the question and the high-quality answer (candidate answer subset) under the question recalled in the recall stage. The user query and the recalled question-answer pair are matched by using the preset matching model. The input data is the concatenated user query and question-answer pair, and the output is the matching degree of the user query and the question-answer pair.

[0111] Subsequently, in step S230, the question-answer pairs are sorted according to the matching degree, so as to be taken as the retrieval result. In one embodiment, the higher the matching degree is, the higher the sorting is.

[0112] According to the embodiments of the present disclosure, the matching process synchronously integrates the semantic information of the user query, the question, and the answer, so as to improve the accuracy and controllability of the sorting.

[0113] To sum up, according to the information retrieval method based on the question and answer library, the training model based on joint learning is constructed, and the model training is performed jointly with the text matching task and the entity recognition task. On the one hand, the information of the user query, the question and the answer is integrated into the semantic vector, so that the model learns more semantic information. On the other hand, the entity consistency of the question and answer matching is improved. In addition, through the offline answer selection (determining the candidate answer subset), the online question recall and the online question and answer sorting, the development amount and the maintenance cost can be greatly reduced.

[0114] Figure 4 A schematic diagram of a question and answer system 400 according to some embodiments of the present disclosure is shown. As shown, the question and answer system 400 includes a question and answer storage device 410, an offline device 420 and an online device 430. Figure 4

[0115] The question and answer storage device 410 stores at least one question and at least one answer corresponding to each question as a question and answer library, wherein each question contains a corresponding answer set. As shown below the question and answer storage device 410, an exemplary question and answer library is shown, which contains Question1 and its corresponding m Answer. It should be understood that only one Question and its answer set is shown here, and a plurality of such questions and answer sets are contained in the question and answer library. Figure 4

[0116] The offline device 420 trains a preset matching model. In addition, the offline device 420 can also use the preset matching model to determine a candidate answer subset for each question in the question and answer library, respectively, wherein the candidate answer subset is from the answer set.

[0117] According to an embodiment, the offline device 420 includes a training unit 422 and an answer selection unit 424.

[0118] The training unit 422 constructs a training model to generate a preset matching model. For the construction and training process of the training model, reference can be made to the related description in the method 200, which will not be repeated here.

[0119] The answer selection unit 424 uses the preset matching model to select at least one answer with high matching degree for each question as a candidate answer subset corresponding to each question. As shown below the offline device 420, an exemplary schematic diagram of selecting a candidate answer subset is shown. For Question1, n Answer (n≤m) with high matching degree is selected from the answer set as a candidate answer subset. Figure 4

[0120] ​​​It should be noted that the process of selecting a subset of candidate answers by answer selection unit 424 can be referred to the relevant description in method 200 above, and will not be repeated here.

[0121] In response to a user query, the online device 430 determines at least one question and its corresponding subset of candidate answers from the question-and-answer database based on a preset matching model, and uses this as the retrieval result.

[0122] According to one embodiment, the online device 430 includes a recall unit 432 and a sorting unit 434.

[0123] Recall unit 432 responds to a user query by determining the questions to be recalled and the corresponding subset of candidate answers from the question-and-answer database, as at least one question-and-answer pair to be recalled. For example... Figure 4 As shown, in response to user queries, at least Question1 and Question2 are identified as issues to be recalled. s (Due to space limitations, other recall questions not shown are represented by ellipses in the figure.) The recall question and answer pairs are composed of the candidate answer subsets corresponding to the recall questions. For the specific execution process of the recall unit 432, please refer to the description of step S210 in method 200 above; it will not be repeated here.

[0124] The sorting unit 434 inputs the user query and the retrieved question-answer pairs into a preset matching model for processing to obtain the matching degree between the user query and each question-answer pair, and sorts the question-answer pairs according to the matching degree as the retrieval results. Figure 4 The final sorting results are shown: Question1 + Answer p Question s +Answer1 is used as the search result. For the specific execution process of sorting unit 434, please refer to the descriptions of steps S220 and S230 in method 200 above; they will not be repeated here.

[0125] This disclosure proposes a question-answering system. By establishing joint learning, not only can entity information be effectively fused into text vectors, but information from user queries, questions, and answers in the question-answering database can also be simultaneously incorporated into the ranking process, thereby matching the most relevant answers in community question-answering. This question-answering system has the following characteristics.

[0126] 1) Joint learning. Joint learning is performed by constructing two sub-tasks of text matching and entity recognition. Among them, the text matching task matches from the semantic rather than the keyword perspective, so that the encoding vector output by the model can learn more semantic information, and the model has stronger generalization ability. At the same time, the entity recognition task emphasizes the attention to the key entity components in the sentence, which helps to improve the entity consistency of the question and answer matching, and thus significantly improves the accuracy and relevance of the question and answer, and improves the quality of the question and answer.

[0127] 2) Comprehensive user query-question-answer information. The constructed training model simultaneously fuses the user query, question and answer three-part text at the input end, and the encoding vector can simultaneously learn the semantic information of the three, which can avoid the transmission error caused by the two-stage matching and sorting.

[0128] 3) Offline answer selection. The candidate answer subset is determined in advance, which can filter out low-quality answers, reduce the data volume in the sorting stage, and reduce the online running time. On the other hand, the task is performed offline, which can decouple the system and improve the performance of the system.

[0129] 4) Online, the recall stage adopts a multi-path recall strategy, simultaneously recalling question and answer pairs with high similarity from the semantic and keyword perspectives, which can improve the recall rate and the processing speed of the sorting stage. The sorting stage uses a preset matching model to calculate the similarity score (matching degree), and the model input is composed of the splicing of the user query, question and answer, and the matching process fuses the information of the three parts, thereby improving the accuracy and controllability of the sorting.

[0130] In summary, according to the information retrieval method based on the question and answer library and the question and answer system of the present disclosure, through the model based on joint learning and the three-stage method of offline answer selection+online recall+online sorting, the sorting effect is greatly improved. The scheme of the present disclosure has the characteristics of accurate, reliable and efficient question and answer, and is suitable for application scenarios of community question and answer in various vertical industries, and can also be popularized to question and answer or search in other scenarios.

[0131] The various techniques described herein can be implemented in connection with hardware or software or, where appropriate, with a combination of hardware / software. Thus, the methods and apparatus of the present disclosure, or certain aspects or portions thereof, can take the form of program code (i.e., instructions) embodied in tangible media, such as removable hard disks, USB flash drives, floppy disks, CD-ROMs, or any other machine-readable storage medium wherein, when the program code is loaded into an apparatus, such as a computer, the apparatus becomes an apparatus for practicing the present disclosure.

[0132] In the case of program code execution on programmable computers, computing devices generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. The storage medium is configured to store program code means described herein. The processor is configured to execute instructions of the program code means stored in the storage medium to perform the question and answer library based information retrieval method of the present disclosure.

[0133] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to avoid obscuring the understanding of this description.

[0134] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to avoid obscuring the understanding of this description.

[0135] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the disclosure can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to avoid obscuring the understanding of this description.

[0136] Similarly, it is to be understood that the disclosure can be practiced with more than one type of computer system, virtual system, or other device, and that the underlying functions with various interactions can be analogous regardless of the physical universe in which they take place. The embodiments are based on flowcharts, algorithmic descriptions, and / or examples of sequences of operations. Although the description might describe the operations as a sequential process, many of the operations can be performed in parallel or concurrently, and / or the description could be performed at different times. In addition, the interrelation of one operation flow to another is described with, for example, the dependency of one or more operations on the outcome of another operation or a decision to perform another operation only if a situation occurs. However, these dependencies are described for enacted purposes to provide context for the operations. If the dependencies are irrelevant under a certain circumstance then these operations are interleaved, performed by different systems, and / or performed at the same time rather than wait for the outcomes of one or more operations.

[0137] Those skilled in the art will understand that the modules, or units, or components of the devices in the examples disclosed herein can be arranged in the devices as described in the examples, or alternatively can be located in one or more devices different from the devices in the examples. The modules in the foregoing examples can be combined as one module, or in the alternative can be divided into multiple sub-modules.

[0138] Those skilled in the art will understand that the modules in the devices in the examples can be adaptively changed and arranged in one or more devices different from the examples. The modules or units or components in the examples can be combined as one module or unit or component, and in the alternative can be divided into multiple sub-modules or sub-units or sub-components. All the features disclosed in this specification (including the claims, abstract, and drawings) and all the processes or units of any methods or apparatuses so disclosed can be combined in any combination, except where such features or processes or units are mutually exclusive. Each feature disclosed in this specification (including the claims, abstract, and drawings) can be replaced by alternative features providing the same, equivalent, or similar functionality unless expressly stated otherwise.

[0139] The present disclosure discloses:

[0140] A8. The method of any one of A2-7, wherein calculating the loss value using the predicted matching degree, the predicted category, and the labeled data comprises: calculating, by a cross-entropy loss function, a first loss of the predicted matching degree and the labeled matching degree, and a second loss of the predicted category and the labeled category; and obtaining the loss value based on the first loss and the second loss.

[0141] A9. The method of any one of A2-8, wherein the text matching component comprises a fully connected layer and a classifier, and the entity recognition component comprises a conditional random field network layer.

[0142] A10. The method of any one of A1-9, wherein determining, in response to a user query, the recalled questions from the question and answer database comprises: determining a first number of questions with a similarity greater than a preset value by calculating a vector similarity between the user query and each question in the question and answer database; searching the questions in the question and answer database using the user query to determine a second number of questions; and performing a deduplication process on the first number of questions and the second number of questions to determine the recalled questions.

[0143] B12. The question-answering system of B11, wherein the offline device comprises: a training unit adapted to construct a training model to generate a preset matching model; and an answer selection unit adapted to select at least one answer with high matching degree to the question as a candidate answer subset of the question by using the preset matching model.

[0144] B13. The question-answering system of B11 or B12, wherein the online device comprises: a recall unit adapted to determine a recalled question and a candidate answer subset corresponding to the recalled question from the question-answering database as at least one recalled question-answer pair in response to a user query; and a ranking unit adapted to input the user query and the recalled question-answer pair into a preset matching model to obtain a matching degree between the user query and each of the question-answer pairs, and rank the question-answer pairs according to the matching degrees to obtain a retrieval result.

[0145] Furthermore, those skilled in the art will appreciate that the features of the various embodiments can be combined with each other, as means within the scope of the present disclosure and form different embodiments. For example, in the following claims, any one of the claimed embodiments can be used in any combination.

[0146] Furthermore, some of the embodiments described herein are combinations of method or method elements with the processor of a computer system or other apparatuses that perform the functions of the method elements. Accordingly, those elements of the method or method elements that are implemented as computer software, whether stored in a memory, executed on a processor, or otherwise, are a means for implementing the steps of the method or elements of the method. The processor of a computer system or other apparatuses that performs the functions of the method elements is a means for implementing the steps of the method or elements of the method.

[0147] As used herein, unless otherwise specified, the use of the ordinal adjectives "first", "second", "third", etc., merely to distinguish different instances of an object to prevent the use of the same adjective repeatedly throughout the detailed description, and are not intended to imply that a combination of objects, for example, "first" and "second", are either ordered in time, ordered in spatial, or are otherwise actually two of a type.

[0148] While the present disclosure has been described in some embodiments, further modifications and variations from the teachings disclosed herein will occur to those skilled in the art once briefed on the above descriptions. Accordingly, it is to be understood that the application is not to be limited by the foregoing description, but is only limited by the scope of the appended claims. Moreover, the language used in this specification has been principally selected for readability and instructional purposes and can not have been selected to delineate or circumscribe the patent rights to which this disclosure relates.

Claims

1. An information retrieval method based on a question-and-answer database, wherein the question-and-answer database contains at least one question and at least one answer corresponding to each question, forming a set of answers for each question, the method comprising: In response to a user query, the question to be recalled and the candidate answer subset corresponding to the question to be recalled are determined from the question and answer database as at least one question and answer pair to be recalled, wherein the candidate answer subset comes from the answer set; The user query and the at least one question-and-answer pair are input into a preset matching model for processing to obtain the matching degree between the user query and each question-and-answer pair; as well as The question-answer pairs are sorted according to the matching degree to serve as the search results; The preset matching model includes a language representation model, a text matching component, and an entity recognition component. The language representation model encodes the user query and the at least one question-answer pair into a semantic vector. The output of the language representation model includes a first semantic vector and a second semantic vector. The first semantic vector is used as input to the text matching component to predict the matching degree and complete the text matching task. The second semantic vector is used as input to the entity recognition component to predict the entity category and complete the entity recognition task. The preset matching model jointly learns the text matching task and the entity recognition task during training.

2. The method as described in claim 1, further comprising the step of generating the preset matching model: User queries, questions, and answers are used as training data, and labeled according to a preset task to obtain labeled data; Build a training model to generate a preset matching model and set the initial parameters of the model; The training data is concatenated and then input into the training model for processing to obtain the predicted matching degree and predicted category; as well as Using the predicted matching degree, predicted category, and labeled data, a loss value is calculated, and the model parameters of the training model are adjusted based on the loss value until training is completed, resulting in a preset matching model.

3. The method of claim 1, further comprising the step of: For each question in the question-and-answer database, at least one answer with a high degree of matching with the question is selected as a subset of candidate answers for the question.

4. The method of claim 3, wherein, The step of selecting at least one answer with a high degree of matching to each question in the question-answering database as a subset of candidate answers for each question includes: For each question in the question-and-answer database, By concatenating the question with its corresponding answer, we obtain the concatenated clauses. Each concatenated clause is input into a preset matching model to obtain the corresponding matching degree between the question and the answer; When the match between a question and an answer is higher than a threshold, that answer is considered a candidate answer for that question.

5. The method of claim 2, wherein, The steps for labeling training data based on a pre-defined task to obtain labeled data include: Based on the text matching task, the matching degree between user queries, questions, and answers is labeled as the labeled matching degree; Based on the entity recognition task, the entity category to which each element in each text in the training data belongs is labeled as the label category.

6. The method as described in claim 2 or 5, wherein, Using the predicted matching degree, predicted category, and labeled data, the loss value is calculated as follows: The first loss between the predicted matching degree and the labeled matching degree, and the second loss between the predicted category and the labeled category are calculated using the cross-entropy loss function. The loss value is obtained based on the first loss and the second loss.

7. The method according to any one of claims 1-3, wherein, The text matching component includes a fully connected layer and a classifier, and the entity recognition component includes a conditional random field network layer.

8. The method according to any one of claims 1-3, wherein, In response to a user query, the steps of determining the questions to be recalled from the question-and-answer database include: By calculating the vector similarity between the user query and each question in the question-and-answer database, a first number of questions with similarity greater than a preset value are identified. The user query is used to retrieve questions from the question-and-answer database to determine a second number of questions; The first number of questions and the second number of questions are deduplicated to determine the questions to be recalled.

9. A question-answering system, comprising: A question-and-answer storage device is adapted to store at least one question and at least one answer corresponding to each question, as a question-and-answer library, wherein each question contains a corresponding set of answers; An offline device is suitable for training and generating a preset matching model, and is also suitable for determining a subset of candidate answers for each question in the question-answering database using the preset matching model. The preset matching model includes a language representation model, a text matching component, and an entity recognition component. The preset matching model jointly learns the text matching task and the entity recognition task during training. The subset of candidate answers comes from the answer set. An online device is adapted to respond to a user query and, based on the preset matching model, determine at least one question and its corresponding subset of candidate answers from the question-and-answer database as a retrieval result. It is also adapted to encode the user query and the at least one question-and-answer pair into a semantic vector based on the language representation model. The output of the language representation model includes a first semantic vector and a second semantic vector. The first semantic vector is used as input to a text matching component to predict the matching degree and complete the text matching task. The second semantic vector is used as input to an entity recognition component to predict the entity category and complete the entity recognition task.

10. The question-answering system as described in claim 9, wherein, The offline device includes: Training units are suitable for building training models to generate preset matching models; The answer selection unit is adapted to use the preset matching model to select at least one answer with a high degree of matching with the question as a subset of candidate answers for the question.

11. The question-answering system as described in claim 9 or 10, wherein, The online device includes: The recall unit is adapted to respond to a user query by determining the questions to be recalled and the subset of candidate answers corresponding to the recalled questions from the question-and-answer database, as at least one question-and-answer pair to be recalled. The sorting unit is adapted to input the user query and the retrieved question-answer pairs into a preset matching model for processing, so as to obtain the matching degree between the user query and each question-answer pair, and sort the question-answer pairs according to the matching degree as the retrieval result.

12. A computing device, comprising: One or more processors; Memory; One or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for performing the method as described in any one of claims 1-8.

13. A computer-readable storage medium storing one or more programs, said one or more programs including instructions that, when executed by a computing device, cause the computing device to perform the method as claimed in any one of claims 1-8.

Images