Knowledge retrieval method, device, equipment and storage medium

Abstract

Embodiments of the present application provide a knowledge retrieval method, apparatus, device, and storage medium. The method includes: obtaining information to be retrieved, the information to be retrieved including a retrieval keyword; determining multiple retrieval documents in a target document set whose relevance scores with the retrieval keyword meet a first preset threshold; determining a pre-trained language ALBERT model based on a question-answer matching (QA-match) model; determining similarities between the information to be retrieved and the multiple retrieval documents based on the ALBERT model; and outputting a target retrieval document from the multiple retrieval documents whose similarities meet a second preset threshold. This application can improve the accuracy of knowledge retrieval.

Description

Technical Field

[0001] The present application belongs to the field of business support technology, and specifically relates to a knowledge retrieval method, apparatus, device and storage medium. Background Art

[0002] Knowledge retrieval involves semantically indexing articles to form a knowledge base, performing business queries or searches within the knowledge base, and finally returning the retrieved information to the user. Traditional retrieval techniques, such as distributed search (Elastic Search, ES) and full-text search (SOLR), are commonly used for knowledge retrieval.

[0003] However, these knowledge retrieval methods only consider the keywords users submit during search, without factoring in the textual features of the knowledge elements themselves. This makes them unable to perform natural language processing (NLP) such as identifying user intent, resulting in low retrieval accuracy and suboptimal retrieval results and quality. Therefore, improving the accuracy of knowledge retrieval is an urgent issue that needs to be addressed. Summary of the Invention

[0004] The purpose of the embodiments of the present application is to provide a knowledge retrieval method, apparatus, device and storage medium that can solve the problem of how to improve the accuracy of knowledge retrieval.

[0005] In a first aspect, a knowledge retrieval method is provided, the method comprising: obtaining information to be retrieved, the information to be retrieved including retrieval keywords; determining multiple retrieval documents in a target document set whose relevance scores with the retrieval keywords meet a first preset threshold; determining a pre-trained language ALBERT model based on a question-answer matching QA-match model; wherein the QA-match model is used to obtain a training sample set for training the ALBERT model; determining the similarity between the information to be retrieved and the multiple retrieval documents based on the ALBERT model; and outputting a target retrieval document in the multiple retrieval documents whose similarity meets a second preset threshold.

[0006] In some implementation methods of the first aspect, the target document set includes first question-answer pair data, and before determining the ALBERT model according to the QA-match model, the method also includes: performing data enhancement on the first question-answer pair data in the target document set to obtain second question-answer pair data containing the first question-answer pair data; determining the ALBERT model according to the QA-match model, including: using the QA-match model to add classification labels to the second question-answer pair data to obtain a training sample set; wherein the training sample set includes the second question-answer pair data and the corresponding classification labels; and training the first preset neural network model based on the training sample set to obtain the ALBERT model.

[0007] In some implementation methods of the first aspect, the second question-answer pair data also includes third question-answer pair data, and data enhancement is performed on the first question-answer pair data in the target document set, including: replacing the phrases in the first question-answer pair data with synonyms of the phrases to obtain the third question-answer pair data.

[0008] In some implementation methods of the first aspect, the first question-answer pair data is in a first language, the second question-answer pair data also includes fourth question-answer pair data, and data enhancement is performed on the first question-answer pair data in the target document set, including: using a preset translation model to translate the first question-answer pair data in the first language into a second language; wherein the second language is a language different from the first language; and translating the first question-answer pair data in the second language into the first language to obtain the fourth question-answer pair data.

[0009] In some implementations of the first aspect, the QA-match model includes an embedding layer, a Transformer model, a multi-layer neural network MLP model, and an output layer; wherein the embedding layer is used to perform word embedding operations on the second question-answer pair data, and convert the second question-answer pair data represented in natural language into a vector or matrix form; and the output layer is a logistic Sigmod function.

[0010] In some implementation methods of the first aspect, the similarity between the information to be retrieved and multiple retrieved documents is determined according to the ALBERT model, including: outputting a relationship classification result between the information to be retrieved and each retrieved document according to the ALBERT model; wherein the relationship classification result is used to characterize the relevance between the retrieved information and the retrieved document; inputting the relationship classification result into the classification network model Softmax, and outputting a probability value used to characterize the similarity between the information to be retrieved and the retrieved document.

[0011] In some implementations of the first aspect, the first question-answer pair data includes a question text and multiple answer documents associated with the question text, and the answer documents correspond to document titles. The method also includes: obtaining user click data on the document title; adding classification labels to the question text and the document titles of the multiple answer documents associated with the question text based on the click data to obtain target data; and training a second preset neural network model based on the target data to obtain a target model.

[0012] In some implementations of the first aspect, determining the ALBERT model according to the QA-match model includes: determining the ALBERT model according to the QA-match model and the target model.

[0013] In some implementations of the first aspect, after obtaining the information to be retrieved, the method further includes: removing modal particles and privacy information in the information to be retrieved to obtain information to be segmented; performing segmentation processing on the information to be segmented to obtain search keywords in the information to be segmented.

[0014] In some implementations of the first aspect, determining multiple retrieval documents in a target document set whose relevance scores with the retrieval keyword meet a first preset threshold includes: determining the relevance score of each document in the target document set with the retrieval keyword based on a BM25L model; and screening out multiple retrieval documents from the target document set whose relevance scores meet the first preset threshold.

[0015] In a second aspect, a knowledge retrieval device is provided, which includes: an acquisition module for acquiring information to be retrieved, wherein the information to be retrieved includes retrieval keywords; a determination module for determining multiple retrieval documents in a target document set whose correlation scores with the retrieval keywords meet a first preset threshold; the determination module is also used to determine a pre-trained language ALBERT model based on a question-answer matching QA-match model; wherein the QA-match model is used to obtain a training sample set for training the ALBERT model; the determination module is also used to determine the similarity between the information to be retrieved and multiple retrieval documents based on the ALBERT model; and an output module is used to output a target retrieval document in the multiple retrieval documents whose similarity meets a second preset threshold.

[0016] In some implementation methods of the second aspect, the target document set includes first question-answer pair data, and the device also includes: a data enhancement module, used to perform data enhancement on the first question-answer pair data in the target document set to obtain second question-answer pair data containing the first question-answer pair data; an adding module, used to use the QA-match model to add classification labels to the second question-answer pair data to obtain a training sample set; wherein the training sample set includes the second question-answer pair data and the corresponding classification labels; a training module, used to train the first preset neural network model based on the training sample set to obtain an ALBERT model.

[0017] In some implementations of the second aspect, the second question-answer pair data also includes third question-answer pair data, and the data enhancement module is specifically used to replace the phrases in the first question-answer pair data with synonymous phrases to obtain the third question-answer pair data.

[0018] In some implementation methods of the second aspect, the first question-answer pair data is in a first language, the second question-answer pair data also includes fourth question-answer pair data, and the data enhancement module is further used to: use a preset translation model to translate the first question-answer pair data in the first language into a second language; wherein the second language is a language different from the first language; translate the first question-answer pair data in the second language into the first language to obtain the fourth question-answer pair data.

[0019] In some implementation methods of the second aspect, the QA-match model includes an embedding layer, a Transformer model, a multi-layer neural network MLP model and an output layer; wherein the embedding layer is used to perform word embedding operations on the second question-answer pair data, and convert the second question-answer pair data represented in natural language into a vector or matrix form; the output layer is a logistic Sigmod function.

[0020] In some implementations of the second aspect, the device also includes: an output module, which is also used to output the relationship classification result between the information to be retrieved and each retrieved document based on the ALBERT model; wherein the relationship classification result is used to characterize the relevance between the retrieved information and the retrieved document; an input module, which is used to input the relationship classification result into the classification network model Softmax, and output a probability value used to characterize the similarity between the information to be retrieved and the retrieved document.

[0021] In some implementation methods of the second aspect, the first question-answer pair data includes a question text and multiple answer documents associated with the question text, and the answer documents correspond to document titles. The device also includes: an acquisition module for obtaining user click data on document titles; an adding module for adding classification labels to the question text and the document titles of the multiple answer documents associated with the question text based on the click data to obtain target data; and a training module for training a second preset neural network model based on the target data to obtain a target model.

[0022] In some implementations of the second aspect, the determination module is specifically used to: determine the ALBERT model based on the QA-match model and the target model.

[0023] In some implementations of the second aspect, the device further includes: a removal module for removing modal particles and privacy information in the information to be retrieved to obtain information to be segmented; and a segmentation module for performing segmentation processing on the information to be segmented to obtain search keywords in the information to be segmented.

[0024] In some implementations of the second aspect, the device also includes: a determination module, which is also used to determine the relevance score between each document in the target document set and the retrieval keyword based on the BM25L model; and a screening module, which is used to screen out multiple retrieval documents whose relevance scores meet a first preset threshold from the target document set.

[0025] In a third aspect, a knowledge retrieval device is provided, comprising: a processor and a memory storing computer program instructions; when the processor executes the computer program instructions, the knowledge retrieval method in the first aspect or some implementable embodiments of the first aspect is implemented.

[0026] In a fourth aspect, a computer storage medium is provided, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the knowledge retrieval method in the first aspect or some implementable embodiments of the first aspect is implemented.

[0027] The present application relates to the field of Internet technology, and in particular to a knowledge retrieval method, apparatus, device and storage medium, which can perform a preliminary search for a retrieval keyword on all documents in the target document set by obtaining information to be retrieved including a retrieval keyword and determining a plurality of retrieval documents in the target document set whose relevance scores with the retrieval keyword meet a first preset threshold. Then, the ALBERT model is determined based on the QA-match model, which enables the ALBERT model to enhance the semantic understanding of the information to be retrieved and the plurality of retrieval documents based on deep learning, thereby accurately determining the semantic similarity between the information to be retrieved and the plurality of retrieval documents, and thus completing further retrieval of the plurality of retrieval documents. In this way, the above two-step retrieval takes into account the literal information of the retrieval keyword and the semantic understanding of the information to be retrieved, and can effectively improve the accuracy of knowledge retrieval. On this basis, by outputting the target retrieval document, the retrieval result corresponding to the information to be retrieved can be successfully returned, and the retrieval process is successfully completed, effectively improving the retrieval effect and retrieval quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0029] Figure 1 This is a flow chart of a knowledge retrieval method provided in an embodiment of the present application;

[0030] Figure 2 This is a schematic diagram of the structure of a QA-match model provided in an embodiment of the present application;

[0031] Figure 3 This is a flow chart of another knowledge retrieval method provided in an embodiment of the present application;

[0032] Figure 4 This is a flow chart of another knowledge retrieval method provided in an embodiment of the present application;

[0033] Figure 5 is a schematic diagram of the retrieval effect of a different model provided in an embodiment of the present application;

[0034] Figure 6 This is a schematic diagram of the structure of a knowledge retrieval device provided in an embodiment of the present application;

[0035] Figure 7 This is a hardware structure diagram of a knowledge retrieval device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] The features and exemplary embodiments of various aspects of the present application will be described in detail below. In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain the present application, rather than to limit the present application. For those skilled in the art, the present application can be implemented without the need for some of these specific details. The following description of the embodiments is merely to provide a better understanding of the present application by illustrating the examples of the present application.

[0037] It should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or device comprising the element.

[0038] The term "and / or" in this article is merely a description of the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone.

[0039] As mentioned in the background art, in related technologies, traditional retrieval technologies such as distributed search (Elastic Search, ES) and full-text retrieval (SOLR) are usually used for knowledge retrieval.

[0040] However, the above-mentioned knowledge retrieval method only considers the keywords proposed by users during the search, but does not consider the text features of the knowledge elements themselves. It is unable to perform natural language processing (NLP) such as user intent recognition, and the retrieval accuracy is not high, resulting in unsatisfactory retrieval results and retrieval quality.

[0041] Therefore, how to improve the accuracy of knowledge retrieval is an urgent problem that needs to be solved.

[0042] In response to the problems arising in the related art, the embodiments of the present application provide a knowledge retrieval method, apparatus, device and storage medium, which can perform a preliminary search for a retrieval keyword on all documents in the target document set by obtaining information to be retrieved including retrieval keywords and determining multiple retrieval documents in the target document set whose relevance scores with the retrieval keywords meet a first preset threshold. Then, the ALBERT model is determined based on the QA-match model, which enables the ALBERT model to enhance the semantic understanding of the information to be retrieved and multiple retrieval documents based on deep learning, thereby accurately determining the semantic similarity between the information to be retrieved and the multiple retrieval documents, thereby completing further retrieval of multiple retrieval documents. In this way, the above two-step retrieval takes into account the literal information of the retrieval keywords and the semantic understanding of the information to be retrieved, which can effectively improve the accuracy of knowledge retrieval and solve the problem of low accuracy of knowledge retrieval in the related art.

[0043] The technical solutions of the embodiments of the present application are described below with reference to the accompanying drawings.

[0044] The following describes in detail the knowledge retrieval method provided by the embodiment of the present application. Figure 1 This is a flow chart of a knowledge retrieval method provided by an embodiment of the present application. The execution subject of the knowledge retrieval method may be a knowledge retrieval system. It should be noted that the execution subject does not constitute a limitation of the present application.

[0045] The knowledge retrieval method may include S101-S105:

[0046] S101, obtaining information to be retrieved, where the information to be retrieved includes a search keyword.

[0047] Here, the information to be retrieved may be information input by a user.

[0048] The information users enter for search often contains irrelevant words that are not searchable. For example, interjections like "ah," "oh," "um," and "please," or phrases like "thank you" and "ask" that lack clear business meaning, often increasing search computation overhead and affecting the accuracy of search results.

[0049] Therefore, in order to avoid the above problems, in one embodiment, after obtaining the information to be retrieved, the method may further include: first, removing the modal particles and privacy information in the information to be retrieved to obtain the information to be segmented; then, performing segmentation processing on the information to be segmented to obtain the search keywords in the information to be segmented.

[0050] In the above embodiment, the knowledge retrieval device can perform data cleansing on the information to be retrieved that contains modal particles to obtain information to be segmented that does not contain modal particles. Private information can include privacy-related content such as a person's name, location, or phone number. The knowledge retrieval device can perform desensitization processing on the information to be retrieved that contains private content to obtain information to be segmented that does not involve personal privacy.

[0051] In one embodiment, performing word segmentation processing on the information to be segmented to obtain search keywords in the information to be segmented may include: performing word segmentation processing on the information to be segmented using a word segmentation algorithm to obtain search keywords in the information to be segmented.

[0052] In one embodiment, in a low-resource business scenario, in order to expand the recall range of the search results, word segmentation can be performed based on the minimum granularity to obtain as many search keywords as possible, and word segmentation can be added to further expand the recall range.

[0053] For example, by segmenting "Guangdong Mobile" according to the minimum granularity, two search keywords "Guangdong" and "Mobile" are obtained.

[0054] In another embodiment, by segmenting "Guangdong Mobile", in addition to obtaining the two search keywords "Guangdong" and "Mobile", the search keyword "Guangdong Mobile" can also be retained.

[0055] By removing irrelevant words and desensitizing private information, we can reduce unnecessary search computational overhead while improving the accuracy of search results. When segmenting words, combining minimum granularity segmentation with character segmentation can effectively increase the recall rate of word segmentation.

[0056] S102: Determine a plurality of search documents in the target document set whose relevance scores with the search keyword meet a first preset threshold.

[0057] Here, the target document set may be a knowledge base, and the second preset threshold may be set according to specific needs.

[0058] In one embodiment, determining a plurality of retrieval documents in a target document set whose relevance scores with the retrieval keyword meet a first preset threshold may include: S1021, determining the relevance score of each document in the target document set with the retrieval keyword based on the BM25L model; S1022, screening out a plurality of retrieval documents from the target document set whose relevance scores meet the first preset threshold.

[0059] In one embodiment, the above S1021 may specifically include the following steps:

[0060] Step 1: Use formula (1) to calculate the search keyword si The inverse document frequency (IDF) value of i ).

[0061]

[0062] Where N is the number of documents in the target document set, n(s i ) contains the search keyword s i The number of documents.

[0063] Step 2: Use formula (2) to calculate the search keyword s i The recall rate e td .

[0064]

[0065] Among them, f(s i ,d i ) is the search keyword s i In the document d i The frequency of , b is the document length penalty, dl is the document d i , avdl is the average length of all documents in the target document collection.

[0066] In one embodiment, the document length penalty term b may be set to 0.75.

[0067] Step 3: Use formula (3) to calculate the search keyword s i With document d i The correlation score S c (s i ,d i ).

[0068]

[0069] Wherein, i is an integer greater than 0, k1 is the word frequency gain term, and δ is the adjustment coefficient term, which is an integer greater than 0.

[0070] In one embodiment, the word frequency gain term k1 may be set to 1.2, and the adjustment coefficient term δ may be set to 0.1.

[0071] In one embodiment, when using the BM25 model to search in a document collection, the longer the document, the lower the relevance score of the document with the search keyword, and the search function may over-penalize very long documents. During the knowledge retrieval process, there are often situations where the title in the knowledge base is long, while the information to be retrieved or the sentence to be retrieved entered by the user is short. Therefore, the BM25L model of this application introduces an additional adjustment coefficient term δ, which provides a controllable lower bound for the penalty for long documents, thereby improving the relevance score of long documents with the search keyword.

[0072] S103: Determine a pre-trained language (ALite Bidirectional Encoder Representations from Transformers, ALBERT) model based on a question-answer matching (Quality Assurance, QA-match) model.

[0073] Among them, the QA-match model is used to obtain the training sample set for training the ALBERT model.

[0074] As a specific example, Figure 2 This is a schematic diagram of the structure of a QA-match model provided in the embodiment of the present application. Figure 2 As shown in the figure, the QA-match model includes an embedding layer, a Transformer model, a Multilayer Perceptron (MLP) model, and an output layer. The embedding layer is used to perform word embedding operations on the second question-answer pair data, converting the second question-answer pair data represented by natural language into a vector or matrix form; the output layer can be a logistic Sigmoid function.

[0075] In the above embodiment, the embedding layer can be composed of query features, answer features, and more features. At the same time, it can also be constructed based on which features are obtained.

[0076] In this way, by fusing features of different levels in the embedding layer, the retrieval effect is greatly improved.

[0077] In related technologies, the ALBERT model actually converts the similarity between two sentences into a binary classification problem. However, this is essentially a supervised problem, and supervised problems inevitably require labels. However, in the case of a cold start of a knowledge retrieval system, there is usually a lack of labeled data, that is, binary classification data. Therefore, when building the ALBERT model, there is often a problem of insufficient training data for deep learning models. In actual production environments and implementation scenarios, binary classification data can only be obtained through manual labeling, which requires a lot of human resources, and the quality of manual labeling is sometimes unstable. Therefore, how to effectively obtain binary classification data in the case of a cold start of a knowledge retrieval system is an urgent problem that needs to be solved.

[0078] In one embodiment, the target document set includes first question-answer pair data. In order to effectively increase the amount of question-answer pair data and improve the training effect of the deep learning model, before determining the ALBERT model based on the QA-match model, the method may also include: performing data enhancement on the first question-answer pair data in the target document set to obtain second question-answer pair data containing the first question-answer pair data.

[0079] In the above embodiment, the second question-answer pair data also includes third question-answer pair data. Data enhancement of the first question-answer pair data in the target document set may include: replacing the phrases in the first question-answer pair data with synonyms of the phrases to obtain the third question-answer pair data.

[0080] In the above embodiment, the first question-answer pair data is in the first language, and the second question-answer pair data also includes fourth question-answer pair data. Data enhancement is performed on the first question-answer pair data in the target document set, including: using a preset translation model to translate the first question-answer pair data in the first language into the second language; wherein the second language is a language different from the first language; and translating the first question-answer pair data in the second language into the first language to obtain the fourth question-answer pair data.

[0081] For example, the first question-answer pair data is "How to get rid of bad emotions", which is translated into the second language "how to getrid of bad emotions", and then the second language is translated into "How to get rid of bad emotions" to obtain the fourth question-answer pair data, thereby achieving the purpose of data enhancement.

[0082] In another embodiment, data enhancement may also be performed using a small sample learning method.

[0083] In this way, during a cold start of a knowledge retrieval system, data augmentation can effectively increase the amount of question-answer data. Furthermore, model training based on this augmented data can improve the training effectiveness of deep learning models.

[0084] In one embodiment, in order to effectively obtain binary classification data when the knowledge retrieval system is cold-started, the present application adopts a QA-match model. Determining the ALBERT model based on the QA-match model includes: first, using the QA-match model to add classification labels to the second question-answer pair data to obtain a training sample set; wherein the training sample set includes the second question-answer pair data and the corresponding classification labels; then, training a first preset neural network model based on the training sample set to obtain the ALBERT model.

[0085] In the above embodiment, when using the QA-match model to add classification labels to the second question-answer pair data, for a given knowledge base, Question1 and Answer1 of a single piece of knowledge can be labeled as positive, and QuestionN and the corresponding AnswerN without the core words of the question can be labeled as negative.

[0086] By using the QA-match model to add classification labels to the augmented second question-answer pair data, a large amount of binary classification data can be effectively acquired during a cold start of the knowledge retrieval system. Compared to manual labeling, this approach saves significant manpower and avoids the instability of manual labeling quality, significantly reducing the labor costs of knowledge base operation and maintenance. Furthermore, training a pre-set neural network model based on this acquired binary classification data can improve the training results of the deep learning model, resulting in a relatively strong ALBERT model.

[0087] S104: Determine the similarity between the information to be retrieved and the multiple retrieval documents based on the ALBERT model.

[0088] In one embodiment, the similarity between the information to be retrieved and multiple retrieved documents is determined based on the ALBERT model, including: outputting a relationship classification result between the information to be retrieved and each retrieved document based on the ALBERT model; wherein the relationship classification result is used to characterize the relevance between the retrieved information and the retrieved document; inputting the relationship classification result into the classification network model Softmax, and outputting a probability value used to characterize the similarity between the information to be retrieved and the retrieved document.

[0089] As a specific example, Figure 3 This is a flow chart of another knowledge retrieval method provided by the embodiment of the present application. Figure 3 As shown in the figure, the information to be retrieved Query and the retrieval document Doc obtained by the preliminary retrieval are input into the trained ALBERT model, and the relationship classification result is output; then, the relationship classification result is input into the classification network model Softmax, and the probability value used to characterize the similarity between the information to be retrieved and the retrieval document is output.

[0090] In one embodiment, when the information to be retrieved is strongly correlated with the retrieved document, the relationship classification result is close to 1; when the information to be retrieved is uncorrelated with the retrieved document, the relationship classification result is close to 0.

[0091] S105: Output a target retrieval document whose similarity among the multiple retrieval documents meets a second preset threshold.

[0092] Here, the second preset threshold can be set according to specific needs.

[0093] The knowledge retrieval method provided in the embodiment of the present application can perform a preliminary search for a retrieval keyword on all documents in the target document set by obtaining information to be retrieved including retrieval keywords and determining multiple retrieval documents in the target document set whose correlation scores with the retrieval keywords meet a first preset threshold. Then, the ALBERT model is determined based on the QA-match model, which enables the ALBERT model to enhance the semantic understanding of the information to be retrieved and multiple retrieval documents based on deep learning, thereby accurately determining the semantic similarity between the information to be retrieved and the multiple retrieval documents, thereby completing further retrieval of multiple retrieval documents. In this way, the above two-step retrieval takes into account the literal information of the retrieval keyword and the semantic understanding of the information to be retrieved, which can effectively improve the accuracy of knowledge retrieval. On this basis, by outputting the target retrieval document, the retrieval result corresponding to the information to be retrieved can be successfully returned, and the retrieval process is successfully completed, effectively improving the retrieval effect and retrieval quality.

[0094] After the knowledge retrieval system is cold-started, a large amount of click data on document titles from users can be received. At this time, these click data can be used for model training.

[0095] Figure 4 This is a flow chart of another knowledge retrieval method provided by the embodiment of the present application. Figure 4 As shown, the method may include S401-S404.

[0096] S401, obtaining user click data on the document title.

[0097] The first question-answer pair data includes a question text and a plurality of answer documents associated with the question text, and the answer documents correspond to document titles.

[0098] S402 , adding classification labels to the question text and the document titles of multiple answer documents associated with the question text according to the click data to obtain target data.

[0099] S403: Training a second preset neural network model based on the target data to obtain a target model.

[0100] Here, the target model may be a QQ-match model.

[0101] S404: Determine the ALBERT model based on the QA-match model and the target model.

[0102] Here, the QA-match model used in the original cold start is fused with the QQ-match model to obtain a fused model, that is, the fused ALBERT model.

[0103] As a specific example, Figure 5 This is a schematic diagram of the retrieval effect of a different model provided in the embodiment of the present application. Figure 5 As shown in the figure, the document recall rate is 82.71% when the BM25 model is used for knowledge retrieval; the document recall rate is 89.32% when the BM25 model and the pre-trained language representation (Bidirectional Encoder Representations from Transformers, BERT) model are used for knowledge retrieval; the document recall rate is 92.04% when the BM25 model and the conversion Transformer model are used for knowledge retrieval; and the document recall rate is the highest when the BM25L model and the fused ALBERT model are used for knowledge retrieval, which is 93.15%.

[0104] In this way, during the knowledge retrieval process, the BM25L model is used for preliminary retrieval, and the fused ALBERT model is used for further retrieval. The above two-step retrieval can effectively improve the overall generalization and retrieval capabilities.

[0105] Based on the knowledge retrieval method provided in the embodiment of the present application, accordingly, the embodiment of the present application also provides a knowledge retrieval device.

[0106] Figure 6 This is a schematic diagram of the structure of a knowledge retrieval device provided in an embodiment of the present application. Figure 6 As shown, the knowledge retrieval device 600 may include: an acquisition module 610 , a determination module 620 , and an output module 630 .

[0107] Among them, the acquisition module 610 is used to obtain information to be retrieved, and the information to be retrieved includes retrieval keywords; the determination module 620 is used to determine multiple retrieval documents in the target document set whose correlation scores with the retrieval keywords meet the first preset threshold; the determination module 620 is also used to determine the pre-trained language ALBERT model based on the question-answer matching QA-match model; wherein the QA-match model is used to obtain a training sample set for training the ALBERT model; the determination module 620 is also used to determine the similarity between the information to be retrieved and multiple retrieval documents based on the ALBERT model; the output module 630 is used to output the target retrieval document whose similarity among the multiple retrieval documents meets the second preset threshold.

[0108] In some embodiments, the target document set includes first question-answer pair data, and the device also includes: a data enhancement module 640, used to perform data enhancement on the first question-answer pair data in the target document set to obtain second question-answer pair data containing the first question-answer pair data; an adding module 650, used to use the QA-match model to add classification labels to the second question-answer pair data to obtain a training sample set; wherein the training sample set includes the second question-answer pair data and the corresponding classification labels; a training module 660, used to train the first preset neural network model based on the training sample set to obtain an ALBERT model.

[0109] In some embodiments, the second question-answer pair data also includes third question-answer pair data, and the data enhancement module 640 is specifically used to replace the phrases in the first question-answer pair data with synonymous phrases to obtain the third question-answer pair data.

[0110] In some embodiments, the first question-answer pair data is in a first language, the second question-answer pair data also includes a fourth question-answer pair data, and the data enhancement module 640 is further used to: use a preset translation model to translate the first question-answer pair data in the first language into a second language; wherein the second language is a language different from the first language; and translate the first question-answer pair data in the second language into the first language to obtain the fourth question-answer pair data.

[0111] In some embodiments, the QA-match model includes an embedding layer, a Transformer model, a multi-layer neural network MLP model and an output layer; wherein the embedding layer is used to perform word embedding operations on the second question-answer pair data, and convert the second question-answer pair data represented by natural language into a vector or matrix form; the output layer is a logistic Sigmod function.

[0112] In some embodiments, the device also includes: an output module 630, which is also used to output the relationship classification results between the information to be retrieved and each retrieved document based on the ALBERT model; wherein the relationship classification results are used to characterize the relevance between the retrieved information and the retrieved document; an input module 670, which is used to input the relationship classification results into the classification network model Softmax, and output a probability value used to characterize the similarity between the information to be retrieved and the retrieved document.

[0113] In some embodiments, the first question-answer pair data includes a question text and multiple answer documents associated with the question text, and the answer documents correspond to document titles. The device also includes: an acquisition module 610, used to obtain user click data on document titles; an adding module 650, which is also used to add classification labels to the question text and the document titles of the multiple answer documents associated with the question text based on the click data to obtain target data; a training module 660, which is also used to train a second preset neural network model based on the target data to obtain a target model.

[0114] In some embodiments, the determination module 620 is specifically configured to determine an ALBERT model based on the QA-match model and the target model.

[0115] In some embodiments, the device further includes: a removal module 670 for removing modal particles and privacy information from the information to be retrieved to obtain information to be segmented; a segmentation module 680 for performing segmentation processing on the information to be segmented to obtain search keywords in the information to be segmented.

[0116] In some embodiments, the device also includes: a determination module 620, which is also used to determine the relevance score of each document in the target document set and the retrieval keyword based on the BM25L model; a screening module 690, which is used to screen out multiple retrieval documents whose relevance scores meet a first preset threshold from the target document set.

[0117] Figure 6 The various modules in the knowledge retrieval device shown can be implemented Figure 1 、 Figure 3 、 Figure 4 The methods / steps performed by the knowledge retrieval system are described briefly and will not be repeated here.

[0118] The knowledge retrieval device of the embodiment of the present application can perform a preliminary search for a retrieval keyword on all documents in the target document set by obtaining information to be retrieved including retrieval keywords and determining multiple retrieval documents in the target document set whose correlation scores with the retrieval keywords meet a first preset threshold. Then, the ALBERT model is determined based on the QA-match model, which enables the ALBERT model to enhance the semantic understanding of the information to be retrieved and multiple retrieval documents based on deep learning, thereby accurately determining the semantic similarity between the information to be retrieved and the multiple retrieval documents, thereby completing further retrieval of multiple retrieval documents. In this way, the above two-step retrieval takes into account the literal information of the retrieval keyword and the semantic understanding of the information to be retrieved, which can effectively improve the accuracy of knowledge retrieval. On this basis, by outputting the target retrieval document, the retrieval result corresponding to the information to be retrieved can be successfully returned, and the retrieval process is successfully completed, effectively improving the retrieval effect and retrieval quality.

[0119] Based on the knowledge retrieval method provided in the embodiment of the present application, accordingly, the embodiment of the present application also provides a knowledge retrieval device.

[0120] Figure 7 This is a hardware structure diagram of a knowledge retrieval device provided in an embodiment of the present application.

[0121] like Figure 7 As shown, the knowledge retrieval device 700 in this embodiment includes an input device 701, an input interface 702, a central processing unit 703, a memory 704, an output interface 705, and an output device 706. The input interface 702, the central processing unit 703, the memory 704, and the output interface 705 are interconnected via a bus 710, and the input device 701 and the output device 706 are connected to the bus 710 via the input interface 702 and the output interface 705, respectively, and further connected to other components of the knowledge retrieval device 700.

[0122] Specifically, the input device 701 receives input information from the outside and transmits the input information to the central processing unit 703 through the input interface 702; the central processing unit 703 processes the input information based on the computer executable instructions stored in the memory 704 to generate output information, stores the output information temporarily or permanently in the memory 704, and then transmits the output information to the output device 706 through the output interface 705; the output device 706 outputs the output information to the outside of the knowledge retrieval device 700 for user use.

[0123] In one embodiment, Figure 7 The knowledge retrieval device 700 shown includes: a memory 704 for storing programs; a processor 703 for running the programs stored in the memory to execute the embodiments of the present application. Figure 1、 Figure 3 、 Figure 4 The method of the embodiment shown.

[0124] In addition, in combination with the knowledge retrieval method in the above embodiment, the embodiment of the present application can provide a computer storage medium for implementation. The computer storage medium stores computer program instructions; when the computer program instructions are executed by the processor, the embodiment of the present application provides Figure 1 、 Figure 3 、 Figure 4 The method of the embodiment shown.

[0125] It should be understood that the present application is not limited to the specific configurations and processes described above and illustrated in the figures. For the sake of brevity, a detailed description of known methods is omitted here. In the above embodiments, several specific steps are described and illustrated as examples. However, the method process of the present application is not limited to the specific steps described and illustrated. Those skilled in the art can make various changes, modifications, and additions, or change the order of the steps after understanding the spirit of the present application.

[0126] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, etc. When implemented in software, the elements of the present application are programs or code segments that are used to perform the required tasks. The program or code segment can be stored in a machine-readable medium, or transmitted on a transmission medium or a communication link via a data signal carried in a carrier wave. "Machine-readable medium" can include any medium that can store or transmit information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, read-only memories (ROMs), flash memories, erasable ROMs (EROMs), floppy disks, CD-ROMs, optical disks, hard disks, optical fiber media, radio frequency (RF) links, etc. The code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0127] It should also be noted that the exemplary embodiments mentioned in this application describe some methods or systems based on a series of steps or devices. However, this application is not limited to the order of the above steps. In other words, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0128] Aspects of the present application have been described above with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer or other programmable data processing device to produce a machine so that these instructions executed via the processor of the computer or other programmable data processing device enable the implementation of the function / action specified in one or more boxes of the flowchart and / or block diagram. This processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor or a field programmable logic circuit. It is also understood that each box in the block diagram and / or the flowchart and the combination of the boxes in the block diagram and / or the flowchart can also be implemented by the dedicated hardware that performs the specified function or action, or can be implemented by the combination of dedicated hardware and computer instructions.

[0129] The above is only a specific implementation method of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules and units described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. It should be understood that the scope of protection of the present application is not limited to this. Any technician familiar with this technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed in this application, and these modifications or replacements should be included in the scope of protection of this application.

Claims

1. A knowledge retrieval method, characterized in that: include: Acquiring information to be retrieved, wherein the information to be retrieved includes a search keyword; Determining a plurality of search documents in a target document set whose relevance scores with the search keyword meet a first preset threshold; the target document set includes first question-answer pair data; Determining a pre-trained language ALBERT model based on a question-answer matching (QA-match) model; wherein, using the QA-match model, adding classification labels to the first question-answer pair data to obtain a training sample set; determining a pre-trained language ALBERT model based on the training sample set; wherein, the training sample set includes the first question-answer pair data and corresponding classification labels; Determining the similarity between the information to be retrieved and the multiple retrieval documents according to the ALBERT model; Outputting a target retrieval document whose similarity among the plurality of retrieval documents meets a second preset threshold; The QA-match model includes an embedding layer, a Transformer model, a multi-layer neural network MLP model and an output layer; The embedding layer is used to perform a word embedding operation on the first question-answer pair data, convert the first question-answer pair data represented in natural language into a vector or matrix form, and fuse features at different levels, where the features at different levels include at least query features and question-answer features; The output layer is a logistic Sigmod function.

2. The method according to claim 1, characterized in that Before determining the ALBERT model according to the question-answer matching QA-match model, the method further includes: Performing data enhancement on the first question-answer pair data in the target document set to obtain second question-answer pair data including the first question-answer pair data; Determining a pre-trained language ALBERT model according to the training sample set includes: A first preset neural network model is trained based on the training sample set to obtain the ALBERT model.

3. The method according to claim 2, characterized in that The second question-answer pair data further includes third question-answer pair data, and the data enhancement of the first question-answer pair data in the target document set includes: The phrase in the first question-answer pair data is replaced with a synonymous phrase of the phrase to obtain the third question-answer pair data.

4. The method according to claim 2 or 3, characterized in that The first question-answer pair data is in a first language, the second question-answer pair data further includes fourth question-answer pair data, and the data enhancement of the first question-answer pair data in the target document set includes: translating the first question-answer pair data in the first language into a second language using a preset translation model; wherein the second language is a language different from the first language; The first question-answer pair data in the second language is translated into the first language to obtain fourth question-answer pair data.

5. The method according to claim 1, wherein Determining the similarity between the information to be retrieved and the multiple retrieval documents according to the ALBERT model includes: Outputting a relationship classification result between the information to be retrieved and each retrieved document according to the ALBERT model; wherein the relationship classification result is used to characterize the relevance between the retrieved information and the retrieved document; The relationship classification result is input into the classification network model Softmax, and a probability value representing the similarity between the information to be retrieved and the retrieved document is output.

6. The method according to claim 2, characterized in that The first question-answer pair data includes a question text and a plurality of answer documents associated with the question text, wherein the answer documents have corresponding document titles. The method further includes: Obtaining user click data on the document title; adding classification labels to the question text and the document titles of a plurality of answer documents associated with the question text according to the click data to obtain target data; A second preset neural network model is trained based on the target data to obtain a target model.

7. The method according to claim 6, characterized in that Determining the ALBERT model based on the question-answer matching QA-match model includes: The ALBERT model is determined according to the QA-match model and the target model.

8. The method according to claim 1, characterized in that After obtaining the information to be retrieved, the method further includes: Removing modal particles and privacy information from the information to be retrieved to obtain information to be segmented; Perform word segmentation processing on the information to be segmented to obtain the search keywords in the information to be segmented.

9. The method according to claim 1, characterized in that The step of determining a plurality of search documents in the target document set whose relevance scores with the search keyword meet a first preset threshold comprises: Determine a relevance score between each document in the target document set and the search keyword based on the BM25L model; The plurality of retrieved documents whose relevance scores meet the first preset threshold are screened out from the target document set.

10. A knowledge retrieval device, characterized in that: include: An acquisition module, configured to acquire information to be retrieved, wherein the information to be retrieved includes a search keyword; a determination module, configured to determine, in a target document set, a plurality of search documents whose relevance scores with the search keyword meet a first preset threshold; the target document set includes first question-answer pair data; The determination module is further configured to determine a pre-trained language ALBERT model based on a question-answer matching (QA-match) model; use the QA-match model to add classification labels to the first question-answer pair data to obtain a training sample set; and determine a pre-trained language ALBERT model based on the training sample set; wherein the training sample set includes the first question-answer pair data and corresponding classification labels; The determination module is further configured to determine the similarity between the information to be retrieved and the plurality of retrieval documents according to the ALBERT model; An output module, configured to output a target retrieval document whose similarity among the plurality of retrieval documents meets a second preset threshold; The QA-match model includes an embedding layer, a Transformer model, a multi-layer neural network MLP model and an output layer; The embedding layer is used to perform a word embedding operation on the first question-answer pair data, convert the first question-answer pair data represented in natural language into a vector or matrix form, and fuse features at different levels, where the features at different levels include at least query features and question-answer features; The output layer is a logistic Sigmod function.

11. A knowledge retrieval device, characterized in that: The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the knowledge retrieval method according to any one of claims 1 to 9 is implemented.

12. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the knowledge retrieval method according to any one of claims 1 to 9 is implemented.

Images