An intelligent retrieval method, device, apparatus and storage medium

Abstract

The application discloses an intelligent retrieval method, device and equipment and a storage medium. The method comprises the following steps: determining a problem that needs to be implemented at present as a target problem, and performing index retrieval based on the target problem to obtain documents related to the target problem as candidate documents; performing semantic retrieval on all candidate documents based on the target problem to obtain semantic scores of each candidate document in semantic retrieval, and sorting all candidate documents according to the semantic scores; splicing the target problem with each candidate document respectively to obtain input documents corresponding to each candidate document respectively, inputting each input document into an analysis model to obtain a confidence score output by the analysis model, and re-sorting all candidate documents according to the confidence score and outputting; wherein the analysis model is obtained by training a plurality of questions and corresponding answers in advance. The retrieval accuracy can be effectively improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information retrieval, and more particularly to an intelligent retrieval method, device, equipment and storage medium. BACKGROUND

[0002] The development of computers and Internet technology has led to an exponential increase in the amount of information people are exposed to. For the financial sector, with the growth of banking business and the application of information technology, the number of documents is also growing. In order to effectively manage the growing amount of document data and enable users to accurately and quickly find the knowledge they need from among numerous documents, information retrieval technology has emerged. The current commonly used search method has the problem of low retrieval accuracy. SUMMARY

[0003] The purpose of the present application is to provide an intelligent retrieval method, device, equipment and storage medium, which can effectively improve the retrieval accuracy.

[0004] In order to achieve the above purpose, the present application provides the following technical scheme:

[0005] An intelligent retrieval method, comprising:

[0006] determining that the problem currently required to be implemented for corresponding retrieval is a target problem, and performing index retrieval based on the target problem to obtain documents related to the target problem as candidate documents;

[0007] performing semantic retrieval in all candidate documents based on the target problem to obtain semantic scores of each candidate document in semantic retrieval, and sorting all candidate documents according to the semantic scores;

[0008] splicing the target problem with each candidate document to obtain input documents corresponding to each candidate document respectively, inputting each input document into an analysis model to obtain a confidence score output by the analysis model, and re-sorting all candidate documents according to the confidence score and outputting; wherein the analysis model is obtained by pre-training a plurality of questions and corresponding answers.

[0009] Preferably, before performing index retrieval based on the target problem, the method further comprises:

[0010] obtaining various types of documents, and uniformly encoding the obtained various types of documents;

[0011] performing word segmentation processing on the documents after uniform encoding to obtain all words contained in the documents, and removing invalid characters in the obtained all words;

[0012] constructing an index of the documents by means of a tool Luence from the all words after removing the invalid characters.

[0013] Preferably, the index retrieval based on the target question obtains candidate documents, comprising:

[0014] The target question is segmented to obtain retrieval words, and each retrieval word is assigned a corresponding word weight;

[0015] The Luence retrieval is performed using the retrieval words and corresponding word weights to obtain documents related to the target question as candidate documents.

[0016] Preferably, each retrieval word is assigned a corresponding word weight, comprising:

[0017] The target question and all retrieval words are input into a word weight model to obtain the word weight of each retrieval word output by the word weight model; wherein the word weight model is obtained by pre-training using a plurality of questions and the word weight of each word in each question.

[0018] Preferably, after the index retrieval based on the target question obtains candidate documents, it further comprises:

[0019] The tf-idf is used to calculate the score of each candidate document for the target question, and the index retrieval score corresponding to each candidate document is calculated, and all candidate documents are sorted according to the index retrieval score.

[0020] Preferably, the semantic retrieval is performed based on the target question in all candidate documents, comprising:

[0021] The target question is spliced with each candidate document to obtain an input document corresponding to each candidate document, and each input document is input into a semantic retrieval model to obtain a semantic score output by the semantic retrieval model; wherein the semantic retrieval model is obtained by pre-training using a plurality of questions, a plurality of corresponding documents, and the semantic score between any question and any document.

[0022] Preferably, after each input document is input into the parsing model, it further comprises:

[0023] The content segment in the corresponding input document that can be used as the answer to the target question is obtained, and the content segment is returned.

[0024] An intelligent retrieval device, comprising:

[0025] An index retrieval module is configured to determine a question that needs to be implemented for corresponding retrieval as a target question, and perform index retrieval based on the target question to obtain documents related to the target question as candidate documents;

[0026] The semantic retrieval module is configured to: perform semantic retrieval on the target question in all candidate documents to obtain semantic scores of the candidate documents in the semantic retrieval, and sort all the candidate documents according to the semantic scores;

[0027] The deep analysis module is configured to: splice the target question with each candidate document respectively to obtain input documents corresponding to the candidate documents respectively, input each input document into an analysis model to obtain a confidence score output by the analysis model, and re-sort all the candidate documents according to the confidence scores and output; wherein the analysis model is trained by using a plurality of questions and corresponding answers in advance.

[0028] An intelligent retrieval device comprises:

[0029] A memory configured to store a computer program;

[0030] A processor configured to implement the steps of the intelligent retrieval method according to any one of the above when executing the computer program.

[0031] A computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the intelligent retrieval method according to any one of the above.

[0032] The application provides an intelligent retrieval method, device and equipment and a storage medium. The method comprises the following steps: determining a problem that needs to be implemented currently as a target problem, and performing index retrieval based on the target problem to obtain documents related to the target problem as candidate documents; performing semantic retrieval on all candidate documents based on the target problem to obtain semantic scores of the candidate documents in semantic retrieval, and sorting all candidate documents according to the semantic scores; splicing the target problem with each candidate document to obtain input documents corresponding to the candidate documents respectively, inputting each input document into an analysis model to obtain a confidence score output by the analysis model, and re-sorting all candidate documents according to the confidence scores and outputting. The analysis model is obtained by training a plurality of questions and corresponding answers in advance. For a problem that needs to be implemented, the application retrieves candidate documents related to the problem from all documents through index retrieval, obtains semantic scores representing the degree to which each candidate document meets the intention behind the problem through semantic retrieval, sorts all candidate documents based on the semantic scores, and reanalyzes the problem and each candidate document based on the analysis model to obtain a confidence score representing the possibility of containing the corresponding answer of the problem in each candidate document, and re-sorts all candidate documents based on the confidence scores. It can be seen that the application can preliminarily retrieve candidate documents related to the problem through index retrieval, and then obtain the degree to which each candidate document meets the intention behind the problem and the possibility of containing the corresponding answer of the problem through semantic retrieval and the analysis model, and sort all candidate documents based on the information obtained through semantic retrieval and the analysis model, so that relevant personnel can view the documents that are most likely to meet the intention behind the problem and most likely to contain the corresponding answer of the problem based on the returned candidate documents, thereby improving the retrieval accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can be obtained without creative labor based on the provided drawings.

[0034] Figure 1 A flowchart of an intelligent retrieval method provided by the embodiment of the present application;

[0035] Figure 2 An implementation schematic diagram of index retrieval based on a word weight model in an intelligent retrieval method provided by the embodiment of the present application;

[0036] Figure 3An implementation schematic diagram of semantic retrieval based on a semantic retrieval model in an intelligent retrieval method provided by an embodiment of the present application is shown in the figure.

[0037] Figure 4 An implementation schematic diagram of deep analysis based on an analysis model in an intelligent retrieval method provided by an embodiment of the present application is shown in the figure.

[0038] Figure 5 A structure schematic diagram of an intelligent retrieval device provided by an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the protection scope of the present application.

[0040] Please refer to Figure 1 The figure shows a flowchart of an intelligent retrieval method provided by an embodiment of the present application, which can specifically include:

[0041] S11: Determine the problem that needs to be implemented currently as a target problem, and perform index retrieval based on the target problem to obtain all documents related to the target problem as candidate documents.

[0042] The technical solutions provided by the present application can be applied in the financial field, or other fields according to actual needs, which are all within the protection scope of the present application. When a user needs to retrieve the content related to a certain problem (or the answer to the certain problem), the problem can be input. After receiving the problem input by the user, the intelligent retrieval device (the execution subject of the intelligent retrieval method provided by the embodiment of the present application) performs index retrieval in all documents based on the problem, and then determines that all the documents retrieved by the index retrieval are candidate documents, and the candidate documents can be added to the document set to facilitate the subsequent retrieval and use of the candidate documents, and other operations. In addition, in order to be able to perform index retrieval on all documents, the embodiment of the present application can construct the corresponding index for all documents in advance. The index is consistent with the corresponding concept in the prior art, and its function is equivalent to the table of contents of a book, which will not be described in detail here.

[0043] S12: Perform semantic retrieval in all candidate documents based on the target problem, obtain semantic scores of each candidate document in the semantic retrieval, and sort all candidate documents according to the semantic scores.

[0044] In the index retrieval based on the user input question, the index retrieved document is a candidate document related to the question. After obtaining the candidate document, the embodiment of the present application continues to implement corresponding semantic retrieval in all candidate documents based on the question to obtain semantic scores of each candidate document for the question through semantic retrieval; specifically, the semantic retrieval of the question is implemented in the candidate document, that is, the context of each word in the candidate document is understood to find the content more in line with the intention behind the question, and the higher the semantic score, the more the corresponding candidate document is in line with the intention behind the question. After obtaining the corresponding semantic score by performing semantic retrieval on each candidate document, the embodiment of the present application can sort all candidate documents in order of high to low semantic score (the candidate document with the higher semantic score is located in the front position), so that when all candidate documents are returned to the user, the user can preferentially view the content most in line with the intention behind the corresponding question, thereby improving the accuracy of the query result.

[0045] S13: splice the target question with each candidate document respectively to obtain input documents corresponding to each candidate document respectively, input each input document into the parsing model to obtain the confidence score output by the parsing model, and re-sort all candidate documents according to the confidence score and output; wherein the parsing model is obtained by pre-training a plurality of questions and corresponding answers.

[0046] After sequentially implementing index retrieval and semantic retrieval, the embodiment of the present application will also perform reading comprehension on each candidate document; specifically, the embodiment of the present application deeply analyzes the semantics of the candidate document and the relationship between the candidate document and the question through the parsing model to obtain the confidence score of the segment in the candidate document that can be the answer to the question (that is, the confidence score of the corresponding candidate document), and the higher the confidence score, the more likely the segment in the corresponding candidate document is the answer to the question, and then re-sort each candidate document based on the order of high to low confidence score of each candidate document (the candidate document with the higher confidence score is located in the front position), so that when all candidate documents are returned to the user, the user can preferentially view the content most likely to contain the answer to the corresponding question, thereby further improving the accuracy of the query result. Wherein, a plurality of questions and answers corresponding to each question can be obtained in advance, and then the parsing model is trained using these questions and answers.

[0047] This application, in its embodiments, addresses a specific problem requiring retrieval by first retrieving candidate documents related to the problem from all documents using an index retrieval system. Then, semantic retrieval yields a semantic score indicating the degree to which each candidate document matches the underlying intent of the problem. Based on these semantic scores, all candidate documents are ranked. Next, a deep analysis of the problem and each candidate document is performed using a parsing model to obtain a confidence score indicating the probability that each candidate document contains the answer to the problem. Finally, all candidate documents are ranked again based on these confidence scores. Therefore, this application can initially retrieve candidate documents related to the problem through an index retrieval system, then sequentially use semantic retrieval and a parsing model to derive the degree to which each candidate document matches the underlying intent of the problem and the probability of containing the answer. Finally, based on the information obtained from the semantic retrieval and parsing model, the candidate documents are ranked. This ensures that relevant users can prioritize viewing documents that are relevant to the problem, most likely to match the underlying intent of the problem, and most likely to contain the answer, effectively improving retrieval accuracy.

[0048] An intelligent retrieval method provided in this embodiment of the invention may further include, before performing index retrieval based on a target question:

[0049] Acquire various types of documents and assign them a unified encoding.

[0050] After the document has been uniformly encoded, it is segmented to obtain all the words contained in the document, and invalid characters are removed from all the obtained words;

[0051] The entire list of words, after removing invalid characters, is used to build an index of the document using the Luence tool.

[0052] It should be noted that, in order to achieve document index retrieval, this application embodiment requires data preprocessing to construct the corresponding retrieval. Specifically, this application embodiment establishes a searchable data structure—an index—for a document knowledge base containing all documents, thereby improving the search response speed of the document knowledge base. Data preprocessing mainly includes: uniformly encoding various types of documents in the document knowledge base, such as converting the text format of various documents (e.g., Unicode-encoded text format) to UTF-8 text format through encoding and decoding; after completing the uniform encoding, performing word segmentation on the text of these documents to obtain all the words contained in each document, and removing invalid characters (such as stop words and punctuation marks); finally, using the tool Lucene to build an index of these processed word information. Lucene is an open-source search technology with the same underlying concept and meaning as its counterpart in existing technologies. The Lucene tool specifically uses an inverted index to build the index, recording the document set corresponding to each word, thereby facilitating fast searching during retrieval.

[0053] The intelligent retrieval method provided by the embodiment of the application can comprise the following steps.

[0054] The target question is processed by word segmentation to obtain to-be-retrieved words, and each to-be-retrieved word is assigned a corresponding word weight; and the to-be-retrieved words and the corresponding word weights are used to perform Lucene retrieval to obtain documents related to the target question, which are all candidate documents.

[0055] The target question is processed by word segmentation to obtain to-be-retrieved words, and each to-be-retrieved word is assigned a corresponding word weight; and the to-be-retrieved words and the corresponding word weights are used to perform Lucene retrieval to obtain documents related to the target question, which are all candidate documents.

[0056] The target question and all to-be-retrieved words are input into a word weight model to obtain word weights of each to-be-retrieved word output by the word weight model; the word weight model is obtained by pre-training a plurality of questions and word weights of each word in each question.

[0057] After the candidate documents are obtained by index retrieval based on the target question, the following steps can be further included.

[0058] The scores of each candidate document with respect to the target question are calculated by tf-idf, and the index retrieval scores corresponding to each candidate document are calculated respectively, and all candidate documents are sorted according to the index retrieval scores.

[0059] After the index is established, corresponding index retrieval (i.e., Lucene retrieval) can be performed in the document knowledge base; specifically, after receiving a question input by a user, the question can be preprocessed, including processing the question by word segmentation to obtain all words contained in the question as to-be-retrieved words, and assigning each to-be-retrieved word a corresponding word weight; then each to-be-retrieved word and the corresponding word weight are used to perform Lucene retrieval to obtain a document set related to the question, including extracting all documents containing one or more to-be-retrieved words among all to-be-retrieved words as candidate documents (this process is called recall), and calculating the scores of each candidate document with respect to the question by tf-idf, and calculating the index retrieval scores corresponding to each document, wherein the higher the index retrieval score is, the higher the relevance of the corresponding candidate document to the question is; finally, the candidate documents are sorted based on the index retrieval scores of each candidate document in the document set (the higher the index retrieval score is, the higher the position is), and the retrieval result of Lucene is returned.

[0060] It should be noted that the word weight distribution can include two parts: 1) user-defined word weight, if the user configures the word weight of a certain search word, the user-defined word weight is used; 2) dynamically allocate word weight, for the search word whose word weight is not configured by the user, a word weight is dynamically allocated to each search word by the word weight model. The word weight model is obtained by pre-training a plurality of questions and the word weight of each word in each question. In a specific implementation, the word weight model is fine-tuned based on the open source pre-training model BERT, and the open source question and answer general data (such as dureader) is used to map each word in the document to a 0-1 value as the word weight of the word. The embodiment of the application can effectively improve the recall rate of the query by the dynamic word weight allocation strategy. The meaning of dynamic word weight is that the same word has different weight values in different contexts; for example, "Apple mobile phone is very good" and "Apple mobile phone image processor is very strong", the weight of "Apple mobile phone" in the former sentence should be higher, and the weight of "image processor" in the latter sentence should be higher. In addition, if there is user click data, the user click data (user click document behavior corresponding data) can be fed back to the word weight model for corresponding optimization and adjustment to improve the relevance of the result and the user question. Specifically, first, the user question and the user clicked document are obtained, and then the question and the user clicked document are segmented, and the overlapping words between the two are found. These words are considered to have higher word weights. Finally, the word weight model learns the mapping relationship between these words with higher weights and word weights. The implementation schematic diagram of the index retrieval based on the word weight model can be as shown in Figure 2

[0061] In addition, the problem input by the user can be input to the quality inspection terminal, so that the corresponding personnel of the quality inspection terminal can correct and rewrite the problem when the problem input by the user is wrong. Of course, other correction methods can also be set according to actual needs, which are all within the protection scope of the application.

[0062] The intelligent retrieval method provided by the embodiment of the application can include the following steps:

[0063] The target question is spliced with each candidate document to obtain an input document corresponding to each candidate document, and each input document is input into a semantic retrieval model to obtain a semantic score output by the semantic retrieval model; wherein the semantic retrieval model is obtained by pre-training a plurality of questions, a plurality of corresponding documents, and a semantic score between any question and any document.

[0064] ​This application embodiment can pre-acquire multiple questions, the documents corresponding to each question, and semantic scores between any question and its corresponding document. Then, it uses this acquired information to train a semantic retrieval model, thereby enabling fast and accurate semantic retrieval of candidate documents. Specifically, the semantic retrieval model can be the Bert-Search model. The Bert-Search model is based on the open-source pre-trained language model BERT and fine-tuned using open-source question-answering datasets (such as Dureader) as training data. During training, questions are concatenated with relevant documents and input into the semantic retrieval model to obtain a higher semantic score, while questions are concatenated with irrelevant documents and input into the semantic retrieval model to obtain a lower semantic score. This method allows training a model capable of judging the relevance between user intent and document relevance. To optimize model performance, this embodiment can introduce adversarial training to increase noise during training and improve generalization ability. It can also append keywords to the document to further enhance semantic capture. Keyword extraction can be performed using the TF-IDF algorithm: first, the document is segmented to obtain all its words; then, the TF-IDF algorithm is used to calculate the TF*IDF value of each word; finally, the top five keywords with the highest values ​​are selected. Furthermore, to improve inference speed, this embodiment can prune the model by reducing its dimensionality (e.g., reducing it to half the commonly used model dimensionality), thereby reducing the number of model parameters and improving inference speed.

[0065] When a semantic score is required, this embodiment of the application can concatenate the question with any candidate document and input it into the semantic retrieval model. The semantic retrieval model provides a semantic score to determine the relevance between the input question and the candidate document. Finally, the candidate documents are reordered based on the semantic score. Additionally, if user click data (data corresponding to user clicks on documents) is available, it can also be used to optimize the semantic retrieval model. The specific optimization method is similar to that of the word weight model: first, relevant documents between the user's question and the user's clicks are selected; then, the model is trained again according to the training method of the semantic retrieval model. This allows for further improvement of the semantic retrieval model's semantic understanding of documents and user questions through optimization of user click behavior. A corresponding schematic diagram of semantic retrieval based on the semantic retrieval model can be shown below. Figure 3 As shown.

[0066] In a specific implementation, after the index retrieval score is used to sort all candidate documents, the semantic score is used to sort all candidate documents again, and for candidate documents with the same semantic score, the higher the index retrieval score, the higher the position, thereby obtaining all subsequent documents sorted by the comprehensive index retrieval score and semantic score; after the semantic score is used to sort all candidate documents, the confidence score is used to sort all candidate documents again, and for candidate documents with the same confidence score, the higher the semantic score, the higher the position, thereby obtaining all subsequent documents sorted by the comprehensive index retrieval score, semantic score and confidence score.

[0067] The intelligent retrieval method provided by the embodiment of the application can further include the following steps after the input documents are input into the analysis model:

[0068] The content segment capable of being an answer to the target question in the corresponding input document output by the analysis model is obtained, and the content segment is returned.

[0069] The analysis model in the embodiment of the application is a task with a finer granularity than semantic retrieval. The analysis model not only requires the model to determine whether the document is relevant, but also requires the model to give an answer to the question (if there is an answer). Therefore, the reading comprehension requires the model to have a deeper semantic understanding of the document and the question. The analysis model is also based on the Bert model and is trained using open-source reading comprehension data. The specific training method is to input the question and the relevant document into the analysis model, and then let the analysis model extract a segment in the document that can answer the question. At the same time, the model also gives a confidence score of the segment, and then uses the score to sort the relevance of the document. Therefore, after the input document is input into the analysis model, the embodiment of the application can also obtain the segment capable of being an answer to the question in the corresponding input document output by the analysis model, and return the document set and the segments at the same time, thereby further improving the retrieval accuracy. As described above, the click data of the user can also be fed back to the analysis model for adjusting and optimizing the model. The implementation schematic diagram of the deep analysis based on the analysis model can be as shown in Figure 4 The reading comprehension model in the figure is the analysis model.

[0070] In a specific implementation, the intelligent retrieval method provided by the embodiment of the application can specifically include the following steps:

[0071] Step 101, data preprocessing: indexing the document knowledge base to improve search response speed.

[0072] Step 102, Lucene retrieval: first, word segmentation is performed on the user question, and word weight is assigned; then, the word segmentation result and the word weight are subjected to Lucene retrieval to obtain a document set related to the user question; and finally, the documents are sorted based on the tf-idf score of each document in the document set, and the retrieval result of Lucene is returned.

[0073] Step 103, Bert-Search semantic retrieval: the candidate documents obtained in step 102 are reordered by using semantic retrieval.

[0074] Step 104, reading comprehension: the document content and the user question are subjected to a parsing model, and the semantics of the document and the relationship between the document and the question are deeply parsed, then a segment that can answer the question is extracted from the document as the answer to the question, and the candidate documents are reordered according to the confidence of the answer.

[0075] The application is a knowledge search engine suitable for the characteristics of industry knowledge, and realizes intelligent retrieval; through active prompt of search content, automatic error correction, pinyin recognition, intelligent guidance and the like, the user is assisted to perform accurate search, and the user is assisted to intuitively understand the knowledge content. Compared with a simple directory selection mode, the present technique greatly reduces the difficulty of searching knowledge in a field for a non-professional, and enables the user to search knowledge in a natural language manner; compared with traditional database-based fuzzy matching of keywords, not only is the relevance of the retrieval result to the user question improved, but also the index-based search ensures a high query response speed in the case of a large amount of document data; in the process of searching knowledge, the user is interacted with in a manner of collecting user click behavior and the like, the corresponding model is reasonably adjusted, the candidate documents are screened multiple times, and the search accuracy is improved; the semantic search technology can greatly improve the accuracy of top1 retrieval.

[0076] The embodiment of the application further provides an intelligent retrieval device, as shown in the accompanying drawings, which can comprise: Figure 5

[0077] The index retrieval module 11 is configured to determine a question currently required to be implemented as a target question, and perform index retrieval based on the target question to obtain documents related to the target question as candidate documents.

[0078] The semantic retrieval module 12 is configured to perform semantic retrieval in all candidate documents based on the target question, obtain semantic scores of the candidate documents in the semantic retrieval, and sort all candidate documents according to the semantic scores.

[0079] ​The depth analysis module 13 is configured to splice the target question with each candidate document respectively to obtain an input document corresponding to each candidate document, input each input document into the analysis model to obtain a confidence score output by the analysis model, and re-sort all candidate documents according to the confidence score and output the same.

[0080] The embodiment of the present application further provides an intelligent retrieval device, which can comprise:

[0081] a memory configured to store the computer program;

[0082] a processor configured to execute the computer program to implement the steps of any one of the intelligent retrieval methods.

[0083] The embodiment of the present application further provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of any one of the intelligent retrieval methods.

[0084] It should be noted that the related parts of the intelligent retrieval device, the intelligent retrieval equipment and the storage medium provided by the embodiment of the present application are described in detail in the corresponding part of the intelligent retrieval method provided by the embodiment of the present application, and will not be described here. In addition, the parts of the above technical solutions provided by the embodiment of the present application which are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail, so as not to be too verbose.

[0085] The above description of the disclosed embodiments enables a person skilled in the art to implement or use the present application. Various modifications to the embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An intelligent retrieval method, characterized in that, The method comprises the following steps: acquiring various types of documents, and uniformly encoding the acquired various types of documents; performing word segmentation processing on the documents after the uniform encoding is completed, obtaining all words contained in the documents, and removing invalid characters in the obtained all words; building an index of the documents by a tool Luence after the invalid characters are removed from the obtained all words; determining a problem currently required to be implemented as a target problem, and performing index retrieval based on the target problem to obtain documents related to the target problem as candidate documents; wherein the index retrieval based on the target problem to obtain the candidate documents comprises: performing word segmentation processing on the target problem to obtain words to be retrieved, and inputting the target problem and all words to be retrieved into a word weight model to obtain word weights of each word to be retrieved output by the word weight model; and using the words to be retrieved and the corresponding word weights to perform Luence retrieval to obtain the documents related to the target problem as the candidate documents; wherein the word weight model is obtained by pre-training using a plurality of problems and word weights of each word in each problem; and the word weight model is further used to dynamically optimize corresponding parameters in the model based on behavior corresponding data of a user clicking a document; performing semantic retrieval on the target problem in all candidate documents to obtain semantic scores of each candidate document in the semantic retrieval, and sorting all candidate documents according to the semantic scores; splicing the target problem with each candidate document respectively to obtain input documents corresponding to each candidate document respectively, inputting each input document into a parsing model to obtain a confidence score output by the parsing model, and re-sorting all candidate documents according to the confidence score and outputting; wherein the parsing model is obtained by pre-training using a plurality of problems and corresponding answers; wherein the semantic retrieval and the parsing model are sequentially executed, and the confidence score is obtained based on the semantic score and deep semantic analysis of the parsing model; wherein after the candidate documents are obtained by the index retrieval based on the target problem, the method further comprises: calculating a score of each candidate document for the target problem as an index retrieval score by tf-idf respectively, and sorting all candidate documents according to the index retrieval score.

2. The method of claim 1, wherein, performing semantic retrieval on the target problem in all candidate documents comprises: splicing the target problem with each candidate document respectively to obtain input documents corresponding to each candidate document respectively, inputting each input document into a semantic retrieval model to obtain a semantic score output by the semantic retrieval model; wherein the semantic retrieval model is obtained by pre-training using a plurality of problems, a plurality of corresponding documents, and semantic scores between any problem and any document.

3. The method of claim 2, wherein, after inputting each input document into the parsing model, the method further comprises: obtaining a content segment in the corresponding input document output by the parsing model which can be an answer to the target problem, and returning the content segment.

4. An intelligent retrieval apparatus, characterized by comprising: The method comprises the following steps: acquiring various types of documents, and uniformly encoding the acquired various types of documents; The document after the unified coding is processed by word segmentation to obtain all the words contained in the document, and the invalid characters in the obtained all the words are removed; The all the words obtained after removing the invalid characters are used to build an index of the document by a tool Luence; An index retrieval module is configured to determine that a problem to be implemented for corresponding retrieval is a target problem, and perform index retrieval based on the target problem to obtain documents related to the target problem as candidate documents; wherein the index retrieval module is specifically configured to perform word segmentation on the target problem to obtain words to be retrieved, and input the target problem and all the words to be retrieved into a word weight model to obtain word weights of each word to be retrieved output by the word weight model; and use the words to be retrieved and the corresponding word weights to perform Luence retrieval to obtain documents related to the target problem as candidate documents; The word weight model is pre-trained by using a plurality of problems and word weights of each word in each problem; and the word weight model is further configured to dynamically optimize corresponding parameters in the model based on behavior corresponding data of users clicking the documents; A semantic retrieval module is configured to perform semantic retrieval on all the candidate documents based on the target problem to obtain semantic scores of each candidate document in the semantic retrieval, and sort all the candidate documents according to the semantic scores; A deep analysis module is configured to splice the target problem with each candidate document respectively to obtain input documents corresponding to each candidate document respectively, input each input document into an analysis model to obtain a confidence score output by the analysis model, and re-sort all the candidate documents according to the confidence score and output; wherein the analysis model is pre-trained by using a plurality of problems and corresponding answers; The semantic retrieval and the analysis model are sequentially executed, and the confidence score is obtained based on the semantic score and deep semantic analysis of the analysis model; After obtaining the candidate documents by index retrieval based on the target problem, the method further includes: calculating a score of each candidate document for the target problem by tf-idf as an index retrieval score, and sorting all the candidate documents according to the index retrieval score.

5. An intelligent retrieval device, characterized by The method comprises: a memory for storing a computer program; a processor for executing the computer program to implement the steps of the intelligent retrieval method according to any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the steps of the intelligent retrieval method according to any one of claims 1 to 3.

Images