Document reordering method and device and electronic equipment

Abstract

The invention discloses a document reordering method and device and electronic equipment, and belongs to the technical field of artificial intelligence. The method comprises the steps that in response to received query information, at least one candidate document matched with the query information is obtained, and the candidate document comprises at least one candidate sentence; dividing the query information into a plurality of query elements, and for each query element, based on the similarity between each candidate sentence in the at least one candidate document and the query element, obtaining k first candidate sentences of which the similarity with the query element is greater than a preset value in all candidate sentences in the at least one candidate document; based on the number of the first candidate sentences included in each candidate document, obtaining a correlation index of each candidate document and the query information; and according to the correlation index of each candidate document, reordering each candidate document. Through the method, the calculation complexity of document reordering can be reduced, and the efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a document reordering method, device and electronic device. Background Art

[0002] Re-ranking technology plays a vital role in the entire process of Retrieval Augmented Generation (RAG).

[0003] In the related art, a deep learning-based method can be used to reorder documents, but this type of reordering method has high computational complexity and low efficiency. Summary of the invention

[0004] The present application provides a document reordering method, device and electronic device, which can at least solve the problems of high computational complexity and low efficiency of document reordering methods in related technologies.

[0005] In order to solve the above technical problems, this application is implemented as follows: In a first aspect, an embodiment of the present application provides a document reordering method, the method comprising: in response to received query information, obtaining at least one candidate document matching the query information, wherein the candidate document includes at least one candidate sentence; dividing the query information into multiple query elements, and for each query element, based on the similarity between each candidate sentence in the at least one candidate document and the query element, obtaining k first candidate sentences whose similarity to the query element is greater than a preset value among all candidate sentences in the at least one candidate document, wherein k is an integer greater than 0; based on the number of the first candidate sentences included in each of the candidate documents, obtaining a relevance index of each of the candidate documents and the query information; and reordering each of the candidate documents according to the relevance index of each of the candidate documents.

[0006] In a second aspect, a document reordering device is provided, which includes: a first acquisition module, used to obtain at least one candidate document matching the query information in response to received query information, wherein the candidate document includes at least one candidate sentence; a second acquisition module, used to divide the query information into multiple query elements, and for each query element, based on the similarity between each candidate sentence in the at least one candidate document and the query element, obtain k first candidate sentences whose similarity with the query element is greater than a preset value among all candidate sentences in the at least one candidate document, wherein k is an integer greater than 0; a third acquisition module, used to obtain a relevance index of each candidate document with the query information based on the number of the first candidate sentences included in each candidate document; and a reordering module, used to reorder each candidate document according to the relevance index of each candidate document.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the programs or instructions are executed by the processor, the steps of the method described in the first aspect above are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect above are implemented.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer executes the steps of the method described in the first aspect above.

[0010] The technical solution provided by this application may have the following beneficial effects: In an embodiment of the present application, in response to received query information, at least one candidate document matching the query information can be obtained; then the query information is divided into multiple query elements, and for each query element, based on the similarity between each candidate sentence in at least one candidate document and the query element, k first candidate sentences in the candidate sentences in at least one candidate document whose similarity with the query element is greater than a preset value are obtained; then, based on the number of first candidate sentences included in each candidate document, a correlation index between each candidate document and the query information can be obtained; and each candidate document can be reordered according to the correlation index of each candidate document. In the above manner, not only the computational complexity of document reordering can be reduced, but also the efficiency can be improved.

[0011] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0013] Figure 1 A schematic diagram of a process flow of a document reordering method provided in an embodiment of the present application is shown; Figure 2 A schematic diagram of a document reordering method provided in an embodiment of the present application is shown; Figure 3 A schematic diagram of the structure of a document reordering device provided in an embodiment of the present application is shown; Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown; Figure 5 A schematic structural diagram of another electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0014] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0015] Re-ranking technology plays a vital role in the whole process of Retrieval Augmented Generation (RAG). In the most primitive RAG method, a large amount of context may be retrieved, but not all of it is relevant to the question. Re-ranking technology can rearrange the order of documents and filter them, excluding irrelevant or unimportant documents and putting relevant documents at the front, thereby improving the accuracy of the RAG system.

[0016] Re-ranking algorithms in related technologies can generally be divided into two types: one is a method based on deep learning, such as neural networks, decision trees, bidirectional encoder representations from transformers (BERT), etc. The other is a method based on a relevance model, which analyzes the interactive features between the user's questions and candidate documents, evaluates the relevance between them, and sorts them according to the relevance score, such as BM25 (full name Best Matching 25, an algorithm used for information retrieval and text mining). Methods based on deep learning generally use cross encoders to score and sort documents based on the semantic information of documents and queries. The typical BERT algorithm and other transformer-based methods (a model that uses an attention mechanism to increase the speed of model training) have the problem of limiting 512 tokens (usually the smallest unit of input text), which cannot cover the full text of longer documents, and these algorithms have high complexity and low computational efficiency. Especially when processing large-scale data sets, it consumes a lot of computing resources and time. The BM25 algorithm is based on term frequency (TF) and inverse document frequency (IDF), and also introduces document length information to calculate the relevance between document (D) and query (Q). However, the BM25 algorithm may have low scores for long documents and short queries. It also does not consider the relevance between terms, which may lead to inaccurate scores.

[0017] The re-ranking algorithms in related technologies have the following main disadvantages: (1) Although deep learning-based methods, such as neural networks, decision trees, and BERT, can effectively combine the semantic information of documents and queries for scoring and ranking, they are often limited by the length of the text (such as the 512 tokens limit of transformer-based methods such as BERT) and cannot fully cover long documents. Moreover, these algorithms have high computational complexity and low efficiency, and consume a lot of computing resources and time, especially when processing large-scale data.

[0018] (2) Methods based on relevance models, such as BM25, can evaluate relevance by analyzing the interaction features between queries and documents, but they may have low scores when dealing with long documents and short queries. At the same time, they ignore the correlation between terms, which may lead to inaccurate scoring.

[0019] Figure 1A schematic diagram of a document reordering method provided by an exemplary embodiment of the present application is shown. The method can be executed by an electronic device, which can be a terminal such as a mobile phone or a computer. Figure 1 As shown, the method mainly includes the following steps: S101: In response to received query information, obtaining at least one candidate document matching the query information.

[0020] The candidate document includes at least one candidate sentence.

[0021] In an embodiment of the present application, at least one candidate document matching the query information can be obtained in response to the received query information. In practical applications, the query information can be a query document or a query statement, which is not specifically limited in the embodiment of the present application. At least one candidate document is a search result that has been retrieved and matches the query information. In practical applications, the number of candidate documents can be n, where n is an integer greater than 1. The specific candidate document retrieval process can adopt a Boolean model (a simple retrieval model based on set theory and Boolean algebra) or a vector space model (VSM), which is not specifically limited in the embodiment of the present application. The document reordering method in the embodiment of the present application mainly reorders the candidate documents after the initial sorting, and the main purpose is to exclude irrelevant or unimportant candidate documents and put the candidate documents with high relevance in front, thereby improving the accuracy of the retrieval system.

[0022] In the embodiment of the present application, the query information can be used To indicate that at least one candidate document can be In practical applications, the candidate document can be divided into multiple candidate sentences. The specific division method is not specifically limited in the embodiment of the present application, and the number of candidate sentences is not specifically limited. Let it be the set of all candidate sentences in D. Assume D, is the set of all candidate sentences in d, is a sentence in d, for In the embodiment of the present application, the method for generating the sentence vector is not specifically limited in the embodiment of the present application.

[0023] S102: Divide the query information into multiple query elements, and for each query element, based on the similarity between each candidate sentence in the at least one candidate document and the query element, obtain k first candidate sentences whose similarity with the query element is greater than a preset value from among all candidate sentences in the at least one candidate document.

[0024] where k is an integer greater than 0.

[0025] In the embodiments of the present application, the query information q can be divided into multiple query elements. The number of query elements and the specific division method are not specifically limited in the embodiments of the present application. In practical applications, if the query information is in the form of a query statement, it can be divided according to specific circumstances. For example, words without actual semantics such as "de" (的) or "a" (啊) can be discarded. If the query information is in the form of a query document, it can be divided according to specific semantics, and a complete sentence is preferably divided into one query element as much as possible. In practical applications, a certain complete syntactic analysis technology can be used to assist in the division of query information, and a sentence or word with complete semantics should be avoided from being split in half as much as possible. The embodiments of the present application do not make specific limitations.

[0026] In the embodiments of the present application, is used to represent all query elements. If the query information is in the form of a query document, can be the set of all query sentences in q, is a sentence in q, is the corresponding sentence vector. In practical applications, different methods may generate vectors with different dimensions and meanings. In the embodiments of the present application, the method for generating the vector of the query element can be consistent with the method for generating the vector of the candidate sentence, which is convenient for calculating their similarity.

[0027] In the embodiments of the present application, for each query element, based on the similarity between each candidate sentence in at least one candidate document and the query element, k first candidate sentences with a similarity greater than a preset value among all candidate sentences in at least one candidate document are obtained. In this way, for each query element, k first candidate sentences with a similarity greater than the preset value among all candidate sentences are obtained. That is, for each query element, k first candidate sentences with a similarity greater than the preset value can be obtained. k is less than the total number of candidate sentences, which can reduce the amount of calculation, improve the calculation efficiency, and can also screen out the first candidate sentences with a similarity greater than the preset value in at least one candidate document, which can be used to further obtain the similarity between the query information and the candidate document, facilitating the re-ranking of the candidate documents. In practical applications, the method for calculating the similarity between each candidate sentence and the query element can adopt the modified cosine similarity. The embodiments of the present application do not make specific limitations.

[0028] In an optional implementation, the obtaining, based on the similarity between each of the candidate sentences in the at least one candidate document and the query element, k first candidate sentences having a similarity with the query element greater than a preset value among all the candidate sentences in the at least one candidate document may include the following steps: Step 1, calculate the adjusted cosine similarity between the query element and each candidate sentence in the at least one candidate document; in an embodiment of the present application, the adjusted cosine similarity between the query element and each candidate sentence in the at least one candidate document can be calculated. Adjusted cosine similarity (ACS) is an improved cosine similarity calculation method used in text mining, information retrieval and other fields. The mean of the document (or vector) can be subtracted from the value of each dimension to eliminate the deviation caused by the mean offset and the difference in document length (or vector size). The larger the result, the higher the similarity. The specific calculation formula is as follows:

[0029] Where n is the dimension of the vector, is the vector representation of the query element, is the vector representation of the candidate sentence, For vector The value in the jth dimension, For vector The value in the jth dimension, For vector The mean of all dimensions, For vector The mean of all dimensions.

[0030] Step 2: sort all candidate sentences in the at least one candidate document in descending order of the modified cosine similarity, and obtain the first k candidate sentences that are ranked first.

[0031] In the embodiment of the present application, the modified cosine similarity can be ordered from large to small for each , sort all candidate sentences in at least one candidate document, and the sorting result can be recorded as , and then get the first k candidate sentences ranked at the top k is a parameter, which can be set based on experience and is not specifically limited in the present embodiment. Recorded as All The set of corresponding first candidate sentences.

[0032] In practical applications, since the length of the document and the length of the query are usually very different, directly using cosine similarity may lead to inaccurate results. Although traditional cosine similarity can be used to calculate the similarity between the query element and the candidate sentence, when the length difference of the text is large, the longer text may get a higher similarity score because it contains more information, even if the actual semantic similarity between them is not high. And the modified cosine similarity can more accurately measure the semantic similarity between texts by considering the difference in text length. The modified cosine similarity can not only better evaluate the real semantic importance, but also avoid the problem of low scores in long documents and short queries. Therefore, in an embodiment of the present application, the modified cosine similarity can be used to calculate the similarity between the query element and the candidate sentence. By modifying the cosine similarity, the influence of the text length difference can be eliminated and the accuracy of retrieval can be improved.

[0033] In the embodiment of the present application, the modified cosine similarity is used to calculate the similarity between the query element and the candidate sentence. Compared with the deep learning method in the related art, the calculation method is simple, and the calculation complexity is improved, making the calculation faster. This is because the modified cosine similarity mainly relies on basic linear algebra operations, without the need for time-consuming neural network training or complex model reasoning.

[0034] S103: Based on the number of the first candidate sentences included in each of the candidate documents, obtain a correlation index between each of the candidate documents and the query information.

[0035] In an embodiment of the present application, the relevance index of each candidate document and the query information can be obtained based on the number of first candidate sentences included in each candidate document. On the whole, the more first candidate sentences are included in a candidate document, the higher the relevance of the candidate document to the query information. The relevance index can be used to indicate the relevance of each candidate document and the query information, and can effectively characterize the degree of match between the candidate document and the query information. By obtaining the relevance index, the accuracy of the re-ranking can be improved.

[0036] In an optional implementation, the relevance index includes: a query relevance index and a document relevance index; in the embodiment of the present application, the relevance index can be further divided into a query relevance index and a document relevance index to obtain a more detailed and comprehensive relevance degree between the candidate document and the query information.

[0037] In the above optional implementation, the obtaining of the relevance index between each candidate document and the query information based on the number of the first candidate sentences included in each candidate document may include the following steps: Step 1, obtaining the weight of each query element; in the embodiment of the present application, the weight of each query element can be obtained. When calculating the relevance index between the candidate document and the query information, the weight of the query element is considered to improve the accuracy of the re-ranking.

[0038] Step 2, obtaining the query relevance index of each candidate document according to the weight of each query element, the number of query sentences, and whether each candidate document includes the first candidate sentence of each query element; in an embodiment of the present application, the query relevance index of each candidate document can be obtained according to the weight of each query element, the number of query sentences, and whether each candidate document includes the first candidate sentence of each query element. In this way, the query relevance index of the candidate document obtained by comprehensively considering the weight of the query element, the number of query sentences, and whether each candidate document includes the first candidate sentence can more accurately evaluate the degree of match between the candidate document and the query information.

[0039] Step 3, according to the weight of each query element, the number of candidate sentences, and whether the candidate sentence of each candidate document is the first candidate sentence of each query element, obtain the document relevance index of each candidate document. In an embodiment of the present application, the document relevance index of each candidate document can be obtained according to the weight of each query element, the number of candidate sentences, and whether the candidate sentence of each candidate document is the first candidate sentence of each query element. In this way, the document relevance index of the candidate document obtained by comprehensively considering the weight of the query element, the number of candidate sentences, and whether the candidate sentence of each candidate document is the first candidate sentence of each query element can more comprehensively evaluate the relevance between the candidate document and the query information.

[0040] In the embodiment of the present application, whether calculating the query relevance index or the document relevance index, all candidate sentences of each candidate document can be traversed. The method of the embodiment of the present application can avoid the problem of limiting 512 tokens in the typical BERT algorithm and other transformer-based methods in the related art, and can fully cover the full text of longer documents and improve the accuracy of calculation.

[0041] In an optional implementation, for the above step 1, for each query element, obtaining the weight of the query element may include: The weight of the query element is obtained according to the number of all candidate sentences included in the at least one candidate document and the number of times the query element appears in the at least one candidate document.

[0042] In an embodiment of the present application, the weight of the query element can be obtained according to the number of all candidate sentences included in at least one candidate document and the number of times the query element appears in at least one candidate document. Optionally, the calculation formula is as follows:

[0043] in, Represents a query element The weight of represents the number of all candidate sentences included in at least one candidate document, Represents a query element The number of times a value appears in at least one candidate document. log means taking the logarithm of the obtained value.

[0044] In practical applications, if a query element (word / sentence) is more common, the larger the denominator, the smaller the corresponding weight and the closer it is to 0. The reason why 0.5 is added to the denominator is to avoid the denominator being 0 (that is, all candidate documents do not contain the word / sentence). You can also choose to add other values, but try not to be too large or too small. The specific values ​​are not specifically limited in the embodiments of this application.

[0045] In an optional implementation, the query relevance index of each candidate document is obtained according to the weight of each query element, the number of query sentences, and whether each candidate document includes the first candidate sentence of each query element, including: for each candidate document, the query relevance index of the candidate document is calculated using the following formula: :

[0046] in, Represents the query element The weight of is the set of all query elements of the query information, is the set of all candidate sentences of the candidate document d, for The set of first candidate sentences, Indicates that for each , if there is at least one candidate sentence in d Corresponding In the example, the min() function returns 1, if not, it returns 0, and N(q) represents the number of query elements; or, According to the weight of each query element, the number of candidate sentences, and whether the candidate sentence of each candidate document is the first candidate sentence of each query element, the document relevance index of each candidate document is obtained, including: for each candidate document, the document relevance index of the candidate document is calculated using the following formula: :

[0047] in, Represents the query element The weight of is the set of all query elements of the query information, is the set of all candidate sentences of the candidate document d, for The corresponding set of first candidate sentences, Represents each candidate sentence for d ,if exist Corresponding In the example, the min() function returns 1, if not, it returns 0, and N(d) represents the number of candidate sentences in the candidate document d.

[0048] In the embodiment of the present application, when calculating the query relevance QP of the candidate document d, it is possible to iteratively calculate For each query element , the higher its weight is, and if there is a corresponding first candidate sentence in the candidate sentences in d, then the higher the query relevance score of d is.

[0049] In the embodiment of the present application, when calculating the document relevance DP of the candidate document d, it is possible to iteratively calculate All candidate sentences in , confirm whether each candidate sentence is The first candidate sentence corresponding to . The candidate sentence in d is The more there are in , the higher the document relevance score of d.

[0050] S104: reordering the candidate documents according to their relevance index.

[0051] In the embodiment of the present application, each candidate document can be reordered according to its relevance index. The higher the relevance index of a candidate document, the higher the relevance of the candidate document to the query information, and the better the match, and the higher the result of the reordering of the candidate document. In practical applications, reordering the candidate documents according to their relevance index can improve the accuracy of retrieval.

[0052] In an optional implementation, the reordering of the candidate documents according to the relevance index of each candidate document may include the following steps: Step 1: Based on the query relevance index of each candidate document and the document relevance index of the candidate document, obtain the reranking index corresponding to each candidate document; in this embodiment of the application, based on the query relevance index of each candidate document and the document relevance index of the candidate document, obtain the reranking index corresponding to each candidate document In this way, more accurate and comprehensive re-ranking indicators corresponding to the candidate documents can be obtained.

[0053] Optionally, the calculation formula is as follows:

[0054] In practical applications, other formulas may be used for calculation, and the embodiments of the present application do not specifically limit this.

[0055] Step 2: reorder each candidate document based on the reorder index corresponding to each candidate document. In an embodiment of the present application, each candidate document can be reordered based on the reorder index corresponding to each candidate document. In actual applications, candidate documents with large reorder indexes can be placed in front, which helps to improve the user's query experience.

[0056] In an embodiment of the present application, in response to received query information, at least one candidate document matching the query information can be obtained; then the query information is divided into multiple query elements, and for each query element, based on the similarity between each candidate sentence in at least one candidate document and the query element, k first candidate sentences in the candidate sentences in at least one candidate document whose similarity with the query element is greater than a preset value are obtained; then, based on the number of first candidate sentences included in each candidate document, a correlation index between each candidate document and the query information can be obtained; and each candidate document can be reordered according to the correlation index of each candidate document. In the above manner, not only the computational complexity of document reordering can be reduced, but also the efficiency can be improved.

[0057] The document reordering method proposed in the embodiment of the present application mainly consists of four steps.

[0058] The first step is to calculate the modified cosine similarity between all query elements in the query information and all candidate sentences in the candidate documents, and rank the candidate sentences in the candidate documents for each query element; The second step is to calculate the query relevance index of the candidate document for each query element. The third step is to calculate the document relevance index with the query element for each candidate document. The fourth step is to obtain a reranking index for each candidate document based on the calculated query relevance index and the candidate relevance index, and rerank the documents based on the reranking index.

[0059] The document re-ranking method in the embodiment of the present application comprehensively considers the length, word frequency, weight and other information of the document, which can effectively improve the accuracy of retrieval ranking and effectively reduce the computational complexity.

[0060] Figure 2 The document reordering method proposed in the embodiment of the present application is shown in the schematic diagram of data processing. , and the initial document sorting sequence , get each query element Based on the ranking of cosine similarity, the top k rankings are obtained. According to the top k rankings, the QP and DP of each candidate document can be calculated to obtain the final sorting sequence D.

[0061] In the embodiment of the present application, in a test environment (with both long document retrieval and short query conditions), the document re-ranking method in the embodiment of the present application can be compared with the BM25 model and the BERT model, and the results are shown in Table 1. The method in the embodiment of the present application is superior to the BM25 model and the BERT model in terms of precision, recall rate, and F1 score. And in terms of time, it is significantly better than the BERT model.

[0062] Table 1

[0063] The method of the embodiment of the present application evaluates the relevance from the perspective of query information and candidate documents, comprehensively considers the relevance and weight between terms, reduces the influencing factors such as sentence length, irrelevant terms and sentences, and repetitiveness, and improves the accuracy of reordering. In terms of computing time, compared with deep learning algorithms, the calculation formula is simple, the computing resources and time are reduced, the computing time is short, and the retrieval efficiency is high.

[0064] The document reordering method in the embodiment of the present application significantly improves the performance of the RAG system and can be widely used in multiple fields, such as search engines, recommendation systems, intelligent question and answer, customer service robots, etc.

[0065] The user experience can be improved by providing more accurate sorting results. It has broad application prospects and can play a role in various occasions that require document sorting and similarity matching. In addition, the document reordering method in the embodiment of the present application is relatively simple and computationally efficient, and can be easily integrated into the existing RAG system, which is conducive to its application and promotion in the market.

[0066] The embodiments of the present application provide Figure 1The document reordering method shown in the figure can be executed by Figure 3 The document reordering device shown. In the embodiment of the present application, the document reordering device executing the document reordering method is taken as an example to illustrate the document reordering device provided by the embodiment of the present application.

[0067] Figure 3 The schematic diagram of the structure of a document reordering device provided by an exemplary embodiment of the present application is shown. The document reordering device can implement the following Figure 1 In the illustrated embodiment, all or part of the content, the document reordering device includes: a first acquisition module 301 , a second acquisition module 302 , a third acquisition module 303 and a reordering module 304 .

[0068] In an embodiment of the present application, a first acquisition module 301 is used to obtain at least one candidate document matching the query information in response to received query information, wherein the candidate document includes at least one candidate sentence; a second acquisition module 302 is used to divide the query information into multiple query elements, and for each query element, based on the similarity between each candidate sentence in the at least one candidate document and the query element, obtain k first candidate sentences whose similarity with the query element is greater than a preset value among all candidate sentences in the at least one candidate document, wherein k is an integer greater than 0; a third acquisition module 303 is used to obtain a relevance index between each candidate document and the query information based on the number of the first candidate sentences included in each candidate document; a reordering module 304 is used to reorder each candidate document according to its relevance index.

[0069] In an optional implementation, the second acquisition module 302, when used to acquire, based on the similarity between each of the candidate sentences in the at least one candidate document and the query element, k first candidate sentences whose similarity with the query element is greater than a preset value among all the candidate sentences in the at least one candidate document, is specifically used to: Calculating the modified cosine similarity between the query element and each candidate sentence in the at least one candidate document; All candidate sentences in the at least one candidate document are sorted in descending order of the modified cosine similarity to obtain the first k candidate sentences that are ranked first.

[0070] In an optional implementation, the relevance index includes: a query relevance index and a document relevance index; The third acquisition module 303, when used to acquire the relevance index between each candidate document and the query information based on the number of the first candidate sentences included in each candidate document, is specifically used to: Obtaining the weight of each of the query elements; Obtaining a query relevance index of each of the candidate documents according to the weight of each of the query elements, the number of the query sentences, and whether each of the candidate documents includes the first candidate sentence of each of the query elements; According to the weight of each query element, the number of the candidate sentences, and whether the candidate sentence of each candidate document is the first candidate sentence of each query element, a document relevance index of each candidate document is obtained.

[0071] In an optional implementation, when the third acquisition module 303 is used to acquire the weight of the query element for each query element, it is specifically used to: The weight of the query element is obtained according to the number of all candidate sentences included in the at least one candidate document and the number of times the query element appears in the at least one candidate document.

[0072] In an optional implementation, the third acquisition module 303, when used to obtain the query relevance index of each candidate document according to the weight of each query element, the number of query sentences, and whether each candidate document includes the first candidate sentence of each query element, is specifically used to: For each candidate document, the query relevance index of the candidate document is calculated using the following formula: :

[0073] in, Represents the query element The weight of is the set of all query elements of the query information, is the set of all candidate sentences of the candidate document d, for The set of first candidate sentences, Indicates that for each , if there is at least one candidate sentence in d Corresponding In the example, the min() function returns 1, if not, it returns 0, and N(q) represents the number of query elements; or, The third acquisition module 303 is used to obtain the document relevance index of each candidate document according to the weight of each query element, the number of candidate sentences, and whether the candidate sentence of each candidate document is the first candidate sentence of each query element, specifically for: For each candidate document, the document relevance index of the candidate document is calculated using the following formula: :

[0074] in, Represents the query element The weight of is the set of all query elements of the query information, is the set of all candidate sentences of the candidate document d, for The corresponding set of first candidate sentences, Represents each candidate sentence for d ,if exist Corresponding In the example, the min() function returns 1, if not, it returns 0, and N(d) represents the number of candidate sentences in the candidate document d. In an optional implementation, when the reordering module 304 reorders the candidate documents according to the relevance index of each candidate document, it is specifically configured to: Based on the query relevance index of each candidate document and the document relevance index of the candidate document, obtaining a reranking index corresponding to each candidate document; Reorder each of the candidate documents based on the reorder index corresponding to each of the candidate documents.

[0075] The document reordering device in the embodiment of the present application can be an electronic device, or a component in the electronic device, such as an integrated circuit or a chip. The electronic device can be a terminal, or it can be other devices other than a terminal. Exemplarily, the electronic device can be a mobile phone, a tablet computer, a laptop computer, a PDA, a vehicle-mounted electronic device, a mobile Internet device (Mobile Internet Device, MID), an augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR) device, a robot, a wearable device, an ultra-mobile personal computer (ultra-mobile personal computer, UMPC), a netbook or a personal digital assistant (personal digital assistant, PDA), etc. It can also be a server, a network attached storage (Network Attached Storage, NAS), a personal computer (personal computer, PC), a television (television, TV), a teller machine or a self-service machine, etc., which is not specifically limited in the embodiment of the present application.

[0076] The document reordering device in the embodiment of the present application may be a device having an operating system. The operating system may be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiment of the present application.

[0077] The document reordering device provided in the embodiment of the present application can achieve Figure 1 To avoid repetition, the various processes implemented by the method embodiment are not described here.

[0078] Alternatively, if Figure 4 As shown, the embodiment of the present application also provides an electronic device 400, including a processor 401 and a memory 402, the memory 402 stores a program or instruction that can be run on the processor 401, and the program or instruction is executed by the processor 401 to achieve the above Figure 1 The various steps of the document reordering method shown can achieve the same technical effect, and will not be repeated here to avoid repetition.

[0079] It should be noted that the electronic devices in the embodiments of the present application include the mobile electronic devices and non-mobile electronic devices mentioned above.

[0080] Figure 5FIG. 1 is a block diagram of another electronic device 500 according to an exemplary embodiment of the present application. The electronic device 500 may be implemented as a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, a TV, etc. The electronic device 500 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, etc.

[0081] Typically, the electronic device 500 includes a processor 501 and a memory 502 .

[0082] The processor 501 may include one or more processing cores, such as a 4-core processor, a 5-core processor, etc. The processor 501 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 501 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 501 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 501 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0083] The memory 502 may include one or more computer-readable storage media, which may be non-transitory. The memory 502 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 502 is used to store at least one instruction, which is used to be executed by the processor 501 to implement all or part of the steps in the document reordering method shown in the method embodiment of the present application.

[0084] In some embodiments, the electronic device 500 may further optionally include: a peripheral device interface 503 and at least one peripheral device. The processor 501, the memory 502 and the peripheral device interface 503 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 503 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 504, a display screen 509, a camera assembly 506, an audio circuit 507 and a power supply 508.

[0085] In some embodiments, the electronic device 500 further includes one or more sensors 509 , including but not limited to: an acceleration sensor 510 , a gyroscope sensor 511 , a pressure sensor 512 , an optical sensor 513 , and a proximity sensor 514 .

[0086] Those skilled in the art will understand that Figure 5 The structure shown in the figure does not constitute a limitation on the electronic device 500, and may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0087] The embodiment of the present application also provides a computer-readable storage medium on which a program or instruction is stored, and when the program or instruction is executed by a processor, the above Figure 1 The various processes of the document reordering method shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0088] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0089] The present application also provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run a program or instruction to implement the above Figure 1 The various processes of the document reordering method shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0090] It should be understood that the chip mentioned in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0091] The present application also provides a computer program product, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the above Figure 1The steps of the document reordering method shown in the figure can achieve the same technical effect. To avoid repetition, they will not be repeated here.

[0092] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the claims.

[0093] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A document reordering method, characterized in that: include: In response to the received query information, obtaining at least one candidate document matching the query information, wherein the candidate document includes at least one candidate sentence; Dividing the query information into a plurality of query elements, for each query element, obtaining, based on the similarity between each candidate sentence in the at least one candidate document and the query element, k first candidate sentences whose similarity with the query element is greater than a preset value from among all candidate sentences in the at least one candidate document, where k is an integer greater than 0; Based on the number of the first candidate sentences included in each of the candidate documents, obtaining a relevance index between each of the candidate documents and the query information; The candidate documents are reordered according to the relevance index of each candidate document.

2. The method according to claim 1, characterized in that The step of obtaining, based on the similarity between each of the candidate sentences in the at least one candidate document and the query element, k first candidate sentences whose similarity with the query element is greater than a preset value from among all the candidate sentences in the at least one candidate document comprises: Calculating the modified cosine similarity between the query element and each candidate sentence in the at least one candidate document; All candidate sentences in the at least one candidate document are sorted in descending order of the modified cosine similarity to obtain the first k candidate sentences that are ranked first.

3. The method according to claim 1, characterized in that The relevance index includes: a query relevance index and a document relevance index; The acquiring, based on the number of the first candidate sentences included in each of the candidate documents, a relevance index between each of the candidate documents and the query information includes: Obtaining the weight of each of the query elements; Obtaining a query relevance index of each of the candidate documents according to the weight of each of the query elements, the number of the query sentences, and whether each of the candidate documents includes the first candidate sentence of each of the query elements; According to the weight of each query element, the number of the candidate sentences, and whether the candidate sentence of each candidate document is the first candidate sentence of each query element, a document relevance index of each candidate document is obtained.

4. The method according to claim 3, characterized in that: For each of the query elements, obtaining the weight of the query element includes: The weight of the query element is obtained according to the number of all candidate sentences included in the at least one candidate document and the number of times the query element appears in the at least one candidate document.

5. The method according to claim 3, characterized in that: The step of obtaining the query relevance index of each candidate document according to the weight of each query element, the number of query sentences, and whether each candidate document includes the first candidate sentence of each query element comprises: for each candidate document, calculating the query relevance index of the candidate document using the following formula: : in, Represents the query element The weight of is the set of all query elements of the query information, is the set of all candidate sentences of the candidate document d, for The set of first candidate sentences, Indicates that for each , if there is at least one candidate sentence in d Corresponding In the example, the min() function returns 1, if not, it returns 0, and N(q) represents the number of query elements; or, According to the weight of each query element, the number of candidate sentences, and whether the candidate sentence of each candidate document is the first candidate sentence of each query element, the document relevance index of each candidate document is obtained, including: for each candidate document, the document relevance index of the candidate document is calculated using the following formula: : in, Represents the query element The weight of is the set of all query elements of the query information, is the set of all candidate sentences of the candidate document d, for The corresponding set of first candidate sentences, Represents each candidate sentence for d ,if exist Corresponding In the example, the min() function returns 1, if not, it returns 0, and N(d) represents the number of candidate sentences in the candidate document d.

6. The method according to claim 3, characterized in that The reordering of the candidate documents according to the relevance index of each candidate document includes: Based on the query relevance index of each candidate document and the document relevance index of the candidate document, obtaining a reranking index corresponding to each candidate document; Reorder each of the candidate documents based on the reorder index corresponding to each of the candidate documents.

7. A document reordering device, characterized in that: include: A first acquisition module, configured to acquire, in response to received query information, at least one candidate document matching the query information, wherein the candidate document includes at least one candidate sentence; A second acquisition module is configured to divide the query information into a plurality of query elements, and for each query element, based on the similarity between each candidate sentence in the at least one candidate document and the query element, acquire k first candidate sentences whose similarity with the query element is greater than a preset value from among all candidate sentences in the at least one candidate document, where k is an integer greater than 0; A third acquisition module, configured to acquire a correlation index between each of the candidate documents and the query information based on the number of the first candidate sentences included in each of the candidate documents; The reordering module is used to reorder each of the candidate documents according to the relevance index of each of the candidate documents.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory, wherein the memory stores a program or instruction that can be run on the processor, and when the program or instruction is executed by the processor, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program or instruction, and when the program or instruction is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program comprises program instructions, and when the program instructions are executed by a computer, the computer is caused to perform the steps of the method according to any one of claims 1 to 6.

Images