Document retrieval method and device, computer equipment, storage medium and computer program product

Abstract

The invention relates to a document retrieval method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: receiving a query word, segmenting the query word into a plurality of sub-words, and querying from a document library to obtain a sub-word sequence and a candidate document matched with any sub-word; the sub-word sequence is obtained by segmenting a document; constructing a plurality of query sub-word sequences, wherein sub-words in the query sub-word sequences are continuous sub-words in the query words; for any candidate document, respectively determining fragments matched with each query sub-word sequence from the sub-word sequences of the candidate document, and determining a correlation score of the candidate document based on the inverse document frequency of each sub-word covered in each fragment and the length of the fragment; and sorting the candidate documents based on the correlation scores of the candidate documents, and outputting the sorted candidate documents. By adopting the method, the relevancy of document sorting and query words can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a document retrieval method, apparatus, computer equipment, storage medium, and computer program product. Background Technology

[0002] Sparse retrieval is a retrieval algorithm that segments documents into tokens, mapping them to a high-dimensional vocabulary, and then selects the best-matching document by calculating the similarity between the query vector and the document vector. This algorithm remains central to modern information retrieval systems due to its efficiency and interpretability.

[0003] The BM25 algorithm is widely recognized in the industry as the most effective and widely used ranking algorithm. However, the BM25 algorithm relies on a word segmenter specific to a particular language. When encountering new words not included in the dictionary, trending online terms, brand names, or model numbers, the word segmenter is prone to segmentation errors, severely impacting retrieval recall. Consequently, the documents output to the user may not be arranged according to the objective degree of matching with the query terms, resulting in poor retrieval accuracy. Summary of the Invention

[0004] Therefore, it is necessary to provide a document retrieval method, apparatus, computer equipment, storage medium, and computer program product to address the aforementioned technical problems.

[0005] Firstly, this application provides a document retrieval method, the method comprising:

[0006] The system receives a query term, segments the query term into multiple sub-words, and retrieves a sub-word sequence and candidate documents matching any of the sub-words from a document database; the sub-word sequence is obtained by segmenting documents.

[0007] Construct multiple query sub-word sequences, wherein the sub-words in the query sub-word sequences are consecutive sub-words in the query term;

[0008] For any candidate document, segments matching each query sub-word sequence are determined from the sub-word sequence of the candidate document. Based on the inverse document frequency of each sub-word covered in each segment and the length of the segment, the relevance score of the candidate document is determined.

[0009] Based on the relevance scores of each candidate document, the candidate documents are sorted and the sorted candidate documents are output.

[0010] In one embodiment, the process of ranking the candidate documents based on their relevance scores includes:

[0011] The candidate documents are sorted based on their coverage factor and / or length penalty factor, and based on their relevance score.

[0012] The coverage factor is determined based on the number of sub-words matched by the candidate document in each sub-word corresponding to the query term;

[0013] The length penalty factor is determined based on the length of the word sequence of the candidate document.

[0014] In one embodiment, the method further includes:

[0015] Identify the unique sub-words from each of the sub-words, and determine the number of unique sub-words;

[0016] For any candidate document, determine the number of matching subwords in the non-repeating subwords of the candidate document, and determine the coverage factor of the candidate document based on the ratio between the number of matching subwords and the number of non-repeating subwords;

[0017] The coverage factor and the ratio have a superlinear relationship.

[0018] In one embodiment, the method further includes:

[0019] The global average subword sequence length is determined based on the subword sequence length of each document in the document library;

[0020] For any of the candidate documents, the length penalty factor of the candidate document is determined based on the ratio of the length of the sub-word sequence of the candidate document to the global average sub-word sequence length.

[0021] The length penalty factor and the ratio are negatively correlated.

[0022] In one embodiment, determining the segments that match each of the query sub-word sequences from the sub-word sequences of the candidate documents includes:

[0023] The query sub-word sequences are traversed from longest to shortest. For the currently traversed query sub-word sequence, a segment matching the query sub-word sequence is retrieved from the sub-word sequence of the candidate document.

[0024] If no matching segment covers the segment, the segment is added as a matching segment; or if there is a matching segment that covers the segment, the segment is discarded, until all the query sub-word sequences have been traversed.

[0025] In one embodiment, determining the relevance score of the candidate document based on the inverse document frequency of each of the subwords covered in each of the segments and the length of the segments includes:

[0026] For any of the segments, a relevance score is determined based on the inverse document frequency of each of the sub-words covered in the segment and the length of the segment; wherein the relevance score of the segment has a sublinear relationship with the inverse document frequency of each of the sub-words and a superlinear relationship with the length of the segment;

[0027] The relevance score of the candidate document is determined based on the relevance scores of each segment contained in the candidate document.

[0028] Secondly, this application also provides a document retrieval device, the device comprising:

[0029] The segmentation module is used to receive query terms, segment the query terms into multiple sub-words, and retrieve sub-word sequences and candidate documents matching any of the sub-words from the document library; the sub-word sequence is obtained by segmenting documents.

[0030] A construction module is used to construct multiple query sub-word sequences, wherein the sub-words in the query sub-word sequences are consecutive sub-words in the query term;

[0031] The first determining module is configured to, for any candidate document, determine segments from the sub-word sequence of the candidate document that match each query sub-word sequence, and determine the relevance score of the candidate document based on the inverse document frequency of each sub-word covered in each segment and the length of the segment;

[0032] The sorting module is used to sort the candidate documents based on their relevance scores and output the sorted candidate documents.

[0033] In one embodiment, the sorting module is further configured to:

[0034] The candidate documents are sorted based on their coverage factor and / or length penalty factor, and based on their relevance score.

[0035] The coverage factor is determined based on the number of sub-words matched by the candidate document in each sub-word corresponding to the query term;

[0036] The length penalty factor is determined based on the length of the word sequence of the candidate document.

[0037] In one embodiment, the device further includes:

[0038] The second determining module is used to determine non-repeating sub-words from each of the sub-words and to determine the number of non-repeating sub-words;

[0039] The third determining module is used to determine, for any candidate document, the number of matching subwords in the non-repeating subwords of the candidate document, and to determine the coverage factor of the candidate document based on the ratio between the number of matching subwords and the number of non-repeating subwords;

[0040] The coverage factor and the ratio have a superlinear relationship.

[0041] In one embodiment, the device further includes:

[0042] The fourth determining module is used to determine the global average sub-word sequence length based on the sub-word sequence length of each document in the document library;

[0043] The fifth determining module is used to determine the length penalty factor of any candidate document based on the ratio of the length of the sub-word sequence of the candidate document to the global average sub-word sequence length.

[0044] The length penalty factor and the ratio are negatively correlated.

[0045] In one embodiment, the first determining module is further configured to:

[0046] The query sub-word sequences are traversed from longest to shortest. For the currently traversed query sub-word sequence, a segment matching the query sub-word sequence is retrieved from the sub-word sequence of the candidate document.

[0047] If no matching segment covers the segment, the segment is added as a matching segment; or if there is a matching segment that covers the segment, the segment is discarded, until all the query sub-word sequences have been traversed.

[0048] In one embodiment, the first determining module is further configured to:

[0049] For any of the segments, a relevance score is determined based on the inverse document frequency of each of the sub-words covered in the segment and the length of the segment; wherein the relevance score of the segment has a sublinear relationship with the inverse document frequency of each of the sub-words and a superlinear relationship with the length of the segment;

[0050] The relevance score of the candidate document is determined based on the relevance scores of each segment contained in the candidate document.

[0051] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement any of the methods described above.

[0052] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, implements any of the above methods.

[0053] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, implements any of the above methods.

[0054] The aforementioned document retrieval methods, apparatuses, computer devices, storage media, and computer program products use sub-words as the basic unit of retrieval. Since sub-word segmentation does not rely on a dictionary, it can effectively handle multilingual retrieval and new word retrieval. To address the potential problems introduced by using sub-word retrieval, such as inverse document frequency distortion (a large number of words sharing the same root word) and word frequency noise pollution (sub-words introducing a large number of semantically irrelevant retrieval results), this application's embodiments use continuous sub-word sequences from the query term for retrieval. Furthermore, the relevance of the retrieved segments is evaluated using a combination of inverse document frequency and segment length. Finally, all retrieved documents are ranked based on the relevance of all retrieved segments. Because continuous sub-word sequences can preserve semantic information, the semantic relevance between the retrieved segments and the query term can be improved. Using the inverse document frequency of all sub-words contained in a segment and the segment length, rather than the inverse document frequency of a single sub-word, for relevance evaluation can also suppress the inverse document frequency distortion problem of individual sub-words. This application's embodiments, based on improving recall by using sub-word retrieval, further enhance the final document ranking and relevance to query terms by calculating the relevance score of fragments, thereby improving the ranking accuracy of output documents. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating a document retrieval method in one embodiment;

[0056] Figure 2 This is a schematic diagram illustrating the retrieval of matching fragments from candidate documents in one embodiment;

[0057] Figure 3 This is a flowchart illustrating the document retrieval method in another embodiment;

[0058] Figure 4 This is a structural block diagram of a document retrieval device in one embodiment;

[0059] Figure 5This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0060] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0061] To better understand this application, the BM25 algorithm and word segmentation involved in this application will be further explained first. The execution process of the BM25 algorithm is as follows: First, the documents and query terms in the document library are segmented using a word segmenter. Then, for any document and any word in the query term, the BM25 algorithm determines the relevance between the document and the word by calculating the inverse document frequency of the word, the frequency of the word in the document, and the length of the document. The BM25 algorithm then sums the relevance of all words to the document to obtain the relevance between the query term and the document. Finally, the BM25 algorithm ranks the relevance between the query term and each document.

[0062] However, the traditional BM25 algorithm and its application models face two major technical bottlenecks when dealing with today's complex application environment characterized by multiple languages, multiple fields, and a constant emergence of new and trending words:

[0063] 1. Reliance on language-specific word segmenters and dictionaries: The basic unit of operation for the traditional BM25 algorithm is the "word". To obtain these "words", the system must be equipped with a dedicated word segmenter and a large maintenance dictionary for each language (such as Chinese, Japanese, Korean, etc.). This results in a complex system architecture, high costs for international expansion, and difficulty in covering the proprietary vocabulary of all vertical fields.

[0064] 2. Inability to handle out-of-vocabulary words: Traditional word segmenters cannot properly handle new words not included in the dictionary, internet slang, brand names, model numbers, or misspelled words, causing this information to be lost during the indexing stage. Users are unable to retrieve relevant documents during queries, which seriously affects the recall rate and user experience.

[0065] To address the aforementioned issues, the word segmentation technique used in the BM25 algorithm can be replaced with a sub-word segmentation technique. This technique does not rely on a dictionary, can segment any text into a finite set of sub-words, and naturally possesses the ability to handle multilingual and out-of-vocabulary words.

[0066] However, in the field of sparse retrieval, directly combining sub-word segmentation technology with traditional ranking algorithms such as BM25 faces a series of significant technical challenges. This is because sub-word segmentation technology breaks down semantically complete "words" into statistically significant "sub-word fragments." This representation is incompatible with the statistical assumptions of the traditional BM25 algorithm, which is based on "words," leading to the following problems:

[0067] 1. Inverse Document Frequency (IDF) Distortion: Many high-frequency functional sub-words or shared root words have document frequencies far exceeding those of the complete word, leading to a severe underestimation of their IDF values ​​and weakening the weight of key retrieval signals. For example, in English, words like "unhappy," "unwise," and "unrestrained" all contain the root "un." If "unwise" is a key retrieval signal in the query, it will be segmented into "un" and "wise" during sub-word segmentation. Because the IDF value of the high-frequency root "un" is low, the BM25 algorithm tends to retrieve a large number of documents containing only "wise" and no "unwise," which are completely irrelevant to the user's needs.

[0068] 2. Term Frequency (TF) Noise Pollution: Term frequency statistics introduce a large amount of semantic noise from irrelevant words, interfering with the accuracy of relevance judgment. Using the example above, when using the high-frequency root word "un" for retrieval, the BM25 algorithm may misjudge documents containing "unhappy" or "unrestrained" as also relevant to the query term, resulting in higher relevance scores for irrelevant documents.

[0069] 3. Document Length Normalization Bias: Normalizing document length using the number of sub-words disproportionately penalizes high-information-density documents containing complex or rare words (thus splitting them into more sub-words). For example, in the example above, the word "unrestrained" might be split into five sub-words: "un," "re," "stra," "in," and "ed." Because the BM25 algorithm penalizes documents containing more words, documents containing multiple instances of "unrestrained" might actually have lower relevance scores than documents containing only a few instances of "unrestrained," resulting in top-ranked documents not being the most relevant to the query term.

[0070] In summary, there is an urgent need in this field for a new technical solution that can effectively utilize the advantages of sub-word segmentation technology, while also designing a sorting mechanism to circumvent the aforementioned technical problems, ultimately providing a sparse retrieval method that requires no training and has robust performance.

[0071] To at least partially solve the above problems, in one embodiment, such as Figure 1 As shown, a document retrieval method is provided. This embodiment illustrates the method applied to a server; however, it is understood that the method can also be applied to a terminal, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0072] Step 102: Receive the query term, segment the query term into multiple sub-words, and retrieve the sub-word sequence and candidate documents that match any sub-word from the document library; the sub-word sequence is obtained by segmenting the document.

[0073] In this embodiment, the document library stores multiple documents. The server can segment the documents into a sequence of subwords based on any sub-word segmentation algorithm (such as BPE (Byte Pair Encoding), single-word language model, etc.). After the user inputs a query term, the server can use the same algorithm as when segmenting the document to segment the query term into a sequence of subwords as well. For example, if the query term is "query the document library by query term", then the segmented sub-word sequence might be: [by, query, term, query, document, library]. The reason for using the same algorithm here is to ensure that the same words will be segmented into the same subwords.

[0074] The server then retrieves documents from the document database that match at least one of the sub-words segmented from the query term. The server can first filter out unique sub-words from each sub-word, and then, for each unique sub-word, if the document contains that sub-word in its sub-word sequence, consider that document as a candidate document for that sub-word. After obtaining candidate documents for all sub-words, the server further filters out unique candidate documents as the final candidate documents.

[0075] For example, in the example above, the server can query the document library based on "by", "query", "word", "document", and "library". If any of the subwords contained in the query term appear in the subword sequence of a document, that document can be considered a candidate document. The candidate documents retrieved at this stage are not necessarily semantically related to the query term. For example, based on the aforementioned subwords, the retrieved documents might be things like "segment the query term into multiple subwords" or "the document library stores multiple documents," which are clearly not semantically related to the query term. Alternatively, the retrieved documents might be things like "retrieve the document library by query term," which are somewhat different from the query term but clearly semantically related.

[0076] Step 104: Construct multiple query sub-word sequences, where the sub-words in the query sub-word sequences are consecutive sub-words in the query terms.

[0077] In this embodiment, based on the sub-words previously segmented from the query term, multiple query sub-word sequences containing consecutive sub-words can be constructed. For example, in the example of the aforementioned embodiment, [through, query, word] can be a query sub-word sequence, and [word, query, document] can also be a query sub-word sequence. It should be noted that a single sub-word can also be regarded as a query sub-word sequence, such as [through], [word], etc.

[0078] In one example, all possible consecutive sub-words can be combined into a query sub-word sequence. For instance, in the previous example, the query term was split into 6 sub-words. We can first combine them into a query sub-word sequence containing 6 consecutive sub-words: [through, query, word, query, document, library]. Then we can combine them into a query sub-word sequence containing 5 consecutive sub-words: [through, query, word, query, document] and [query, word, query, document, library]. Then we can combine them into a query sub-word sequence containing 4 consecutive sub-words, and so on, until all possible consecutive sub-words are combined into a query sub-word sequence.

[0079] In another example, the resulting query term sequence can be filtered again after performing the above steps. Since short query term sequences with high inverse document frequencies for all their terms (e.g., "[by]" and "[word]") are unlikely to be key retrieval signals, the following operations can be performed on query term sequences whose length is less than a length threshold: determine the inverse document frequency factor corresponding to the query term sequence based on the inverse document frequency of each term in the query term sequence; determine the length factor corresponding to the query term sequence based on the length of the query term sequence; determine the key score corresponding to the query term sequence based on the inverse document frequency factor and the length factor, and delete query term sequences whose key score is less than a preset threshold.

[0080] The length threshold can be a fixed value (such as 2 or 3), or it can be determined based on the length of the longest query term sequence (such as one-tenth of the longest query term sequence), or it can be the larger of the value determined based on the length of the longest query term sequence and the fixed value.

[0081] The inverse document frequency factor can be the average of the inverse document frequencies of all subwords, or it can be calculated based on the average and a certain factor (e.g., the factor power based on the average). This allows the factor to have a superlinear relationship between the average inverse document frequency and the inverse document frequency factor, meaning that the higher the inverse document frequency, the faster the inverse document frequency factor grows.

[0082] The length factor can be determined solely based on the length of the query sub-word sequence, for example, by setting the length factor to be equal to the length of the query sub-word sequence. Alternatively, it can be determined based on the ratio of the length of the query sub-word sequence to the length of the longest query sub-word sequence.

[0083] The key score can be determined based on the sum or product of the inverse document frequency factor and the length factor.

[0084] By screening the query sub-word sequence, the number of segments for which relevance needs to be calculated in subsequent steps can be reduced at the cost of a small loss in query accuracy, thereby improving query efficiency.

[0085] In step 106, for any candidate document, segments that match each query sub-word sequence are respectively determined from the sub-word sequence of the candidate document, and the relevance score of the candidate document is determined based on the inverse document frequency of each sub-word covered in each segment and the length of the segment.

[0086] In the embodiments of the present application, a segment in the sub-word sequence of a candidate document that matches a query sub-word sequence refers to multiple consecutive sub-words in the sub-word sequence that are exactly the same as the query sub-word sequence. For example, if the candidate document is "segment the query term into multiple sub-words" (corresponding sub-word sequence: [segment, query, term, into, multiple, sub-words]) and the query sub-word sequence is [query, term], then the segment in this candidate document that matches the query sub-word sequence is [query, term] from the 2nd element to the 3rd element of the sub-word sequence.

[0087] Each query sub-word sequence may have a matching segment in the candidate document. To prevent the same segment from being repeatedly counted, duplicate removal processing can also be performed on the retrieved segments after retrieving all segments that match the query sub-word sequences. Segment duplication means that all elements included in a certain segment appear in another segment. For example, when retrieving using the query sub-word sequence [query], the segment in the candidate document that matches the query sub-word sequence is the 2nd element of the sub-word sequence. When retrieving using the query sub-word sequence [query, term], the segment in the candidate document that matches the query sub-word sequence is from the 2nd element to the 3rd element of the sub-word sequence. Then the segment corresponding only to the 2nd element is completely covered by the segment corresponding to the 2nd to 3rd elements. When segments overlap, the longest segment can be retained and other segments can be deleted to preferentially use longer matching segments to calculate the relevance score.

[0088] In one embodiment, duplicate segments can also be removed in real time during the process of querying for matching segments, as Figure 2 shown, including:

[0089] Traverse each query sub-word sequence from long to short. For the currently traversed query sub-word sequence, query for segments that match the query sub-word sequence from the sub-word sequence of the candidate document;

[0090] If no matching fragment overlaps with another fragment, add the fragment as a matching fragment; or if a matching fragment exists that overlaps with another fragment, discard the fragment until all query sub-word sequences have been traversed.

[0091] In this embodiment, the longest query sub-sequence is first used to query matching segments. The specific query process can be to start from the beginning of the candidate document's sub-sequence, sequentially extract segments of length equal to the query sub-sequence, and match them with the query sub-sequence. If this segment is identical to the query sub-sequence, then this segment is determined as the matching segment.

[0092] After each matched segment is identified, it is checked whether there is a matching segment that covers this segment among the already marked matching segments. Coverage means that all elements contained in the original segment appear in the matching segment. For example, if the original segment contains elements 1 to 5 of the sub-word sequence, and the matching segment contains elements 1 to 8 of the sub-word sequence, then the matching segment covers the original segment. If there is any element that does not appear in the matching segment, such as if the matching segment contains elements 2 to 9 of the sub-word sequence, then the matching segment does not cover the original segment.

[0093] If a matching segment exists that covers this segment, it means that this segment has already been recorded, and it should be discarded to avoid duplicate recording. If no such matching segment exists, then this segment is recorded as a matching segment. Repeat this process until the end of the candidate document's sub-word sequence is found, and then repeat the above process with the next query sub-word sequence until all query sub-word sequences have been traversed.

[0094] The above process is illustrated with a practical example. Assume the candidate document is "retrieve document library by query term", and the corresponding sub-word sequence is [by, query, term, retrieve, document, library]. The query sub-word sequences are "[by, query, term]", "[query, term]", and "[document, library]".

[0095] First, the longest query term sequence is used for retrieval. Starting from the beginning of the candidate document's term sequence, segments of length equal to the query term sequence are sequentially extracted and matched against the query term sequence. The first extracted segment is "[through, query, term]", which matches the query term sequence. Since there is currently no matching segment, there is no segment that covers this segment, so it can be marked as a matching segment. The second extracted segment is "[query, term, retrieval]", which does not match the query term sequence. Extraction is repeated until the end of the candidate document's term sequence is reached. At this point, the extracted segment is "[retrieval, document, library]", which also does not match the query term sequence. The retrieval of "[through, query, term]" is now complete, and the retrieval of the next query term sequence "[query, term]" begins.

[0096] Starting from the beginning of the candidate document's sub-word sequence, segments of length equal to the query sub-word sequence are sequentially extracted and matched against the query sub-word sequence. The second extracted segment is "[query, word]", which matches the query sub-word sequence. However, since this segment is the 2nd to 3rd element of the sub-word sequence, it is covered by the currently matched segments (the 1st to 3rd elements of the sub-word sequence), and therefore needs to be discarded. This extraction process is repeated until the end of the candidate document's sub-word sequence is reached, completing the retrieval of "[query, word]", and starting the retrieval of the next query sub-word sequence "[document, library]".

[0097] Starting from the beginning of the candidate document's sub-word sequence, segments of length equal to the query sub-word sequence are sequentially extracted and matched against the query sub-word sequence. The last extracted segment is "[document, library]", which matches the query sub-word sequence. Furthermore, this segment is the 5th to 6th element of the sub-word sequence and is not covered by the currently matched segments (the 1st to 3rd elements of the sub-word sequence). Therefore, this segment is also added to the matching segments. The final matching segments are: "[through, query, word]" and "[document, library]".

[0098] The algorithm retrieves all segments in candidate documents that match the query sub-word sequence. Further, based on the inverse document frequency of each sub-word covered in each segment and the segment length, it determines the relevance score of the candidate documents. This approach ensures that the inverse document frequency of each sub-word and the segment length are positively correlated with the relevance score, thereby rewarding rarer sub-words and longer matching segments.

[0099] In one embodiment, the steps for calculating the relevance score include:

[0100] For any given segment, a relevance score is determined based on the inverse document frequency of each word covered in the segment and the length of the segment. The relevance score of the segment has a sublinear relationship with the inverse document frequency of each word and a superlinear relationship with the length of the segment.

[0101] The relevance score of a candidate document is determined based on the relevance scores of each segment contained in the candidate document.

[0102] In this embodiment, the inverse document frequency factor can be obtained by summing, averaging, or summing or averaging the inverse document frequencies of each sub-word after processing them with a factor. The length factor can be obtained by using the fragment length or by processing the fragment length with a factor. The relevance score can be obtained by summing or multiplying the inverse document frequency factor and the length factor. This embodiment does not limit the specific calculation methods for the inverse document frequency factor, length factor, and relevance score, as long as the relevance score has a sublinear relationship with the inverse document frequency of each sub-word (i.e., the higher the inverse document frequency, the slower the relevance score increases) and a superlinear relationship with the length (i.e., the longer the fragment length, the faster the relevance score increases), so that the relevance score can suppress the inaccuracy of the inverse document frequency of sub-words and reward longer matching fragments.

[0103] For example, the inverse document frequency factor can be calculated using formula (1):

[0104] Formula (1)

[0105] in, This is the inverse document frequency factor, where s represents a segment, t is each subword belonging to the segment, and IDF is the inverse document frequency. A 1 is added here as a correction term to avoid negative or zero values ​​calculated by the logarithmic function.

[0106] For example, the length factor can be calculated using formula (ii):

[0107] Formula (II)

[0108] in, It is a length factor. It is the segment length. It is a reward factor, which can be determined according to actual needs, for example, it can be set to 2.0.

[0109] After obtaining the relevance score for each segment, the relevance scores of each segment can be summed or multiplied to obtain the relevance score of the candidate document.

[0110] Step 108: Based on the relevance scores of each candidate document, sort the candidate documents and output the sorted candidate documents.

[0111] In this embodiment, after obtaining the relevance score of each candidate document, the candidate documents can be sorted according to the relevance score from high to low or from low to high, as long as the sorting method reflects the relevance between the candidate documents and the query terms. The server can then display the sorted candidate documents to the user.

[0112] The document retrieval method provided in this application uses sub-words as the basic unit of retrieval. Since sub-word segmentation does not rely on a dictionary, it can effectively handle multilingual retrieval and new word retrieval. To address issues such as inverse document frequency distortion (a large number of words sharing the same root word) and word frequency noise pollution (sub-words introducing a large number of semantically irrelevant retrieval results) that may be introduced by using sub-word retrieval, this application uses continuous sub-word sequences from the query term for retrieval. Furthermore, it uses a combination of inverse document frequency and segment length to evaluate the relevance between the retrieved segments and the sub-word sequence. Finally, it determines the document ranking based on the relevance of all retrieved segments in the document. Since continuous sub-word sequences can preserve semantic information, they can improve the semantic relevance between the retrieved segments and the query term. Using the inverse document frequency of all sub-words contained in a segment and the segment length, rather than the inverse document frequency of a single sub-word, for relevance evaluation can also suppress the inverse document frequency distortion problem of a single sub-word. This application, while improving recall by using sub-word retrieval, further improves the relevance of the final determined document ranking to the query term by calculating the relevance score of the segments, thereby improving the ranking accuracy of the output documents.

[0113] In one embodiment, in addition to relevance scores, at least one of coverage factors and length penalty factors may be used together to rank documents, including:

[0114] The candidate documents are ranked based on their coverage factor and / or length penalty factor, as well as their relevance score.

[0115] The coverage factor is determined based on the number of sub-words matched by the candidate document in each sub-word corresponding to the query term;

[0116] The length penalty factor is determined based on the length of the word sequence in the candidate document.

[0117] In this embodiment, if a subword in the query term appears in the subword sequence of a candidate document, then that subword is called a subword that matches the candidate document. The coverage factor is calculated based on the number of matched subwords. For example, if the subwords of the query term are "[through, query, word, query, document, library]", and the candidate document is "[through, query, word, retrieval, document, library]", then the number of matched subwords can be 5 (if duplicate subwords are not counted) or 6 (if duplicate subwords are counted). The coverage factor can be set to the number of matched subwords, or calculated based on the number of matched subwords and the highest number of matched subwords in each candidate document. This factor is positively correlated with the number of matched subwords.

[0118] The length penalty factor, used to suppress the weight of excessively long documents, can be determined based on the word sequence length of the candidate documents. Since documents with longer matching segments have already been rewarded through relevance scores, using the length penalty factor here to suppress the weight of long documents will not have a significant negative impact on retrieval accuracy. This factor is negatively correlated with word sequence length.

[0119] The following methods can be used simultaneously for sorting: 1) Calculate a final score using the sum or product of the coverage factor, length penalty factor, and relevance score; 2) Sort candidate documents using the relevance score, then further sort candidate documents whose relevance score is greater than a certain preset threshold using the coverage factor or length penalty factor; 3) Sort candidate documents using the coverage factor or length penalty factor, then further sort candidate documents whose coverage factor or length penalty factor is greater than another preset threshold using the relevance score. This application does not limit the specific use of these three parameters for sorting.

[0120] In one embodiment, a method for calculating the coverage factor includes:

[0121] Identify the unique sub-words from each sub-word and determine the number of unique sub-words;

[0122] For any candidate document, determine the number of matching words in the non-repeating sub-words of the candidate document, and determine the coverage factor of the candidate document based on the ratio between the number of matching words and the number of non-repeating sub-words;

[0123] Among them, the coverage factor and the ratio have a superlinear relationship.

[0124] In this embodiment, each sub-word of the query term is first deduplicated, and the number of remaining unique sub-words is counted. Then, the number of sub-words matched by the candidate document among the unique sub-words is determined. Subsequently, a coverage factor can be determined based on the ratio between the number of matched sub-words and the number of unique sub-words, and the coverage factor needs to have a superlinear relationship with the ratio to reward candidate documents that match more unique sub-words.

[0125] For example, the coverage factor can be calculated using formula (iii):

[0126] Formula (3)

[0127] in, Coverage factor This refers to the number of non-repeating words matched, or the number of matched words. It is the number of non-repeating subwords. It is a penalty factor used to achieve superlinear relationships, which can be set according to actual needs, for example, 2.0.

[0128] In one embodiment, a method for calculating the length penalty factor includes:

[0129] Determine the global average subword sequence length based on the subword sequence length of each document in the document library;

[0130] For any candidate document, the length penalty factor of the candidate document is determined based on the ratio of the candidate document's sub-word sequence length to the global average sub-word sequence length.

[0131] Among them, the length penalty factor and the ratio are negatively correlated.

[0132] In this embodiment, the global average word sequence length is obtained by averaging the word sequence lengths of all documents in the document library. The length penalty factor of a candidate document can be determined by the ratio of the word sequence length of the candidate document to the global average word sequence length. The length penalty factor and the ratio are negatively correlated, so that the length penalty factor of long documents is less than that of short documents, avoiding the unfair advantage of long documents being calculated with higher relevance scores.

[0133] For example, the length penalty factor can be calculated using formula (iv):

[0134] Formula (IV)

[0135] in, It is a length penalty factor. It is the length of the word sequence of the candidate document. is the global average sub-word sequence length, and b is the normalization strength factor, used to prevent the denominator from being zero and to control the degree of penalty for long documents. The larger the b is, the more severe the penalty for long documents. The value of b can be selected according to actual needs, for example, 0.75.

[0136] The document retrieval method provided in this application further employs a coverage factor and a length penalty factor for document sorting, which can further consider the number of matching sub-words and document length during sorting, thereby improving the accuracy of sorting.

[0137] In one embodiment, a document retrieval method is provided, such as Figure 3 As shown, it includes:

[0138] S1: Perform sub-word segmentation on all documents in the document library to obtain the sub-word sequence of each document, and generate the inverted index and inverse document frequency for each sub-word.

[0139] The inverted index stores the documents in which the keyword appears and their positions within those documents. This step is performed to improve the speed of subsequent candidate document retrieval and to allow for direct access to the calculated data when calculating inverse document frequency and the number of matching keywords, thus improving real-time query speed. This step can be performed once each time a new document is added to the document database.

[0140] S2 accepts the query term, segments the query term into multiple sub-words, and retrieves the sub-word sequence and candidate documents that match any sub-word from the document library.

[0141] Since S1 already records the documents in which each subword appears, this step can be performed as follows: retrieve the inverted index of each subword in the query term, extract the documents in which the subword appears from the inverted index, and take the union of the documents to obtain candidate documents.

[0142] For each document, continue executing S3-S5:

[0143] S3, calculate the coverage factor of candidate documents.

[0144] S4, calculate the relevance score of the candidate documents.

[0145] S5, calculate the length penalty factor for candidate documents.

[0146] S6 calculates the final score based on the product of the coverage factor, the relevance score, and the length penalty factor.

[0147] The calculation methods for S3-S5 are described in the aforementioned embodiments, and will not be repeated in the embodiments of this application.

[0148] S7: After calculating the final score for all documents, sort the candidate documents in descending order based on the final score and output the sorted candidate documents.

[0149] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0150] Based on the same inventive concept, this application also provides a document retrieval device for implementing the document retrieval method described above. The solution provided by this device is similar to the solution described in the above method; therefore, the specific limitations in one or more document retrieval device embodiments provided below can be found in the limitations of the document retrieval method described above, and will not be repeated here.

[0151] In one embodiment, such as Figure 4 As shown, a document retrieval device 400 is provided, including: a segmentation module 402, a construction module 404, a first determination module 406, and a sorting module 408, wherein:

[0152] The segmentation module 402 is used to receive a query term, segment the query term into multiple sub-words, and query a sub-word sequence and candidate documents that match any of the sub-words from a document library; the sub-word sequence is obtained by segmenting documents.

[0153] The construction module 404 is used to construct multiple query sub-word sequences, wherein the sub-words in the query sub-word sequences are consecutive sub-words in the query terms;

[0154] The first determining module 406 is configured to, for any candidate document, determine segments from the sub-word sequence of the candidate document that match each query sub-word sequence, and determine the relevance score of the candidate document based on the inverse document frequency of each sub-word covered in each segment and the length of the segment;

[0155] The sorting module 408 is used to sort the candidate documents based on their relevance scores and output the sorted candidate documents.

[0156] In one embodiment, the sorting module 408 is further configured to:

[0157] The candidate documents are sorted based on their coverage factor and / or length penalty factor, and based on their relevance score.

[0158] The coverage factor is determined based on the number of sub-words matched by the candidate document in each sub-word corresponding to the query term;

[0159] The length penalty factor is determined based on the length of the word sequence of the candidate document.

[0160] In one embodiment, the device further includes:

[0161] The second determining module is used to determine non-repeating sub-words from each of the sub-words and to determine the number of non-repeating sub-words;

[0162] The third determining module is used to determine, for any candidate document, the number of matching subwords in the non-repeating subwords of the candidate document, and to determine the coverage factor of the candidate document based on the ratio between the number of matching subwords and the number of non-repeating subwords;

[0163] The coverage factor and the ratio have a superlinear relationship.

[0164] In one embodiment, the device further includes:

[0165] The fourth determining module is used to determine the global average sub-word sequence length based on the sub-word sequence length of each document in the document library;

[0166] The fifth determining module is used to determine the length penalty factor of any candidate document based on the ratio of the length of the sub-word sequence of the candidate document to the global average sub-word sequence length.

[0167] The length penalty factor and the ratio are negatively correlated.

[0168] In one embodiment, the first determining module 406 is further configured to:

[0169] The query sub-word sequences are traversed from longest to shortest. For the currently traversed query sub-word sequence, a segment matching the query sub-word sequence is retrieved from the sub-word sequence of the candidate document.

[0170] If no matching segment covers the segment, the segment is added as a matching segment; or if there is a matching segment that covers the segment, the segment is discarded, until all the query sub-word sequences have been traversed.

[0171] In one embodiment, the first determining module 406 is further configured to:

[0172] For any of the segments, a relevance score is determined based on the inverse document frequency of each of the sub-words covered in the segment and the length of the segment; wherein the relevance score of the segment has a sublinear relationship with the inverse document frequency of each of the sub-words and a superlinear relationship with the length of the segment;

[0173] The relevance score of the candidate document is determined based on the relevance scores of each segment contained in the candidate document.

[0174] Each module in the above-mentioned device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0175] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a document retrieval method.

[0176] Those skilled in the art will understand that Figure 5 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0177] In one embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0178] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0179] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0180] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0181] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0182] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0183] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A document retrieval method, characterized in that, The method includes: The system receives a query term, segments the query term into multiple sub-words, and retrieves a sub-word sequence and candidate documents matching any of the sub-words from a document database; the sub-word sequence is obtained by segmenting documents. Construct multiple query sub-word sequences, wherein the sub-words in the query sub-word sequences are consecutive sub-words in the query term; For any candidate document, segments matching each query sub-word sequence are determined from the sub-word sequence of the candidate document. Based on the inverse document frequency of each sub-word covered in each segment and the length of the segment, the relevance score of the candidate document is determined. Based on the relevance scores of each candidate document, the candidate documents are sorted and the sorted candidate documents are output.

2. The method according to claim 1, characterized in that, The process of ranking the candidate documents based on their relevance scores includes: The candidate documents are sorted based on their coverage factor and / or length penalty factor, and based on their relevance score. The coverage factor is determined based on the number of sub-words matched by the candidate document in each sub-word corresponding to the query term; The length penalty factor is determined based on the length of the word sequence of the candidate document.

3. The method according to claim 2, characterized in that, The method further includes: Identify the unique sub-words from each of the sub-words, and determine the number of unique sub-words; For any candidate document, determine the number of matching subwords in the non-repeating subwords of the candidate document, and determine the coverage factor of the candidate document based on the ratio between the number of matching subwords and the number of non-repeating subwords; The coverage factor and the ratio have a superlinear relationship.

4. The method according to claim 2, characterized in that, The method further includes: The global average subword sequence length is determined based on the subword sequence length of each document in the document library; For any of the candidate documents, the length penalty factor of the candidate document is determined based on the ratio of the length of the sub-word sequence of the candidate document to the global average sub-word sequence length. The length penalty factor and the ratio are negatively correlated.

5. The method according to claim 1, characterized in that, The step of determining the segments that match each of the query sub-word sequences from the sub-word sequences of the candidate documents includes: The query sub-word sequences are traversed from longest to shortest. For the currently traversed query sub-word sequence, a segment matching the query sub-word sequence is retrieved from the sub-word sequence of the candidate document. If no matching segment covers the segment, the segment is added as a matching segment; or if there is a matching segment that covers the segment, the segment is discarded, until all the query sub-word sequences have been traversed.

6. The method according to claim 1, characterized in that, The determination of the relevance score of the candidate document based on the inverse document frequency of each of the sub-words covered in each of the segments and the length of the segments includes: For any of the segments, a relevance score is determined based on the inverse document frequency of each of the sub-words covered in the segment and the length of the segment; wherein the relevance score of the segment has a sublinear relationship with the inverse document frequency of each of the sub-words and a superlinear relationship with the length of the segment; The relevance score of the candidate document is determined based on the relevance scores of each segment contained in the candidate document.

7. A document retrieval device, characterized in that, The device includes: The segmentation module is used to receive query terms, segment the query terms into multiple sub-words, and retrieve sub-word sequences and candidate documents matching any of the sub-words from the document library; the sub-word sequence is obtained by segmenting documents. A construction module is used to construct multiple query sub-word sequences, wherein the sub-words in the query sub-word sequences are consecutive sub-words in the query term; The first determining module is configured to, for any candidate document, determine segments from the sub-word sequence of the candidate document that match each query sub-word sequence, and determine the relevance score of the candidate document based on the inverse document frequency of each sub-word covered in each segment and the length of the segment; The sorting module is used to sort the candidate documents based on their relevance scores and output the sorted candidate documents.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images