Document retrieval method, device and equipment based on window multi-level extension, storage medium and program product

Abstract

The invention discloses a document retrieval method and device based on window multi-level expansion, equipment, a storage medium and a program product, and relates to the technical field of document retrieval, the document retrieval method comprises the following steps: segmenting a document to obtain a plurality of sub-documents, and configuring an initial window for the plurality of sub-documents; obtaining user query information, and performing initial retrieval based on the plurality of sub-documents and the user query information to obtain an initial similarity between the user query information and the initial window; expanding the initial window to obtain an expanded window and a corresponding expanded sub-document, and performing retrieval for multiple times according to the user query information and the expanded sub-document to obtain a reference similarity between the user query information and the expanded window; window expansion is completed based on the initial similarity and the reference similarity, the target expansion window and the corresponding target sub-document are obtained, document retrieval is completed, the retrieval process is optimized through a dynamic window expansion strategy, and the accuracy and correlation of a text retrieval system are improved.

Description

Technical Field

[0001] The present application relates to the technical field of document retrieval, and in particular to a document retrieval method, apparatus, device, storage medium and program product based on multi-level window expansion. Background Art

[0002] Traditional document retrieval methods usually rely on keyword matching or full-text retrieval, which have limitations when processing long documents or requiring contextual information, resulting in low document retrieval accuracy and efficiency. Summary of the invention

[0003] The main purpose of this application is to provide a document retrieval method, device, equipment, storage medium and program product based on window multi-level expansion, aiming to solve the technical problems of low accuracy and efficiency of document retrieval.

[0004] To achieve the above-mentioned purpose, the present application proposes a document retrieval method based on multi-level window expansion, and the document retrieval method based on multi-level window expansion includes:

[0005] Segmenting the document to obtain multiple sub-documents, and configuring initial windows for the multiple sub-documents;

[0006] Acquire user query information, perform a primary search based on the plurality of sub-documents and the user query information, and obtain an initial similarity between the user query information and the initial window;

[0007] Expanding the initial window to obtain an extended window and a corresponding extended sub-document, and performing multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window;

[0008] The window is expanded based on the initial similarity and the reference similarity, a target expanded window and a corresponding target sub-document are obtained, and document retrieval is completed.

[0009] In one embodiment, the step of extending the initial window to obtain an extended window and a corresponding extended sub-document, and performing multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window includes:

[0010] Acquire an extended sentence, the number of extensions of the extended sentence, and context information of the sub-document;

[0011] Expanding the initial window according to the expanded sentence, the expanded quantity and the context information to obtain an expanded window and a corresponding expanded sub-document;

[0012] Calculating an extended embedding vector of the extended window;

[0013] Obtaining a query sentence according to the user query information, and obtaining a query embedding vector of the query sentence;

[0014] Multiple searches are performed based on the extended embedding vector and the query embedding vector to obtain a reference similarity between the query embedding vector and the extended embedding vector.

[0015] In one embodiment, the step of completing window expansion based on the initial similarity and the reference similarity to obtain a target expansion window and a corresponding target sub-document and completing document retrieval includes:

[0016] comparing the initial similarity with the reference similarity;

[0017] When the difference between the initial similarity and the reference similarity is greater than a preset difference threshold, returning to the step of expanding the initial window to obtain an extended window and a corresponding extended sub-document, and performing multiple searches according to the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window, and obtaining the number of expansions;

[0018] When the number of expansions reaches a preset number threshold, the expansion is completed, the target expansion window and the corresponding target sub-document are obtained, and the document retrieval is completed.

[0019] In one embodiment, when the number of expansions reaches a preset number threshold, the expansion is completed, a target expansion window and a corresponding target sub-document are obtained, and the step of completing document retrieval includes:

[0020] When the number of expansions reaches a preset number threshold, the expansion is completed to obtain a plurality of candidate expansion windows;

[0021] Sorting the similarities between each of the candidate extension windows and the user query information to obtain a sorting result;

[0022] The candidate expansion window corresponding to the similarity greater than the preset similarity threshold in the sorting result is used as the target expansion window;

[0023] A target sub-document in the target extension window is obtained, and sentences in the target sub-document are used as target retrieval results.

[0024] In one embodiment, the step of obtaining the user query information, performing a primary search based on the plurality of sub-documents and the user query information, and obtaining an initial similarity between the user query information and the initial window comprises:

[0025] Acquire user query information, and obtain a query sentence according to the user query information;

[0026] Generate vector representations of sentences in the plurality of sub-documents by using a preset embedding model, and obtain an initial embedding vector of the initial window according to the vector representations;

[0027] Calculate the query embedding vector of the query sentence by using the preset embedding model;

[0028] An initial search is performed based on the initial embedding vector and the query embedding vector, and an initial similarity between the query embedding vector and the initial embedding vector is calculated.

[0029] In one embodiment, the step of segmenting the document to obtain a plurality of sub-documents and configuring initial windows for the plurality of sub-documents includes:

[0030] Obtaining a sentence order in a document, and segmenting the document according to the sentence order to obtain multiple sub-documents;

[0031] Obtaining sentence information in the plurality of sub-documents to obtain the center position of the sub-document;

[0032] An initial window size is obtained, and the initial window is configured according to the initial window size and the center position of the sub-document.

[0033] In addition, to achieve the above-mentioned purpose, the present application also proposes a document retrieval device based on multi-level window expansion, and the document retrieval device based on multi-level window expansion includes:

[0034] A segmentation module, used to segment a document to obtain multiple sub-documents, and configure initial windows for the multiple sub-documents;

[0035] A retrieval module, used to obtain user query information, perform an initial search based on the plurality of sub-documents and the user query information, and obtain an initial similarity between the user query information and the initial window;

[0036] An expansion module, used to expand the initial window to obtain an extended window and a corresponding extended sub-document, and perform multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window;

[0037] The acquisition module is used to complete the window expansion based on the initial similarity and the reference similarity, obtain the target expansion window and the corresponding target sub-document, and complete the document retrieval.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a document retrieval device based on multi-level window expansion, the device comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the document retrieval method based on multi-level window expansion as described above.

[0039] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the document retrieval method based on multi-level window expansion as described above are implemented.

[0040] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps of the document retrieval method based on multi-level window expansion as described above are implemented.

[0041] One or more technical solutions proposed in the present application divide a document into multiple sub-documents and configure an initial window for the multiple sub-documents; obtain user query information, perform an initial search based on the multiple sub-documents and the user query information, and obtain an initial similarity between the user query information and the initial window; expand the initial window to obtain an extended window and a corresponding extended sub-document, and perform multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window; complete window expansion based on the initial similarity and the reference similarity to obtain a target extended window and a corresponding target sub-document, complete document retrieval, and decide whether to continue expanding the window based on changes in similarity, thereby optimizing the retrieval process through a dynamic window expansion strategy, improving the accuracy and relevance of the text retrieval system, and ensuring that the retrieval results better meet the user's query needs and expectations. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0043] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0044] Figure 1 A flowchart diagram of the first embodiment of the document retrieval method based on multi-level window expansion of the present application;

[0045] Figure 2 A flowchart diagram of the second embodiment of the document retrieval method based on multi-level window expansion of the present application;

[0046] Figure 3 A flowchart diagram of the third embodiment of the document retrieval method based on multi-level window expansion of the present application;

[0047] Figure 4 A flowchart diagram of a fourth embodiment of a document retrieval method based on multi-level window expansion according to the present application;

[0048] Figure 5 This is a brief flowchart of an embodiment of a document retrieval method based on multi-level window expansion of the present application;

[0049] Figure 6 This is a schematic diagram of the module structure of a document retrieval device based on multi-level window expansion according to an embodiment of the present application;

[0050] Figure 7 Schematic diagram of the device structure of the hardware operating environment involved in the document retrieval method based on multi-level window expansion in the embodiment of the present application.

[0051] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0052] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0053] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0054] The main solution of the embodiment of the present application is: split the document to obtain multiple sub-documents, and configure an initial window for the multiple sub-documents; obtain user query information, perform an initial search based on the multiple sub-documents and the user query information, and obtain the initial similarity between the user query information and the initial window; expand the initial window to obtain an extended window and a corresponding extended sub-document, and perform multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window; complete window expansion based on the initial similarity and the reference similarity, obtain a target extended window and a corresponding target sub-document, and complete document retrieval.

[0055] Since existing technologies usually rely on keyword matching or full-text retrieval, these methods have limitations when processing long documents or requiring contextual information. With the development of deep learning technology, document representation methods based on embedding models can provide richer and more accurate semantic information, but how to efficiently use this information for document retrieval remains a challenge.

[0056] The present application provides a solution by dividing a document into multiple sub-documents, configuring an initial window for each sub-document, obtaining the embedding vector of the sentence, and then optimizing the retrieval process through a dynamic window expansion strategy. The method first calculates the similarity between the query sentence and the embedding vector of the sentence in the initial window, and then decides whether to continue to expand the window based on the change in similarity. Through this process, a most relevant candidate result is determined for each sub-document. Finally, multiple results are selected from all the candidate results based on relevance as the final retrieval result.

[0057] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a document retrieval device based on multi-level window expansion, etc. The following takes the document retrieval device based on multi-level window expansion as an example to illustrate this embodiment and the following embodiments.

[0058] Based on this, the embodiment of the present application provides a document retrieval method based on multi-level window expansion, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the document retrieval method based on multi-level window expansion of the present application.

[0059] In this embodiment, the document retrieval method based on window multi-level expansion includes steps S10 to S40:

[0060] Step S10: Segment the document to obtain multiple sub-documents, and configure initial windows for the multiple sub-documents.

[0061] It should be noted that the document can be segmented according to certain rules, so that the entire document can be segmented into multiple sub-documents. The document is a document to be retrieved, and the corresponding data is stored in the document. When the user has a retrieval demand, he or she can search the document and query the corresponding data in the document.

[0062] In a specific implementation, each sub-document contains several sentences, for example, each sub-document contains 10 sentences.

[0063] It is understandable that after obtaining multiple sub-documents, context information can be saved for each sub-document. The context information of a sub-document includes a certain number of sentences before and after the sub-document, for example, 5 sentences before and after, to ensure the context integrity of the sub-document.

[0064] The context information of a sub-document is the sentences in the previous sub-document and the next sub-document. If a sub-document is the first sub-document of a document, then the sub-document has no previous context information; if a sub-document is the last sub-document of a document, then the sub-document has no following context information.

[0065] For each sub-document, save its context information to facilitate subsequent window expansion.

[0066] In a specific implementation, the size of the initial window may be set for each sub-document, thereby configuring the initial window for multiple sub-documents.

[0067] In a feasible implementation, step S10 may include steps A11 to A13:

[0068] Step A11: Obtain the sentence sequence in the document, and segment the document according to the sentence sequence to obtain multiple sub-documents.

[0069] It should be understood that when segmenting a document, it can be segmented according to the order of sentences in the document, and the number of segments can also be set. For example, every 10 sentences are a segment segment. The number of segments can be set according to specific needs. By obtaining the order of sentences in the document, the segmentation can be performed according to the order of sentences. For example, every 10 sentences are segmented in sequence, thereby obtaining a segmented document, that is, multiple sub-documents.

[0070] Step A12: Acquire sentence information in the plurality of sub-documents to obtain the center position of the sub-document.

[0071] In a specific implementation, for each sub-document, the size and position of the initial window may be set.

[0072] For example, if a sub-document consists of [sentence (K), sentence (K+1), sentence (K+2), ..., sentence (N)], the center position of the sub-document can be obtained by adding the first sentence position (K) and the last sentence position (N), dividing by 2, and then rounding down the result.

[0073] Step A13: Acquire an initial window size, and configure the initial window according to the initial window size and the center position of the sub-document.

[0074] In a specific implementation, the initial window size can also be obtained. For example, the initial window size is set to include 5 sentences. For example: the sub-document consists of [sentence (K), sentence (K+1), sentence (K+2), ..., sentence (N)], then the center position of the sub-document can be obtained by adding the first sentence position (K) and the last sentence position (N) and dividing by 2, and then rounding down the calculation result.

[0075] For example, if a sub-document consists of [sentence (1), sentence (2), sentence (3), ..., sentence (10)], then the center position of the sub-document is sentence (5). Based on the center position, two sentences are expanded upward and downward respectively, and the resulting initial window includes [sentence (3), sentence (4), sentence (5), sentence (6), sentence (7)].

[0076] Directly segmenting by sentence can control the size of sub-documents more finely. Regardless of the length of the paragraph, the size of the sub-document (the number of sentences) can be kept consistent, which is convenient for subsequent processing. This is applicable to documents of various lengths and structures, and there is no need to consider the specific format of the paragraph. Document segmentation methods that do not rely on paragraph structure can avoid the problem of inconsistent paragraph divisions, especially in some documents with unclear paragraph divisions or complex paragraph structures. Some documents (such as online texts and informal documents) may not have clear paragraph divisions, and segmentation by sentence is more applicable. The method of segmenting documents based on paragraph semantics is itself a difficult task, and the solution of using the center position as the initial window and then expanding the window based on this is more universal.

[0077] Step S20: obtaining user query information, performing an initial search based on the plurality of sub-documents and the user query information, and obtaining an initial similarity between the user query information and the initial window.

[0078] It should be noted that the user query information may include the query statement that the user needs to query, and a first search may be performed based on multiple sub-documents and the user query statement, thereby calculating the initial similarity between the query statement and the initial window, that is, the initial correlation.

[0079] It can be understood that the initial similarity can be calculated by using a cosine similarity calculation method, so that the calculated cosine similarity is used as the initial similarity.

[0080] Step S30: Expand the initial window to obtain an extended window and a corresponding extended sub-document, and perform multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window.

[0081] Based on the initial window, the initial window can be expanded outward layer by layer to obtain an extended window and an extended sub-document corresponding to the extended window. Specifically, multiple searches can be performed based on the extended sub-document after expansion and the query statement in the user query information to obtain the reference similarity between the query statement and the extended window, that is, the reference correlation.

[0082] Step S40: completing window expansion based on the initial similarity and the reference similarity, obtaining a target expansion window and a corresponding target sub-document, and completing document retrieval.

[0083] It is understandable that the initial similarity and the reference similarity can be compared to determine whether to continue to expand downward or complete the expansion, obtain the final expanded target expansion window and the target sub-document corresponding to the target expansion window, and use the target sub-document as the final document retrieval result.

[0084] This embodiment provides a document retrieval method based on multi-level window expansion, which divides the document to obtain multiple sub-documents, and configures an initial window for the multiple sub-documents; obtains user query information, performs an initial search based on the multiple sub-documents and the user query information, and obtains the initial similarity between the user query information and the initial window; expands the initial window to obtain an extended window and a corresponding extended sub-document, and performs multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window; completes window expansion based on the initial similarity and the reference similarity, obtains a target extended window and a corresponding target sub-document, and completes document retrieval. Whether to continue to expand the window is determined according to the change in similarity, so as to optimize the retrieval process through a dynamic window expansion strategy, improve the accuracy and relevance of the text retrieval system, and ensure that the retrieval results better meet the user's query needs and expectations.

[0085] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 2 , step S30 includes steps S301 to S305:

[0086] Step S301: Acquire an extended sentence, the number of extensions of the extended sentence, and context information of the sub-document.

[0087] It should be noted that, based on the initial window, the window can be expanded outward layer by layer, so the sentences to be expanded, the number of expanded sentences, and the context information of the stored sub-documents can be obtained, and the number of sentences n added when expanding the window can be configured by yourself. For example, the value of n is comprehensively considered based on the precision and efficiency of the retrieval.

[0088] Step S302: Expand the initial window according to the expanded sentence, the expanded quantity and the context information to obtain an expanded window and a corresponding expanded sub-document.

[0089] In a specific implementation, the initial window can be expanded according to the expanded sentences, the expanded quantity and the context information, and the expanded sentences can be added to the initial window, and each expansion includes adding a certain number of sentences forward and backward. For example, each expansion adds 2 sentences forward and backward on the basis of the original sentences.

[0090] When expanding, you also need to consider boundary conditions, such as the boundary judgment and processing of the context information of the corresponding sub-documents in the collection. For example, there are sub-documents (10 sentences), and the context information is the last 5 sentences of the previous sub-document and the first 5 sentences of the following sub-document. When the window is expanded (expanded between sub-documents), its boundary is between these 20 sentences.

[0091] For example, if there is not enough context processing at the beginning or end, and if the sub-document at the beginning or end does not have enough sentences to expand, the similarity score between the current sub-document (less than 10 sentences) and the retrieval is directly returned as the result value, that is, no expansion is performed, and sentences are directly filtered based on the similarity between the current sub-document and the query sentence to obtain the filtering result, which is used as the retrieval result.

[0092] Step S303: Calculate the extended embedding vector of the extended window.

[0093] After each window expansion, the expanded embedding vector of the expanded window can be calculated.

[0094] In a feasible implementation manner, step S303 includes:

[0095] Acquire the number of sentences in the sub-document and the initial embedding vector of the initial window; calculate the sentence embedding vector of the extended sentence; and calculate the extended embedding vector of the extended window through the number of sentences, the extended number, the initial embedding vector and the sentence embedding vector.

[0096] The initial embedding vector of the initial window is recorded as V0. The initial window can be used as input by using an embedding model (such as BERT (Bidirectional Encoder Representations from Transformers, natural language processing model), GPT (Generative Pre-trained Transformer, generative pre-trained language model), etc.) to generate the initial embedding vector. The sentence embedding vector of the extended sentence can be obtained by the output of the embedding model. The extended embedding vector of the extended window is calculated by the number of sentences, the number of extensions, the initial embedding vector and the sentence embedding vector as shown in Formula 1:

[0097]

[0098] In the above formula 1, N0 is the number of sentences contained in the window before expansion (such as 5), that is, the number of sentences in the sub-document, n is the number of newly added sentences (such as 4), that is, the number of expansions, V new,i is the embedding vector of the newly added i-th sentence, that is, the sentence embedding vector, and V0 is the initial embedding vector.

[0099] Assume that the window before expansion contains 5 sentences, and record the embedding vector of the window as V0. Each time the window is expanded, n sentences are added to the original sentences. For example, if n is 4, 2 sentences are added forward and backward, for a total of 4 sentences. After the window is expanded, the current window contains 9 sentences (the original 5 sentences plus the newly added 4 sentences). Get the initial embedding vector V0 of the window before expansion, and get the sentence embedding vector V of the expanded sentence new , calculate the extended embedding vector of the extended window

[0100] Step S304: obtaining a query sentence according to the user query information, and acquiring a query embedding vector of the query sentence.

[0101] In a specific implementation, the query sentence can be obtained through the user query information, and the query sentence can be converted into a vector representation. For example, an embedding model (such as BERT, GPT, etc.) is used to take the query sentence as the input of the model to obtain the output vector, and then the embedding vector of the query sentence is obtained by the average pooling method, that is, the query embedding vector.

[0102] Step S305: performing multiple searches based on the extended embedding vector and the query embedding vector to obtain a reference similarity between the query embedding vector and the extended embedding vector.

[0103] In a specific implementation, multiple searches can be performed based on the extended embedding vector and the query embedding vector, that is, the reference similarity between the query embedding vector and the extended vector is calculated, for example, the reference similarity is calculated using cosine similarity, so that the correlation between the query question and the extended window can be evaluated based on the reference similarity.

[0104] This embodiment obtains an extended sentence, the number of extensions of the extended sentence, and the context information of the sub-document; expands the initial window according to the extended sentence, the number of extensions, and the context information to obtain an extended window and a corresponding extended sub-document; calculates an extended embedding vector of the extended window; obtains a query sentence according to the user query information, and obtains a query embedding vector of the query sentence; performs multiple searches according to the extended embedding vector and the query embedding vector to obtain a reference similarity between the query embedding vector and the extended embedding vector, configures an initial window on each sub-document based on a window expansion mechanism to capture the extended context information of the document at different levels, calculates the similarity between the query sentence and the sentence vector in the window, and as the window expands layer by layer, more context information is taken into consideration, thereby providing a more comprehensive and in-depth semantic understanding, which can improve the accuracy of the retrieval results.

[0105] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 , step S40 includes steps S401 to S403:

[0106] Step S401: Compare the initial similarity with the reference similarity.

[0107] It can be understood that the initial similarity can be compared with the reference similarity between the expanded window and the query sentence after the expansion, so as to determine whether the similarity has increased.

[0108] Step S402: When the difference between the initial similarity and the reference similarity is greater than a preset difference threshold, return to the step of expanding the initial window to obtain an extended window and a corresponding extended sub-document, and perform multiple searches based on the user query information and the extended sub-document to obtain the reference similarity between the user query information and the extended window, and obtain the number of expansions.

[0109] In a specific implementation, if the difference obtained by subtracting the initial similarity from the reference similarity is greater than a preset difference threshold, the preset difference threshold may be 0.05, 0.08, etc. The value of the preset difference threshold may be adjusted according to demand. When the difference is greater than the preset difference threshold, it proves that the correlation is increasing. Only when the correlation increases, a deeper context analysis is performed, so that the extended window is used as the initial window, and the initial window is expanded to obtain the extended window and the corresponding extended sub-document, and the step of performing multiple searches based on the user query information and the extended sub-document to obtain the reference similarity between the user query information and the extended window is returned, the initial window is continuously expanded, and the number of expansions is obtained.

[0110] In a specific implementation, if the similarity between the query sentence and the current window is not significantly improved after the current window expansion, for example, the difference is less than or equal to the preset difference threshold, the expansion is stopped and the current sub-document is directly returned as the result.

[0111] Step S403: When the number of expansions reaches a preset number threshold, the expansion is completed, a target expansion window and a corresponding target sub-document are obtained, and document retrieval is completed.

[0112] The preset number threshold can be set to 5 times, 8 times, etc. When the number of expansions reaches the preset number threshold, the expansion can be stopped, and the expanded window and the corresponding sub-document obtained by the current expansion are used as the target expanded window and the target sub-document corresponding to the target expanded window, thereby obtaining the document retrieval result and completing the document retrieval.

[0113] In a feasible implementation manner, step S403 includes steps B11 to B11:

[0114] Step B11: When the number of expansions reaches a preset number threshold, the expansion is completed to obtain a plurality of candidate expansion windows.

[0115] It should be noted that when the window expansion times reaches a set number, for example, 5 times, the expansion is completed, thereby obtaining a plurality of windows after the expansion is completed, that is, a plurality of candidate expansion windows.

[0116] Step B12: sorting the similarities between each of the candidate extension windows and the user query information to obtain a sorting result.

[0117] The similarity between each candidate expansion window and the user query information can be calculated and sorted to obtain a sorting result. The sorting can be from large to small. The greater the similarity, the higher the ranking.

[0118] Step B13: taking the candidate expansion window corresponding to the similarity greater than the preset similarity threshold in the sorting result as the target expansion window.

[0119] A preset similarity threshold or a cutoff position can be set to screen the candidate extension windows. Based on the relevance of each candidate extension window, K candidate extension windows with higher relevance are selected from multiple candidate extension windows to obtain a target extension window, that is, the candidate extension windows are sorted from high to low according to the relevance, and TopK candidate extension windows are selected therefrom, wherein each candidate extension window contains a corresponding sub-document. K is a natural number of 1 or greater.

[0120] Step B14: Acquire the target sub-document in the target extension window, and use the sentences in the target sub-document as target retrieval results.

[0121] In a specific implementation, the target sub-documents corresponding to the target extension window may be obtained, each sub-document containing multiple sentences, so that the sentences in each target sub-document are used as the final target retrieval results.

[0122] The target retrieval result is the sentence contained in the window with the highest relevance during the retrieval and recall process for the target sub-document.

[0123] This embodiment compares the initial similarity with the reference similarity; when the difference between the initial similarity and the reference similarity is greater than a preset difference threshold, the method returns to the step of expanding the initial window to obtain an expanded window and a corresponding expanded sub-document, and performs multiple searches based on the user query information and the expanded sub-document to obtain a reference similarity between the user query information and the expanded window, and obtains the number of expansions; when the number of expansions reaches a preset number threshold, the expansion is completed, the target expanded window and the corresponding target sub-document are obtained, and the document retrieval is completed. By dynamically adjusting the size of the retrieval window to better match the user's query needs, the accuracy of the retrieval is improved, and the response speed is improved. By setting the stop condition, the expansion depth of the context information can be controlled.

[0124] Based on the first embodiment of the present application, in the fourth embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above description, and will not be described in detail later. Figure 4 , step S20 includes steps S201 to S204:

[0125] Step S201: Obtain user query information, and obtain a query sentence according to the user query information.

[0126] It should be noted that user query information may be obtained and parsed to obtain a query sentence, that is, the content that the user needs to query.

[0127] Step S202: Generate vector representations of the sentences in the plurality of sub-documents by using a preset embedding model, and obtain an initial embedding vector of the initial window according to the vector representations.

[0128] The preset embedding model can be BERT, GPT, etc., which generates a vector representation of each sentence by taking each sentence of each sub-document in multiple sub-documents as the input of the embedding model, and obtains the context embedding vector of the initial window of each sub-document according to the vector representation of each sentence, that is, the initial embedding vector.

[0129] Specifically, the following scheme is adopted to determine the embedding vector of the window: the average vector method is used to take the average value of the embedding vectors of each sentence contained in the window, and the average value is used as the initial embedding vector of the window.

[0130] Step S203: Calculate the query embedding vector of the query sentence using the preset embedding model.

[0131] In a specific implementation, the query sentence can be used as the input of the embedding model to obtain the output vector, and then the embedding vector of the query sentence is obtained through the average pooling method.

[0132] Step S204: Performing a first search based on the initial embedding vector and the query embedding vector, and calculating an initial similarity between the query embedding vector and the initial embedding vector.

[0133] In a specific implementation, an initial search may be performed based on the initial embedding vector and the query embedding vector, and the cosine similarity between the query embedding vector and the initial embedding vector may be calculated using cosine similarity to obtain an initial similarity.

[0134] This embodiment obtains user query information, and obtains a query sentence based on the user query information; generates vector representations of sentences in multiple sub-documents through a preset embedding model, and obtains an initial embedding vector of the initial window based on the vector representation; calculates the query embedding vector of the query sentence through the preset embedding model; performs an initial search based on the initial embedding vector and the query embedding vector, and calculates the initial similarity between the query embedding vector and the initial embedding vector. By converting the text into a vector representation, semantic matching and similarity calculation can be performed more effectively. The selection of the embedding model and the similarity calculation method can improve the accuracy of the retrieval results.

[0135] For example, in order to help understand the implementation process of the document retrieval method based on window multi-level expansion obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 5 , Figure 5 A brief flowchart of a document retrieval method based on multi-level window expansion is provided, specifically:

[0136] Step 1 is to segment the document and configure the initial window for the sub-documents obtained by segmentation. Step 2 is to obtain the embedding vector of the sentence in the sub-document and perform the first search. Step 3 is to expand the window and perform multiple searches. Step 4 is to generate the final search results.

[0137] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the document retrieval method based on multi-level window expansion of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0138] This application also provides a document retrieval device based on multi-level window expansion, please refer to Figure 6 , the document retrieval device based on window multi-level expansion includes:

[0139] The segmentation module 10 is used to segment the document to obtain multiple sub-documents and configure initial windows for the multiple sub-documents.

[0140] The retrieval module 20 is used to obtain user query information, perform an initial search based on the multiple sub-documents and the user query information, and obtain an initial similarity between the user query information and the initial window.

[0141] The expansion module 30 is used to expand the initial window to obtain an extended window and a corresponding extended sub-document, and perform multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window.

[0142] The acquisition module 40 is used to complete the window expansion based on the initial similarity and the reference similarity, obtain the target expansion window and the corresponding target sub-document, and complete the document retrieval.

[0143] The document retrieval device based on multi-level window expansion provided by the present application adopts the document retrieval method based on multi-level window expansion in the above embodiment, which can solve the technical problems of low accuracy and efficiency of document retrieval. Compared with the prior art, the beneficial effects of the document retrieval device based on multi-level window expansion provided by the present application are the same as the beneficial effects of the document retrieval method based on multi-level window expansion provided by the above embodiment, and other technical features in the document retrieval device based on multi-level window expansion are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0144] In one embodiment, the expansion module 30 is further used to obtain an expanded sentence, an expanded number of the expanded sentence, and context information of the sub-document; expand the initial window according to the expanded sentence, the expanded number, and the context information to obtain an expanded window and a corresponding expanded sub-document; calculate an expanded embedding vector of the expanded window; obtain a query sentence according to the user query information, and obtain a query embedding vector of the query sentence; perform multiple searches based on the expanded embedding vector and the query embedding vector to obtain a reference similarity between the query embedding vector and the expanded embedding vector.

[0145] In one embodiment, the acquisition module 40 is also used to compare the initial similarity with the reference similarity; when the difference between the initial similarity and the reference similarity is greater than a preset difference threshold, return to the step of expanding the initial window to obtain an extended window and a corresponding extended sub-document, and perform multiple searches based on the user query information and the extended sub-document to obtain the reference similarity between the user query information and the extended window, and obtain the number of expansions; when the number of expansions reaches a preset number threshold, complete the expansion, obtain the target extended window and the corresponding target sub-document, and complete the document retrieval.

[0146] In one embodiment, the acquisition module 40 is further used to complete the expansion when the number of expansions reaches a preset number threshold, and obtain multiple candidate expansion windows; sort the similarities between each of the candidate expansion windows and the user query information to obtain a sorting result; use the candidate expansion window corresponding to the similarity greater than the preset similarity threshold in the sorting result as the target expansion window; obtain the target sub-document in the target expansion window, and use the sentence in the target sub-document as the target retrieval result.

[0147] In one embodiment, the retrieval module 20 is further used to obtain user query information, and obtain a query sentence based on the user query information; generate vector representations of sentences in multiple sub-documents through a preset embedding model, and obtain an initial embedding vector of the initial window based on the vector representation; calculate the query embedding vector of the query sentence through the preset embedding model; perform an initial retrieval based on the initial embedding vector and the query embedding vector, and calculate the initial similarity between the query embedding vector and the initial embedding vector.

[0148] In one embodiment, the segmentation module 10 is also used to obtain the sentence order in the document, segment the document according to the sentence order, and obtain multiple sub-documents; obtain sentence information in the multiple sub-documents to obtain the center position of the sub-documents; obtain the initial window size, and configure the initial window according to the initial window size and the center position of the sub-document.

[0149] The present application provides a document retrieval device based on multi-level window expansion, and the document retrieval device based on multi-level window expansion includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the document retrieval method based on multi-level window expansion in the above-mentioned embodiment one.

[0150] Reference below Figure 7 , which shows a schematic diagram of the structure of a document retrieval device based on window multi-level expansion suitable for implementing the embodiment of the present application. The document retrieval device based on window multi-level expansion in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 7 The document retrieval device based on multi-level window expansion shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0151] like Figure 7As shown, the document retrieval device based on multi-level window expansion may include a processing device 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the document retrieval device based on multi-level window expansion are also stored in the RAM 1004. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems may be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 may allow the document retrieval device based on multi-level extension of windows to communicate wirelessly or wired with other devices to exchange data. Although the document retrieval device based on multi-level extension of windows with various systems is shown in the figure, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have alternatively.

[0152] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0153] The document retrieval device based on multi-level window expansion provided by the present application adopts the document retrieval method based on multi-level window expansion in the above embodiment, which can solve the technical problems of low accuracy and efficiency of document retrieval. Compared with the prior art, the beneficial effects of the document retrieval device based on multi-level window expansion provided by the present application are the same as the beneficial effects of the document retrieval method based on multi-level window expansion provided by the above embodiment, and other technical features in the document retrieval device based on multi-level window expansion are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0154] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0155] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0156] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the document retrieval method based on multi-level window expansion in the above-mentioned embodiment.

[0157] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0158] The computer-readable storage medium may be included in the document retrieval device based on the multi-level expansion of the window; or may exist independently without being assembled into the document retrieval device based on the multi-level expansion of the window.

[0159] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by a document retrieval device based on window multi-level expansion, the document retrieval device based on window multi-level expansion: divides the document to obtain multiple sub-documents, and configures an initial window for the multiple sub-documents; obtains user query information, performs an initial search based on the multiple sub-documents and the user query information, and obtains an initial similarity between the user query information and the initial window; expands the initial window to obtain an extended window and a corresponding extended sub-document, and performs multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window; completes window expansion based on the initial similarity and the reference similarity, obtains a target extended window and a corresponding target sub-document, and completes document retrieval.

[0160] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0161] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0162] The modules involved in the embodiments described in this application may be implemented by software or hardware, wherein the name of the module does not constitute a limitation on the unit itself in some cases.

[0163] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned document retrieval method based on multi-level window expansion, and can solve the technical problems of low accuracy and efficiency of document retrieval. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the document retrieval method based on multi-level window expansion provided by the above-mentioned embodiment, and will not be repeated here.

[0164] The present application also provides a computer program product, including a computer program, which implements the steps of the document retrieval method based on multi-level window expansion as described above when executed by a processor.

[0165] The computer program product provided by the present application can solve the technical problems of low accuracy and efficiency of document retrieval. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the document retrieval method based on multi-level window expansion provided by the above embodiment, which will not be described in detail here.

[0166] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A document retrieval method based on multi-level window expansion, characterized in that: The document retrieval method based on window multi-level expansion includes: Segmenting the document to obtain multiple sub-documents, and configuring initial windows for the multiple sub-documents; Acquire user query information, perform a primary search based on the plurality of sub-documents and the user query information, and obtain an initial similarity between the user query information and the initial window; Expanding the initial window to obtain an extended window and a corresponding extended sub-document, and performing multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window; The window is expanded based on the initial similarity and the reference similarity, a target expanded window and a corresponding target sub-document are obtained, and document retrieval is completed.

2. The method according to claim 1, characterized in that The step of extending the initial window to obtain an extended window and a corresponding extended sub-document, and performing multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window includes: Acquire an extended sentence, the number of extensions of the extended sentence, and context information of the sub-document; Expanding the initial window according to the expanded sentence, the expanded quantity and the context information to obtain an expanded window and a corresponding expanded sub-document; Calculating an extended embedding vector of the extended window; Obtaining a query sentence according to the user query information, and obtaining a query embedding vector of the query sentence; Multiple searches are performed based on the extended embedding vector and the query embedding vector to obtain a reference similarity between the query embedding vector and the extended embedding vector.

3. The method according to claim 1, characterized in that The step of completing window expansion based on the initial similarity and the reference similarity to obtain a target expansion window and a corresponding target sub-document and completing document retrieval comprises: comparing the initial similarity with the reference similarity; When the difference between the initial similarity and the reference similarity is greater than a preset difference threshold, returning to the step of expanding the initial window to obtain an extended window and a corresponding extended sub-document, and performing multiple searches according to the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window, and obtaining the number of expansions; When the number of expansions reaches a preset number threshold, the expansion is completed, the target expansion window and the corresponding target sub-document are obtained, and the document retrieval is completed.

4. The method according to claim 3, characterized in that When the number of expansions reaches a preset number threshold, the expansion is completed, and a target expansion window and a corresponding target sub-document are obtained. The step of completing document retrieval includes: When the number of expansions reaches a preset number threshold, the expansion is completed to obtain a plurality of candidate expansion windows; Sorting the similarities between each of the candidate extension windows and the user query information to obtain a sorting result; The candidate expansion window corresponding to the similarity greater than the preset similarity threshold in the sorting result is used as the target expansion window; A target sub-document in the target extension window is obtained, and sentences in the target sub-document are used as target retrieval results.

5. The method according to claim 1, characterized in that The step of obtaining user query information, performing a primary search based on the plurality of sub-documents and the user query information, and obtaining an initial similarity between the user query information and the initial window comprises: Acquire user query information, and obtain a query sentence according to the user query information; Generate vector representations of sentences in the plurality of sub-documents by using a preset embedding model, and obtain an initial embedding vector of the initial window according to the vector representations; Calculate the query embedding vector of the query sentence by using the preset embedding model; An initial search is performed based on the initial embedding vector and the query embedding vector, and an initial similarity between the query embedding vector and the initial embedding vector is calculated.

6. The method according to any one of claims 1 to 5, characterized in that The step of dividing the document to obtain a plurality of sub-documents and configuring initial windows for the plurality of sub-documents includes: Obtaining a sentence order in a document, and segmenting the document according to the sentence order to obtain multiple sub-documents; Obtaining sentence information in the plurality of sub-documents to obtain the center position of the sub-document; An initial window size is obtained, and the initial window is configured according to the initial window size and the center position of the sub-document.

7. A document retrieval device based on multi-level window expansion, characterized in that: The device comprises: A segmentation module, used to segment a document to obtain multiple sub-documents, and configure initial windows for the multiple sub-documents; A retrieval module, used to obtain user query information, perform an initial search based on the plurality of sub-documents and the user query information, and obtain an initial similarity between the user query information and the initial window; An expansion module, used to expand the initial window to obtain an extended window and a corresponding extended sub-document, and perform multiple searches based on the user query information and the extended sub-document to obtain a reference similarity between the user query information and the extended window; The acquisition module is used to complete the window expansion based on the initial similarity and the reference similarity, obtain the target expansion window and the corresponding target sub-document, and complete the document retrieval.

8. A document retrieval device based on multi-level window expansion, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the document retrieval method based on window multi-level expansion as claimed in any one of claims 1 to 6.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the document retrieval method based on multi-level window expansion as described in any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the document retrieval method based on multi-level window expansion as claimed in any one of claims 1 to 6 are implemented.

Images