Document retrieval method, device and storage medium

Abstract

The present application discloses a document retrieval method, device and storage medium, which relates to the technical field of natural language processing. The method includes: pre-processing a plurality of acquired documents to be processed to obtain corresponding target text data; recursively segmenting the paragraphs included in the target text data according to preset character intervals to divide the target text data into a plurality of parent fragments; segmenting each parent fragment into a plurality of sub-fragments, and generating a sub-fragment vector corresponding to each sub-fragment, and storing it in a vector database; searching the vector database for a target sub-fragment vector that matches the search data input by the user, so as to determine the search result corresponding to the search data according to the parent fragment corresponding to the target sub-fragment vector. By taking the paragraph as the base number and the preset character interval as the basis for the segmentation interval, the target text data is divided into fragments of appropriate size, which can not only retain more context information, but also improve the calculation efficiency and the search accuracy.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a document retrieval method, device and storage medium. Background Art

[0002] In the evolution of artificial intelligence and machine learning, natural language processing (NLP) has always been a key area. Large language model (LLM), as a powerful tool in NLP, can generate high-quality natural language text through training of large-scale corpus. However, with the increase of data volume and the diversification of user needs, LLM has encountered bottlenecks in answering knowledge in certain professional fields. To solve this problem, retrieval augmented generation (RAG) technology came into being. RAG combines information retrieval and language generation, and improves the accuracy and relevance of answers by retrieving relevant documents and integrating them into the generation model. Although RAG technology has improved a lot in answering specific questions, if the text block of the document segment is large, when users use the question for similarity search and matching, it will cause large amount of calculation and slow retrieval. If the text block of the document segment is small, the information content of the found segment is small, and there is a lack of sensitivity to the context, resulting in the generated answer being inaccurate and not highly relevant, which limits the quality of information answers. Summary of the invention

[0003] The purpose of the embodiments of the present application is to provide a document retrieval method, device and storage medium, so as to solve the problems in the prior art such as slow retrieval, incorrect retrieval fragments, incomplete generated content or repeated generated content.

[0004] In order to achieve the above-mentioned object, the first aspect of the present application provides a document retrieval method, the method comprising:

[0005] Get multiple documents to be processed;

[0006] Preprocessing multiple documents to be processed to obtain corresponding target text data;

[0007] Obtaining a preset character interval for slicing target text data;

[0008] Recursively segmenting the paragraphs included in the target text data according to preset character intervals to divide the target text data into a plurality of parent fragments;

[0009] Split each parent shard into multiple sub-shards, and generate a sub-shard vector corresponding to each sub-shard;

[0010] Store all sub-slice vectors into the vector database;

[0011] Get the search data entered by the user;

[0012] The target sub-shard vector matching the search data is searched in the vector database to determine the search result corresponding to the search data according to the parent shard corresponding to the target sub-shard vector.

[0013] In an embodiment of the present application, a recursive segmentation operation is performed on the paragraphs included in the target text data according to a preset character interval to divide the target text data into multiple mother fragments, including: recursively performing a segmentation operation on the paragraphs included in the target text data to determine the number of first characters included in the recursive current paragraph; when the number of first characters is less than the preset character interval, recursively performing the segmentation operation on the target text data starting from the current paragraph; determining in real time whether there is a preset separator in the paragraphs recursively starting from the current paragraph, and determining in real time the number of second characters included in multiple paragraphs recursively starting from the current paragraph; when there is a preset separator in the paragraphs recursively starting from the current paragraph, segmenting all paragraphs included between the current paragraph and the paragraph corresponding to the preset separator to obtain corresponding mother fragments; when the number of second characters is greater than the preset character interval, segmenting multiple paragraphs recursively starting from the current paragraph to obtain corresponding mother fragments.

[0014] In an embodiment of the present application, the method further includes: when the number of first characters is greater than or equal to a preset character interval, segmenting the current paragraph into corresponding parent segments.

[0015] In an embodiment of the present application, preprocessing multiple documents to be processed to obtain corresponding target text data includes: performing data cleaning on multiple documents to be processed to obtain text data corresponding to each document to be processed; and superimposing all text data based on preset separators to obtain corresponding target text data.

[0016] In an embodiment of the present application, searching a vector database for a target sub-shard vector matching the retrieval data to determine a retrieval result corresponding to the retrieval data according to the mother shard corresponding to the target sub-shard vector includes: generating a retrieval vector corresponding to the retrieval data; searching a vector database for multiple target sub-shard vectors matching the retrieval vector; determining a first target mother shard corresponding to the retrieval vector according to the mother shards corresponding to the multiple target sub-shard vectors; storing the multiple mother shards in a mother shard list in order; for each first target mother shard, merging the mother shards adjacent to the first target mother shard in the mother shard list with the first target mother shard to generate a second target mother shard; generating a prompt word according to all the second target mother shards and the retrieval data; and inputting the prompt word into a preset large language model to output a retrieval result corresponding to the prompt word through the preset large language model as the retrieval result corresponding to the retrieval data.

[0017] In an embodiment of the present application, determining the first target mother shard corresponding to a retrieval vector based on the mother shards corresponding to multiple target sub-shard vectors includes: determining the mother shard content and the mother shard position of the mother shard corresponding to each target sub-shard vector; and deduplicating the mother shards that have the same mother shard content and / or the same mother shard position in the mother shards corresponding to all the target sub-shard vectors to obtain the first target mother shard corresponding to the retrieval vector.

[0018] In an embodiment of the present application, generating a retrieval vector corresponding to retrieval data includes: rewriting the retrieval data to generate multiple similar retrieval data corresponding to the retrieval data; inputting the retrieval data and each similar retrieval data into a preset embedding model respectively, so as to output corresponding retrieval vectors respectively through the preset embedding model.

[0019] In an embodiment of the present application, generating prompt words based on all second target mother shards and retrieval data includes: inputting all second target mother shards and retrieval data into a preset rearrangement model to determine the similarity between each second target mother shard and the retrieval data through the preset rearrangement model; sorting all second target mother shards from large to small according to the similarity to output the first N second target mother shards through the preset rearrangement model, where N is a positive integer; inputting the first N second target mother shards and retrieval data into a preset prompt project to generate prompt words through the preset prompt project.

[0020] In an embodiment of the present application, the mother shards adjacent to the first target mother shard are two mother shards that are arranged after the first target mother shard in the mother shard list.

[0021] In an embodiment of the present application, searching a vector database for multiple target sub-slice vectors that match a search vector includes: determining the cosine similarity between each sub-slice vector included in the vector database and the search vector; sorting all sub-slice vectors from large to small according to the cosine similarity to determine the first M sub-slice vectors as target sub-slice vectors that match the search vector, where M is a positive integer.

[0022] A second aspect of the present application provides a document retrieval device, comprising:

[0023] a memory configured to store instructions;

[0024] The processor is configured to call instructions from the memory and implement the document retrieval method according to the above when executing the instructions.

[0025] A third aspect of the present application provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the document retrieval method described above.

[0026] Through the above technical solution, the paragraphs included in the target text data are recursively segmented according to the preset character intervals, so that the target text data generated by combining different documents is divided into multiple parent fragments, and then each parent fragment is divided into multiple sub-fragments, and a sub-fragment vector corresponding to each sub-fragment is generated and stored in the vector database. Therefore, in the retrieval process, the matching target sub-fragment vector is searched in the vector database to determine the retrieval result corresponding to the retrieval data according to the parent fragment corresponding to the target sub-fragment vector. By taking the paragraph as the base number and the preset character interval as the basis for the segmentation interval, the target text data is divided into fragments of appropriate size, which can not only retain more context information, but also improve the calculation efficiency and the retrieval accuracy.

[0027] Other features and advantages of the embodiments of the present application will be described in detail in the subsequent specific implementation section. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the following specific implementations, they are used to explain the embodiments of the present application, but do not constitute a limitation on the embodiments of the present application. In the accompanying drawings:

[0029] Figure 1 A schematic diagram of a process flow of a document retrieval method according to an embodiment of the present application is schematically shown;

[0030] Figure 2 A schematic diagram schematically shows document segmentation according to an embodiment of the present application;

[0031] Figure 3A structural block diagram of a document retrieval device according to an embodiment of the present application is schematically shown;

[0032] Figure 4 The schematic diagram shows a structure of a computer device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the specific implementation methods described herein are only used to illustrate and explain the embodiments of the present application, and are not used to limit the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0034] In addition, if there are descriptions involving "first", "second", etc. in the embodiments of the present application, the descriptions of "first", "second", etc. are only used for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by this application.

[0035] Figure 1 The following schematically shows a flow chart of a document retrieval method according to an embodiment of the present application. Figure 1 As shown, an embodiment of the present application provides a document retrieval method, which may include the following steps.

[0036] S102, obtaining a plurality of documents to be processed.

[0037] Specifically, professional data documents of various formats and sizes can be searched on the Internet as documents to be processed, thereby improving the diversity and comprehensiveness of the data.

[0038] S104, pre-processing the multiple documents to be processed to obtain corresponding target text data.

[0039] It can be understood that the target text data is the text data generated by integrating multiple documents to be processed. Specifically, different data conversion methods can be applied to documents of different formats to obtain corresponding text data. Then, some irrelevant information and noise are removed by corresponding text cleaning methods to obtain corresponding clean and standardized text data. Finally, the obtained text data are superimposed to obtain the final target text data.

[0040] S106, obtaining a preset character interval for segmenting the target text data.

[0041] S108, performing a recursive segmentation operation on the paragraphs included in the target text data according to a preset character interval, so as to divide the target text data into a plurality of parent fragments.

[0042] It can be understood that the preset character interval refers to the character interval preset by the technician. For example, if the preset character interval is n characters, then recursively start from the first character of the target text data to the nth character, segment the paragraphs included in these n characters, and then further recursively segment n characters, repeat the recursive segmentation operation, so as to divide the target text data into multiple mother fragments. The mother fragment is a fragment in the target text data, which is composed of one or more paragraphs of the target text data.

[0043] S110, split each mother shard into multiple sub-shards, and generate a sub-shard vector corresponding to each sub-shard.

[0044] Specifically, the mother slice can be divided into multiple sub-slices based on character intervals, or based on words or sentences one by one. For each sub-slice, a corresponding sub-slice vector can be generated. Specifically, the sub-slice can be mapped to a real vector through natural language processing technology, and the vocabulary can be converted from discrete, high-dimensional one-hot encoding to continuous, low-dimensional vector representation through mathematical methods, so that the relationship between words can be measured by the similarity between vectors. Specifically, the sub-slices are encoded into sub-slice vectors, which can represent the semantic information of the text and are a form of expression of text data in mathematical space. The dimension of the sub-slice vector represents the feature space of the text. The dimension and value of the vector can reflect the similarity and difference between texts. Specifically, any one of the embedding model, the word2vec model (a related model for generating word vectors), and the BERT pre-trained model (BidirectionalEncoder Representations from Transformers, a bidirectional encoder transformer based on the Transformer model) can be used to determine the sub-slice vector corresponding to each sub-slice. Embedding models include ge-m3 model, Text-Embedding-Ada-002 model, etc., to facilitate fine-tuning of various tests. Figure 2 As shown, the input target text data is divided into parent shard 1, parent shard 2, parent shard 3... wherein parent shard 2 can be divided into sub-shard 1, sub-shard 2, sub-shard 3...

[0045] S112, storing all sub-slice vectors into a vector database.

[0046] Specifically, each row of data in the vector database includes the id, each sub-shard, the shard vector corresponding to the sub-shard, and metadata (a json format that stores the corresponding parent shard content and parent shard location), and then establishes a vector index for efficient similarity search. The vector database can be the Faiss vector library, the Milvus vector library, or the Weaviate vector library for various test fine-tuning.

[0047] S114, obtaining search data input by the user.

[0048] S116, searching the vector database for a target sub-shard vector that matches the search data, so as to determine a search result corresponding to the search data according to the parent shard corresponding to the target sub-shard vector.

[0049] Specifically, during the retrieval process, the retrieval data input by the user can be vectorized to obtain a retrieval vector, and then calculated based on the distance between the vectors. For example, by calculating the distance between the retrieval vector and the sharding vector stored in the vector database, such as the Euclidean distance or cosine similarity, the sharding vector most similar to the retrieval vector is found in the vector database, that is, the target sub-sharding vector. Among them, there can be multiple target sub-sharding vectors. The recalled mother shard is determined based on the target sub-sharding vector, and the retrieval result corresponding to the retrieval data is determined based on the mother shard.

[0050] Through the above technical solution, the paragraphs included in the target text data are recursively segmented according to the preset character intervals, so that the target text data generated by combining different documents is divided into multiple parent fragments, and then each parent fragment is divided into multiple sub-fragments, and a sub-fragment vector corresponding to each sub-fragment is generated and stored in the vector database. Therefore, in the retrieval process, the matching target sub-fragment vector is searched in the vector database to determine the retrieval result corresponding to the retrieval data according to the parent fragment corresponding to the target sub-fragment vector. By taking the paragraph as the base number and the preset character interval as the basis for the segmentation interval, the target text data is divided into fragments of appropriate size, which can not only retain more context information, but also improve the calculation efficiency and the retrieval accuracy.

[0051] In an embodiment of the present application, preprocessing multiple documents to be processed to obtain corresponding target text data includes: performing data cleaning on multiple documents to be processed to obtain text data corresponding to each document to be processed; and superimposing all text data based on preset separators to obtain corresponding target text data.

[0052] It can be understood that Retrieval Augmented Generation (RAG) combines information retrieval and language generation, and improves the accuracy and relevance of answers by retrieving relevant documents and integrating them into the generation model. However, RAG technology is difficult to adapt to documents of various types and sizes, and is difficult to integrate with other advanced models. The process is too rigid, which limits the application scope and expansion capabilities of the system. The preset delimiter is used by technicians to superimpose multiple texts to be processed and distinguish the set delimiter. Specifically, different data conversion methods can be applied to documents of different formats to obtain corresponding text data. Then, some irrelevant information and noise are removed through the corresponding text cleaning method, and the corresponding clean and standardized text data is obtained. A specific delimiter "======" is added to superimpose the obtained text data to obtain the final target text data. In this way, RAG technology can adapt to mixed input of documents of different formats and sizes.

[0053] In an embodiment of the present application, a recursive segmentation operation is performed on the paragraphs included in the target text data according to a preset character interval to divide the target text data into multiple mother fragments, including: recursively performing a segmentation operation on the paragraphs included in the target text data to determine the number of first characters included in the recursive current paragraph; when the number of first characters is less than the preset character interval, recursively performing the segmentation operation on the target text data starting from the current paragraph; determining in real time whether there is a preset separator in the paragraphs recursively starting from the current paragraph, and determining in real time the number of second characters included in multiple paragraphs recursively starting from the current paragraph; when there is a preset separator in the paragraphs recursively starting from the current paragraph, segmenting multiple paragraphs included between the current paragraph and the paragraph corresponding to the preset separator to obtain corresponding mother fragments; when the number of second characters is greater than the preset character interval, segmenting all paragraphs recursively starting from the current paragraph to obtain corresponding mother fragments.

[0054] In an embodiment of the present application, the method further includes: when the number of first characters is greater than or equal to a preset character interval, segmenting the current paragraph into corresponding parent segments.

[0055] It can be understood that the first number of characters refers to the number of characters included in the current paragraph when recursively starting from the current paragraph. The second number of characters refers to the number of characters included in multiple paragraphs recursively starting from the current paragraph. The second number of characters needs to be determined in real time. In the case where the first number of characters is greater than or equal to the preset character interval, the current paragraph can be segmented into a mother segment. In the case where the first number of characters is less than the preset character interval, it is necessary to continue to recurse the paragraphs included in the target text data until the second number of characters included in the multiple paragraphs recursively starting from the current paragraph is greater than or equal to the preset character interval or there is a preset separator in the paragraph recursively starting from the current paragraph. When the recursion reaches the case where the second number of characters is greater than the preset character interval, the multiple paragraphs recursively starting from the current paragraph can be segmented to obtain the corresponding mother segment. Among them, the multiple paragraphs recursively starting from the current paragraph include all paragraphs from the current paragraph to the paragraph corresponding to the second number of characters greater than the preset character interval, and include the current paragraph but do not include the paragraph corresponding to the second number of characters greater than the preset character interval. When there is a preset delimiter in the paragraphs recursively starting from the current paragraph, the current paragraph and the multiple paragraphs included between the preset delimiter are segmented to obtain the corresponding parent fragment. Among them, the multiple paragraphs included between the current paragraph and the paragraph corresponding to the preset delimiter include the current paragraph, but do not include the paragraph corresponding to the preset delimiter. The data in different documents usually involve different contents. Separating the data in different documents for fragmentation can reduce the error rate of retrieval.

[0056] Specifically, the preset character interval is n characters, and n can be adjusted at will, making our model more flexible. We take the content between the previous line break and the next line break as a paragraph. Initially, the starting point L is set as the position before the first character of the first paragraph of the document. Then, we determine whether the number of characters from the starting point L to the next paragraph exceeds k. If it does not exceed K, the paragraph is added to the next paragraph for re-judgment. If it exceeds k and only one paragraph is recursed, the paragraph is divided into a mother fragment, L is positioned before the next paragraph of the paragraph, and the above operation is repeated. If it exceeds k and more than one paragraph is recursed, the content from L to the first character of the paragraph is taken as a mother fragment, L is positioned before the first character of the paragraph, and the above operation is repeated. If the next paragraph is a specific delimiter "======", regardless of whether it exceeds k, the content from L to the first character of the paragraph is divided into a mother fragment, L is repositioned to the first character of the paragraph after the specific delimiter, and the above operation is repeated.

[0057] In an embodiment of the present application, searching a vector database for a target sub-shard vector matching the retrieval data to determine a retrieval result corresponding to the retrieval data according to the mother shard corresponding to the target sub-shard vector includes: generating a retrieval vector corresponding to the retrieval data; searching a vector database for multiple target sub-shard vectors matching the retrieval vector; determining a first target mother shard corresponding to the retrieval vector according to the mother shards corresponding to the multiple target sub-shard vectors; storing the multiple mother shards in a mother shard list in order; for each first target mother shard, merging the mother shards adjacent to the first target mother shard in the mother shard list with the first target mother shard to generate a second target mother shard; generating a prompt word according to all the second target mother shards and the retrieval data; and inputting the prompt word into a preset large language model to output a retrieval result corresponding to the prompt word through the preset large language model as the retrieval result corresponding to the retrieval data.

[0058] Specifically, during the retrieval process, the retrieval data input by the user can be vectorized to obtain a retrieval vector, and then calculated based on the distance between the vectors. For example, by calculating the distance between the retrieval vector and the shard vector stored in the vector database, such as the Euclidean distance or cosine similarity, multiple shard vectors with high similarity are found in the vector database, that is, the target sub-shard vector. Each row of data information in the vector database includes the id, each sub-shard, the shard vector corresponding to the sub-shard, and metadata. Then, the mother shard corresponding to the target sub-shard vector can be queried. Among the mother shards corresponding to the multiple target sub-shard vectors, there may be duplicate mother shards, and the mother shards corresponding to the multiple target sub-shard vectors can be screened to determine the first target mother shard corresponding to the retrieval vector among the multiple mother shards of the target document data split.

[0059] On the other hand, after the target text data is divided into multiple mother shards, they can be stored in the mother shard list in order for subsequent window sliding. Then, for each first target mother shard, a window frame can be set to select the mother shard adjacent to the first target mother shard, and the adjacent mother shard can be merged with the first target mother shard to generate a second target mother shard, so that more context information can be obtained. After that, each mother shard is recursively split, and each sentence generates a corresponding sub-shard, which helps to improve the overall flexibility and manageability of the RAG model, making data update, maintenance and expansion easier. Further, prompt words are generated based on all second target mother shards and retrieval data. The prompt words are based on artificial intelligence instruction technology, which is used to prompt the AI ​​model of the context of input information and parameter information of the input model. The prompt words are input into the preset large language model to output the retrieval results corresponding to the prompt words through the preset large language model as the retrieval results corresponding to the retrieval data. The preset large language model can be a deep learning model pre-trained by a technician using a large amount of text data, which can generate natural language text or understand the meaning of language text. Among them, there are many different preset large language models to choose from, such as the GPT-4o model (Generative Pre-TrainedTransformer-4 Omni, generative pre-trained transformer), the deepseek model (deep seek large language model), the qwen model, etc.

[0060] In an embodiment of the present application, searching a vector database for multiple target sub-slice vectors that match a search vector includes: determining the cosine similarity between each sub-slice vector included in the vector database and the search vector; sorting all sub-slice vectors from large to small according to the cosine similarity to determine the first M sub-slice vectors as target sub-slice vectors that match the search vector, where M is a positive integer.

[0061] In an embodiment of the present application, the mother shards adjacent to the first target mother shard are the two mother shards that are ranked after the first target mother shard in the mother shard list. Specifically, two windows r_window and l_window (both sizes are adjustable) can be set for the mother shard list. Assuming that the value r_window is 2 and l_window is 0, then the two mother shards behind the position can be added to each obtained mother shard without adding the mother shard in front of the position, thereby forming a new large shard and obtaining the corresponding second target mother shard. By setting the window, the context information of the recalled mother shard can be obtained, making the final retrieval results more accurate and complete. Both l_window and r_window can be set to 0, that is, the context window is not opened, and there is no need to merge adjacent mother shards.

[0062] In an embodiment of the present application, determining the first target mother shard corresponding to a retrieval vector based on the mother shards corresponding to multiple target sub-shard vectors includes: determining the mother shard content and the mother shard position of the mother shard corresponding to each target sub-shard vector; and deduplicating the mother shards that have the same mother shard content and / or the same mother shard position in the mother shards corresponding to all the target sub-shard vectors to obtain the first target mother shard corresponding to the retrieval vector.

[0063] It can be understood that each row of data information in the vector database includes id, each sub-shard, the shard vector corresponding to the sub-shard, and metadata. After finding multiple target sub-shard vectors with high similarity to the retrieval vector in the vector database, M rows of data with a high degree of match with the query data are obtained, where M can be adjusted by the technician, that is, M target sub-shard vectors with high similarity can be taken. However, there may be duplication problems in the content of the recalled mother shard. This is because the recalled mother shard is retrieved based on the target sub-shard vector. Therefore, it is necessary to perform deduplication based on the mother shard content and mother shard position of the mother shard corresponding to each target sub-shard vector. Specifically, the metadata field corresponding to the mother shard corresponding to all target sub-shard vectors can be used to identify the mother shards with the same mother shard content and / or the same mother shard position for deduplication. The purpose of deduplication is to reduce the number of input tokens and remove some interfering duplicate information.

[0064] In an embodiment of the present application, generating a retrieval vector corresponding to retrieval data includes: rewriting the retrieval data to generate multiple similar retrieval data corresponding to the retrieval data; inputting the retrieval data and each similar retrieval data into a preset embedding model respectively, so as to output corresponding retrieval vectors respectively through the preset embedding model.

[0065] Specifically, the search data entered by the user is input into the language model for rewriting to generate several similar search data in order to better locate keywords and more accurately capture user intent. Then the original search data and the generated similar search data are respectively input into the preset embedding model to obtain the corresponding search vector.

[0066] Then, you can search the vector database, and for each search vector, you can calculate the match based on the cosine similarity to obtain the M target sub-shard vectors with the highest matching degree corresponding to each search vector. Further, determine the mother shard content and mother shard position of the mother shard corresponding to each target sub-shard vector, and deduplicate the mother shards with the same mother shard content and / or the same mother shard position in the mother shards corresponding to all target sub-shard vectors to obtain the first target mother shard corresponding to the search vector.

[0067] In an embodiment of the present application, generating prompt words based on all second target mother shards and retrieval data includes: inputting all second target mother shards and retrieval data into a preset rearrangement model to determine the similarity between each second target mother shard and the retrieval data through the preset rearrangement model; sorting all second target mother shards from large to small according to the similarity to output the first N second target mother shards through the preset rearrangement model, where N is a positive integer; inputting the first N second target mother shards and retrieval data into a preset prompt project to generate prompt words through the preset prompt project.

[0068] It can be understood that the preset rearrangement model is a model for reordering multiple second target mother shards. Reordering is a key step in optimizing retrieval results, and the accuracy is improved by methods such as secondary retrieval and cross-model scoring. Specifically, all second target mother shards and retrieval data are input into the preset rearrangement model to determine the similarity between each second target mother shard and the retrieval data through the preset rearrangement model, and all second target mother shards are sorted from large to small according to the similarity, so as to output the first N second target mother shards through the preset rearrangement model, where N is a positive integer. In this way, some unimportant shards or shards with low relevance to the retrieval data can be removed. For example, the preset rearrangement model can be BAAI / bge-reranker-v2.5-gemma2-lightweight (lightweight multilingual vector model), which is specifically used to handle rearrangement tasks. The task of rearrangement is to evaluate the relevance of the context and give priority to those contents that are most likely to provide accurate and relevant information. Then, the first N second target mother shards and retrieval data can be input into the preset prompt project to generate prompt words through the preset prompt project. In addition, the prompt words are continuously rewritten and optimized based on the final search results, making the large model's answers more accurate and more in line with user needs.

[0069] Through the above technical solution, by taking paragraphs as the base, and using preset character intervals and preset separators as the basis for segmentation intervals, the target text data is divided into segments of appropriate sizes, which can not only retain more context information, but also improve computing efficiency and retrieval accuracy. In addition, the above solution is more flexible and the process is more modular. It can analyze different documents, fine-tune different embedding models, generate text based on different models, and adjust the context through windows of different sizes. Among them, through window settings and new document segmentation technology, the generated answers can be more complete and more accurate.

[0070] Figure 1 FIG. 1 is a flowchart of a document retrieval method in one embodiment. It should be understood that although Figure 1The steps in the flowchart are shown in sequence as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified in this document, there is no strict order restriction for the execution of these steps, and these steps can be executed in other orders. Moreover, Figure 1 At least part of the steps may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily executed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least part of the sub-steps or stages of other steps.

[0071] Figure 3 The structure block diagram of a document retrieval device according to an embodiment of the present application is schematically shown. Figure 3 As shown, the embodiment of the present application provides a document retrieval device, which may include:

[0072] a memory configured to store instructions; and

[0073] The processor is configured to call instructions from the memory and implement the above document retrieval method when executing the instructions.

[0074] Specifically, in the embodiment of the present application, the processor may be configured to:

[0075] Acquire multiple documents to be processed; pre-process the multiple documents to be processed to obtain corresponding target text data; acquire preset character intervals for slicing the target text data; perform recursive slicing operations on paragraphs included in the target text data according to the preset character intervals to divide the target text data into multiple parent slicings; slice each parent slicing into multiple child slicings, and generate child slicing vectors corresponding to each child slicing; store all child slicing vectors in a vector database; acquire search data input by a user; search the vector database for target child slicing vectors that match the search data, so as to determine the search results corresponding to the search data according to the parent slicings corresponding to the target child slicing vectors.

[0076] In the embodiment of the present application, the processor may also be configured to:

[0077] Recursively perform the paragraphs included in the target text data to determine the number of first characters included in the recursive current paragraph; when the number of first characters is less than the preset character interval, recursively perform the paragraphs included in the target text data starting from the current paragraph; determine in real time whether there is a preset delimiter in the paragraphs recursively performed starting from the current paragraph, and determine in real time the number of second characters included in multiple paragraphs recursively performed starting from the current paragraph; when there is a preset delimiter in the paragraphs recursively performed starting from the current paragraph, segment the multiple paragraphs included between the current paragraph and the paragraph corresponding to the preset delimiter to obtain corresponding mother fragments; when the number of second characters is greater than the preset character interval, segment all paragraphs recursively performed starting from the current paragraph to obtain corresponding mother fragments.

[0078] In the embodiment of the present application, the processor may also be configured to:

[0079] When the number of first characters is greater than or equal to the preset character interval, the current paragraph is segmented into corresponding parent segments.

[0080] In the embodiment of the present application, the processor may also be configured to:

[0081] Data cleaning is performed on multiple documents to be processed to obtain text data corresponding to each document to be processed; all text data are superimposed based on preset delimiters to obtain corresponding target text data.

[0082] In the embodiment of the present application, the processor may also be configured to:

[0083] Generate a search vector corresponding to the search data; search a plurality of target sub-shard vectors matching the search vector in a vector database; determine a first target mother shard corresponding to the search vector according to the mother shards corresponding to the plurality of target sub-shard vectors; store the plurality of mother shards in a mother shard list in order; for each first target mother shard, merge the mother shard adjacent to the first target mother shard in the mother shard list with the first target mother shard to generate a second target mother shard; generate a prompt word according to all the second target mother shards and the search data; input the prompt word into a preset large language model to output a search result corresponding to the prompt word through the preset large language model as the search result corresponding to the search data.

[0084] In the embodiment of the present application, the processor may also be configured to:

[0085] Determine the mother shard content and mother shard position of the mother shard corresponding to each target sub-shard vector; deduplicate the mother shards with the same mother shard content and / or the same mother shard position in the mother shards corresponding to all target sub-shard vectors to obtain a first target mother shard corresponding to the search vector.

[0086] In the embodiment of the present application, the processor may also be configured to:

[0087] The search data is rewritten to generate a plurality of similar search data corresponding to the search data; the search data and each similar search data are respectively input into a preset embedding model to output corresponding search vectors respectively through the preset embedding model.

[0088] In the embodiment of the present application, the processor may also be configured to:

[0089] Input all the second target mother shards and the retrieval data into a preset rearrangement model to determine the similarity between each second target mother shard and the retrieval data through the preset rearrangement model; sort all the second target mother shards from large to small according to the similarity to output the first N second target mother shards through the preset rearrangement model, where N is a positive integer; input the first N second target mother shards and the retrieval data into a preset prompt project to generate prompt words through the preset prompt project.

[0090] In an embodiment of the present application, the mother shards adjacent to the first target mother shard are two mother shards that are arranged after the first target mother shard in the mother shard list.

[0091] In the embodiment of the present application, the processor may also be configured to:

[0092] Determine the cosine similarity between each sub-slice vector included in the vector database and the search vector; sort all the sub-slice vectors from large to small according to the cosine similarity to determine the first M sub-slice vectors as target sub-slice vectors matching the search vector, where M is a positive integer.

[0093] An embodiment of the present application also provides a machine-readable storage medium, on which instructions are stored, and the instructions are used to enable a machine to execute the above-mentioned document retrieval method.

[0094] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 4As shown. The computer device includes a processor A01, a network interface A02, a memory (not shown in the figure) and a database (not shown in the figure) connected through a system bus. Among them, the processor A01 of the computer device is used to provide computing and control capabilities. The memory of the computer device includes an internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02 and a database (not shown in the figure). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 in the non-volatile storage medium A04. The database of the computer device is used to store document retrieval data. The network interface A02 of the computer device is used to communicate with an external terminal through a network connection. When the computer program B02 is executed by the processor A01, a document retrieval method is implemented.

[0095] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0096] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0097] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0098] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0099] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0100] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0101] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0102] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0103] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0104] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A document retrieval method, characterized in that: The method comprises: Get multiple documents to be processed; Preprocessing the plurality of documents to be processed to obtain corresponding target text data; Obtaining a preset character interval for slicing target text data; Performing a recursive segmentation operation on the paragraphs included in the target text data according to the preset character interval to divide the target text data into a plurality of parent fragments; Split each parent shard into multiple sub-shards, and generate a sub-shard vector corresponding to each sub-shard; Store all sub-slice vectors into the vector database; Get the search data entered by the user; generating a search vector corresponding to the search data; Searching the vector database for a plurality of target sub-slice vectors that match the search vector; Determine a first target mother shard corresponding to the search vector according to the mother shards corresponding to the multiple target sub-shard vectors; Store the multiple parent shards in the parent shard list in order; For each first target mother shard, merge the mother shards adjacent to the first target mother shard in the mother shard list with the first target mother shard to generate a second target mother shard; Generate prompt words according to all second target parent shards and the search data.

2. The document retrieval method according to claim 1, characterized in that: The recursive segmentation operation is performed on the paragraphs included in the target text data according to the preset character interval to divide the target text data into a plurality of parent fragments, including: Recursively perform the paragraphs included in the target text data to determine the number of first characters included in the recursive current paragraph; In the case where the first number of characters is less than the preset character interval, recursively searching the paragraphs included in the target text data starting from the current paragraph; Determine in real time whether there is a preset separator in the paragraphs recursively starting from the current paragraph, and determine in real time the number of second characters included in the multiple paragraphs recursively starting from the current paragraph; In the case where the preset delimiter exists in the paragraphs recursively starting from the current paragraph, multiple paragraphs included between the current paragraph and the paragraph corresponding to the preset delimiter are segmented to obtain corresponding parent fragments; When the second number of characters is greater than the preset character interval, all paragraphs recursively starting from the current paragraph are segmented to obtain corresponding parent fragments.

3. The document retrieval method according to claim 2, characterized in that: The method further comprises: When the first number of characters is greater than or equal to the preset character interval, the current paragraph is segmented into corresponding parent segments.

4. The document retrieval method according to claim 1, characterized in that: The preprocessing of the plurality of documents to be processed to obtain corresponding target text data includes: Performing data cleaning on the multiple documents to be processed to obtain text data corresponding to each document to be processed; All text data are superimposed based on the preset separator to obtain corresponding target text data.

5. The document retrieval method according to claim 1, characterized in that: The determining, according to the mother shards corresponding to the multiple target sub-shard vectors, the first target mother shard corresponding to the search vector comprises: Determine the mother shard content and mother shard position of the mother shard corresponding to each target sub-shard vector; De-duplicate the mother shards that have the same mother shard content and / or the same mother shard position in the mother shards corresponding to the target child shard vector to obtain a first target mother shard corresponding to the search vector.

6. The document retrieval method according to claim 1, characterized in that: The generating a search vector corresponding to the search data comprises: rewriting the search data to generate a plurality of similar search data corresponding to the search data; The search data and each similar search data are respectively input into a preset embedding model, so as to output corresponding search vectors respectively through the preset embedding model.

7. The document retrieval method according to claim 1, characterized in that: The generating of prompt words according to all second target parent shards and the search data comprises: Inputting all the second target parent shards and the search data into a preset rearrangement model to determine the similarity between each second target parent shard and the search data through the preset rearrangement model; Sort all the second target mother shards according to their similarity from large to small, so as to output the first N second target mother shards through the preset rearrangement model, where N is a positive integer; The first N second target mother shards and the search data are input into a preset prompting project to generate the prompting words through the preset prompting project.

8. The document retrieval method according to claim 1, characterized in that: The mother shards adjacent to the first target mother shard are two mother shards that are arranged after the first target mother shard in the mother shard list.

9. The document retrieval method according to claim 1, characterized in that: Searching the vector database for a plurality of target sub-slice vectors matching the search vector includes: Determining a cosine similarity between each sub-slice vector included in the vector database and the search vector; All sub-slice vectors are sorted from large to small according to cosine similarity, so as to determine the first M sub-slice vectors as target sub-slice vectors matching the search vector, where M is a positive integer.

10. A document retrieval device, characterized in that: include: a memory configured to store instructions; A processor is configured to call the instructions from the memory and implement the document retrieval method according to any one of claims 1 to 9 when executing the instructions.

11. A machine-readable storage medium, characterized in that: The machine-readable storage medium stores instructions, which are used to enable a machine to execute the document retrieval method according to any one of claims 1 to 9.

Images