Distributed retrieval method and system based on retrieval enhancement generation model

Abstract

The invention discloses a distributed retrieval method and system based on a retrieval enhancement generation model, and the method comprises the steps: a central server receives a retrieval condition sent by a client, carries out the retrieval in a central database through the retrieval enhancement generation model, and generates first target data corresponding to the retrieval condition; the center server side returns the first target data to the client side, and if the retrieval condition is received again, the retrieval condition is distributed to all node equipment sides; the node equipment end retrieves node data corresponding to the retrieval condition in a node database and sends the node data to the central server end; and the central server generates second target data through the retrieval enhancement generation model according to the node data and the retrieval conditions, and returns the second target data to the client. According to the method, the data retrieval performance can be ensured, and meanwhile, the requirements for the computing power and the storage performance of the server side are reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of data retrieval processing, and in particular to a distributed retrieval method and system based on a retrieval enhancement generation model. Background Art

[0002] With the rapid development of large language model (LLM) technology, LLMs are gradually penetrating into every corner of our daily life, providing corresponding technical support for various tools and applications. Among them, generative artificial intelligence such as ChatGPT and GLM have shown excellent performance in tasks such as text generation and text-to-image generation, but there are also some inherent limitations. These models lack the interpretability of generated text, have poor understanding of professional domain knowledge, and have limited understanding of the latest knowledge. In order to overcome these limitations, the model is usually combined with knowledge data in the professional field to make the output of the large model more timely and professional.

[0003] RAG (Retrieval Augmented Generation) is a generative model. The main function of RAG is similar to that of a search engine. It can efficiently retrieve and find the most relevant knowledge data or historical conversations based on user questions, and combine the original questions to create informative prompts to guide the model to generate accurate output. In the field of large language models (LLMs), achieving production-level performance and availability is a difficult task, especially building a high-performance RAG system.

[0004] RAG has been widely used in areas such as enterprise private domain knowledge Q&A. Among them, QAnything is a self-developed RAG (retrieval enhanced generation) engine launched by NetEase Youdao, which provides enterprises and individuals with a free local knowledge base Q&A solution that supports various file formats and databases and allows offline installation and use. With QAnything, users only need to simply drag and drop locally stored files or documents of any format and ask them questions to quickly obtain accurate, efficient and reliable information feedback. In addition, Ragflow is another comprehensive RAG solution. Through Ragflow, users can flexibly upload and manage documents in various formats. Through intelligent parsing, it accurately enters data into the database and allows users to use any large language model to query their uploaded documents.

[0005] However, these retrieval methods require that the database and the large language model run on the same device, which places high demands on the computing power of the running device. In addition, all data needs to be uploaded to the server, resulting in a large amount of data stored on the server. Therefore, a central knowledge database needs to be established for management, resulting in high costs for data storage and management.

[0006] In summary, although RAG technology can effectively improve the retrieval and output performance of large language models, it still faces some challenges and limitations. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a distributed retrieval method and system based on a retrieval enhancement generation model, which can reduce the requirements on the computing power and storage performance of the server while ensuring the data retrieval performance.

[0008] In order to solve the above technical problems, a technical solution adopted by the present invention is: A distributed retrieval method based on a retrieval enhancement generation model, comprising: The central server receives the search condition sent by the client, and searches and generates first target data corresponding to the search condition in the central database through the search enhancement generation model; The central server returns the first target data to the client, and if the search condition is received again, distributes the search condition to all node devices; The node device side retrieves the node data corresponding to the search condition in the node database, and sends the node data to the central service side; The central server generates second target data according to the node data and the search condition through the search enhancement generation model, and returns the second target data to the client.

[0009] In order to solve the above technical problems, another technical solution adopted by the present invention is: A distributed retrieval system based on a retrieval enhancement generation model includes a central server and multiple node device ends, the central server end includes a first memory, a first processor, and a first computer program stored in the first memory and running on the first processor; the node device end includes a second memory, a second processor, and a second computer program stored in the second memory and running on the second processor; when the first processor executes the first computer program, it implements the steps executed by the central server of the above-mentioned distributed retrieval method based on the retrieval enhancement generation model, and when the second processor executes the second computer program, it implements the steps executed by the node device end of the above-mentioned distributed retrieval method based on the retrieval enhancement generation model.

[0010] The beneficial effects of the present invention are as follows: upon receiving the retrieval conditions sent by the client, the target data is retrieved and generated in the central database of the central server through the retrieval enhancement generation model, and the target data generated by the central server is returned to the client. If the client sends the retrieval conditions again, it means that the user believes that the current target data does not meet the retrieval conditions. At this time, the central server distributes the retrieval conditions to the node device end for relevant retrieval. If the relevant node data is retrieved at the node device end, the node data is first returned to the central server end, and the central server end generates the target data according to the node data through the retrieval enhancement generation model and then returns it to the client. The present invention realizes distributed retrieval based on the central server end and the node device end, and makes full use of the storage resources of the node device end, so that the central server end only needs to store part of the data to realize retrieval, which effectively reduces the cost of data storage and management. In addition, the present invention disperses the computing power required for retrieval to each node device end, effectively reducing the computing power demand of the central server end and improving the retrieval performance of the central server end. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 A flowchart of the steps of a distributed retrieval method based on a retrieval enhancement generation model provided by an embodiment of the present invention; Figure 2 Another flow chart of a distributed retrieval method based on a retrieval enhancement generation model provided by an embodiment of the present invention; Figure 3 A database topology diagram of a distributed retrieval method provided by an embodiment of the present invention; Figure 4 A schematic diagram of the structure of a distributed retrieval system based on a retrieval enhancement generation model provided by an embodiment of the present invention; Figure 5 A schematic diagram of the architecture of a distributed retrieval system based on a retrieval enhancement generation model provided by an embodiment of the present invention; Description of labels: 100. A distributed retrieval system based on a retrieval enhancement generation model; 101. a central server; 102. a node device; 1011. a first memory; 1012. a second processor; 1021. a second memory; 1022. a second processor. DETAILED DESCRIPTION

[0012] In order to explain the technical content, achieved objectives and effects of the present invention in detail, the following is an explanation in combination with the implementation modes and the accompanying drawings.

[0013] An embodiment of the present invention provides a distributed retrieval method based on a retrieval enhancement generation model, comprising: The central server receives the search condition sent by the client, and searches and generates first target data corresponding to the search condition in the central database through the search enhancement generation model; The central server returns the first target data to the client, and if the search condition is received again, distributes the search condition to all node devices; The node device side retrieves the node data corresponding to the search condition in the node database, and sends the node data to the central service side; The central server generates second target data according to the node data and the search condition through the search enhancement generation model, and returns the second target data to the client.

[0014] From the above description, it can be seen that the beneficial effects of the present invention are: when the search conditions sent by the client are received, the target data is retrieved and generated in the central database of the central server through the search enhancement generation model, and the target data generated by the central server is returned to the client. If the client sends the search conditions again, it means that the user believes that the current target data does not meet the search conditions. At this time, the central server distributes the search conditions to the node device end for relevant search. If the relevant node data is retrieved at the node device end, the node data is first returned to the central server end, and the central server end generates the target data according to the node data through the search enhancement generation model and then returns it to the client. The present invention realizes distributed retrieval based on the central server end and the node device end, and makes full use of the storage resources of the node device end, so that the central server end only needs to store part of the data to realize retrieval, which effectively reduces the cost of data storage and management. In addition, the present invention disperses the computing power required for retrieval to each node device end, effectively reducing the computing power demand of the central server end and improving the retrieval performance of the central server end.

[0015] Furthermore, before the central server receives the search condition sent by the client, it also includes: The central server reads a preset local question-and-answer file and stores the local question-and-answer file in a pre-created central database, wherein the local question-and-answer file includes a plurality of standard data; After the central server receives the search conditions sent by the client, it also includes: The central server searches the central database according to the search condition to obtain matching values ​​between the search condition and the plurality of standard data; If there is a first matching value greater than the first threshold value among the matching values, the central server determines first standard data corresponding to the first matching value, and returns the first standard data to the client; If the first matching value does not exist in the matching values, the central server retrieves and generates first target data corresponding to the search condition in the central database through the retrieval enhancement generation model.

[0016] It can be seen from the above description that compared to the traditional central database that stores all data, the central database of the present invention only needs to store local question and answer files, which greatly reduces the amount of data stored on the central server. Therefore, there is no need to build a complex database and query architecture to manage data, which effectively reduces the cost of data storage and management. During the retrieval process, when the retrieval condition has a high match value with the preset standard data, it indicates that the retrieval condition can directly obtain the standard data in the central database without calling the retrieval enhancement model for retrieval and generation, which effectively improves the retrieval efficiency. When the retrieval condition has a low match value with the preset standard data, it indicates that the retrieval condition has relevant standard data in the central database, but it is not a complete match, so it is necessary to call the retrieval enhancement model for optimized retrieval and generation to ensure the accuracy of the retrieval results.

[0017] Furthermore, the central server retrieves and generates the first target data corresponding to the search condition in the central database through the search enhancement generation model, including: The central server detects whether there is a second matching value greater than a second threshold value and not greater than the first threshold value in the matching values; If so, the central server determines second standard data corresponding to the second matching value, the first threshold is greater than the second threshold, and performs enhancement processing on the second standard data by retrieving the enhancement generation model to generate first target data; If it does not exist, the central server will distribute the search conditions to all node devices.

[0018] As can be seen from the above description, the central server retrieves the standard data whose matching value is between the first threshold and the second threshold in the central database, and then generates the target data after aggregating and processing these standard data through the retrieval enhancement generation model. If the retrieval matching value is lower than the second threshold, the retrieval condition is directly distributed to the node device end for retrieval processing. In this way, while ensuring the best match between the retrieval results and the retrieval conditions, the retrieval efficiency can be effectively improved, thereby improving the accuracy of the retrieval results.

[0019] Furthermore, before the central server receives the search condition sent by the client, it also includes: The node device reads the local file, performs data quantization processing on the local file through the Embdding model to obtain vector data, and stores the vector data in a pre-created node database.

[0020] From the above description, it can be seen that the node device uses its own computing resources to locally call the Embdding model to quantize data, so that the quantization computing power originally concentrated on the central server is dispersed to each node device, effectively reducing the computing power demand of the central server.

[0021] Furthermore, the node device side retrieves the node data corresponding to the search condition in the node database, including: The node device side searches the node database according to the search condition to obtain the correlation between the search condition and the vector data; If the correlation complies with a preset correlation rule, the vector data corresponding to the correlation in the node database is marked as node data.

[0022] From the above description, it can be seen that since the central database only stores part of the data, the central server has certain limitations when retrieving data. Therefore, when the central database cannot retrieve the target data with high relevance, the search conditions are distributed to each node device, and the node device searches the local node database. This avoids the increase in the amount of data stored in the central server, effectively utilizes the data scattered on each node device, and expands the data volume of the basic database.

[0023] Furthermore, the node device performs data quantization processing on the local file through the embedding model to obtain vector data including: The node device splits the local file according to the text segmentation algorithm to obtain basic units, and performs data quantization processing on the basic units through the Embdding model to obtain vector data.

[0024] From the above description, we can see that the local file is split into basic units through the text segmentation algorithm, and the basic units are vectorized through the embedding model. When searching the vectorized text, not only the matching of words or characters can be achieved, but also the matching of the meaning expressed in the text can be achieved, thereby improving the retrieval accuracy.

[0025] Furthermore, the local files include public files and private files; The node device side stores the vector data in a pre-created node database including: The node device side stores the vector data, the basic unit corresponding to the vector data, and the file source of the vector data in a pre-created node database; Before the node device sends the node data to the central server, the method further includes: The node device detects the file source of the node data; If the file source of the node data is a public file, the node device sends the node data to the central server; If the file source of the node data is a private file, the node device sends the passive data frame of the node data to the central server.

[0026] From the above description, it can be seen that the data uploaded to the central server can be accessed as long as it is connected to the central server, resulting in the data privacy not being guaranteed. Therefore, the data on the node device side is divided into public files and private files. For public files, as long as the relevant data is retrieved, it is directly uploaded to the central server; for private files, the passive data frame of the data is sent to the central server to inform the central server that the current node device has relevant data, but the specific content of the data will not be informed, which effectively protects the privacy of the data and improves data security.

[0027] Furthermore, the central server generates the second target data through the retrieval enhancement generation model according to the node data and the retrieval condition, including: The central server ranks the relevance between the node data and the search condition through the Rerank model; The central server generates second target data through the retrieval enhancement generation model according to the node data of the sorting result that meets the preset association rule and the retrieval condition.

[0028] From the above description, it can be seen that since there are multiple node data returned from each node device, the node data retrieved from the node device is optimized and sorted again through the Rerank model, so that the node data with higher relevance to the search conditions has the highest sorting priority, so as to improve the accuracy and quality of the search results.

[0029] Furthermore, after the central server returns the second target data to the client, the method further includes: The central server stores the search condition and the second target data in the central database.

[0030] As can be seen from the above description, the search condition and the second target data are stored in the central database, so that when the user sends the same search condition again, there is no need to repeatedly call the node device end for retrieval, thereby improving the search efficiency.

[0031] Another embodiment of the present invention provides a distributed retrieval system based on a retrieval enhancement generation model, including a central server and multiple node device ends, the central server end including a first memory, a first processor, and a first computer program stored in the first memory and running on the first processor; the node device end includes a second memory, a second processor, and a second computer program stored in the second memory and running on the second processor; when the first processor executes the first computer program, it implements the steps executed by the central server of the above-mentioned distributed retrieval method based on the retrieval enhancement generation model, and when the second processor executes the second computer program, it implements the steps executed by the node device end of the above-mentioned distributed retrieval method based on the retrieval enhancement generation model.

[0032] From the above description, it can be seen that the beneficial effects of the present invention are: when the search conditions sent by the client are received, the target data is retrieved and generated in the central database of the central server through the search enhancement generation model, and the target data generated by the central server is returned to the client. If the client sends the search conditions again, it means that the user believes that the current target data does not meet the search conditions. At this time, the central server distributes the search conditions to the node device end for relevant search. If the relevant node data is retrieved at the node device end, the node data is first returned to the central server end, and the central server end generates the target data according to the node data through the search enhancement generation model and then returns it to the client. The present invention realizes distributed retrieval based on the central server end and the node device end, and makes full use of the storage resources of the node device end, so that the central server end only needs to store part of the data to realize retrieval, which effectively reduces the cost of data storage and management. In addition, the present invention disperses the computing power required for retrieval to each node device end, effectively reducing the computing power demand of the central server end and improving the retrieval performance of the central server end.

[0033] The above-mentioned distributed retrieval method and system based on the retrieval enhancement generation model of the present invention can be applied to the search engine of the knowledge question and answer database, which can reduce the requirements on the computing power and storage performance of the server while ensuring the data retrieval performance. The following is an explanation through specific implementation methods: Please refer to Figures 1 to 3 , Embodiment 1 of the present invention is: A distributed retrieval method based on a retrieval enhancement generation model, comprising: S1. The central server receives the search conditions sent by the client, and searches and generates first target data corresponding to the search conditions in the central database through the search enhancement generation model.

[0034] It should be noted that the search conditions sent by the client are the questions that the user needs to search. The retrieval enhancement generation model is the RAG model. The RAG model includes three steps: retrieval, enhancement, and generation. Among them, retrieval is to retrieve information related to the question from a pre-established knowledge base; enhancement is to integrate external knowledge into the generation process to make the generated text content richer, more accurate, and in line with user needs; generation is to combine the LLM model to generate answers that meet user needs.

[0035] Specifically, before the central server receives the search conditions sent by the client, it also includes: S101. The central server reads a preset local question-and-answer file and stores the local question-and-answer file in a pre-created central database, wherein the local question-and-answer file includes a plurality of standard data.

[0036] In some embodiments, the central database is automatically created when the central server is started for the first time.

[0037] In some embodiments, the central server also reads the preset local public data, and performs data quantization processing on the local question and answer file and the local public data through the embedding model to obtain vector data and store it in the central database. Among them, the local public data read by the central server is public data.

[0038] In some embodiments, public data and non-public data may be stored in different directories, respectively, so as to perform differentiated management of data publicity.

[0039] Specifically, after the central server receives the search conditions sent by the client, it also includes: S11. The central server searches the central database according to the search condition to obtain matching values ​​between the search condition and the plurality of standard data.

[0040] In some embodiments, since the retrieval module in the retrieval enhancement generation model can calculate the vector similarity between the input text and the documents in the central database, the central server can search the central database based on the retrieval module of the retrieval enhancement model to obtain the vector similarity between the retrieval conditions and the standard data in the central database as the matching value between the two.

[0041] S12: If there is a first matching value greater than a first threshold value among the matching values, the central server determines first standard data corresponding to the first matching value, and returns the first standard data to the client.

[0042] In some embodiments, when the matching value between the search condition and the standard data is greater than a first threshold, it indicates that the search condition completely matches the current standard data, and thus the standard data can be directly returned to the client.

[0043] S13: If the first matching value does not exist in the matching values, the central server retrieves and generates first target data corresponding to the search condition in the central database through the retrieval enhancement generation model.

[0044] In some embodiments, when the matching value between the retrieval condition and the standard data is not greater than a first threshold, it indicates that the retrieval condition is not completely matched with the current standard data, and the standard data cannot be returned directly. Therefore, it is necessary to call the retrieval enhancement generation model to optimize the standard data before returning it to the client.

[0045] Specifically, in step S1, the central server retrieves and generates the first target data corresponding to the search condition in the central database through the search enhancement generation model, including: S14. The central server detects whether there is a second matching value in the matching values ​​that is greater than a second threshold and not greater than the first threshold.

[0046] S15. If so, the central server determines the second standard data corresponding to the second matching value, the first threshold is greater than the second threshold, and enhances the second standard data by retrieving the enhanced generation model to generate the first target data.

[0047] In some embodiments, the second standard data determined by the central server may be first stored in a historical question and answer database of the central database, so that the central server can generate the first target data based on the second standard data.

[0048] S16. If it does not exist, the central service end distributes the search condition to all node device ends.

[0049] In some embodiments, when the matching value between the search condition and the standard data is not greater than the first threshold value, but greater than the second threshold value, it indicates that the search condition is relevant to the current standard data, but not completely matched. Therefore, the standard data is retained to generate first target data with a higher matching value after being enhanced by the search enhancement generation model.

[0050] In some embodiments, if the number of determined first standard data or second standard data is large, the first standard data or second standard data may be sorted from high to low according to the matching value, and the first standard data or second standard data with the highest sorting result may be selected.

[0051] S2. The central server returns the first target data to the client, and if the search condition is received again, distributes the search condition to all node devices.

[0052] In some embodiments, if the central server receives the same search condition sent by the same client within a preset time interval, it means that the central server receives the search condition again.

[0053] S3. The node device side retrieves the node data corresponding to the search condition in the node database, and sends the node data to the central server side.

[0054] Specifically, before the central server receives the search conditions sent by the client, it also includes: S102, the node device reads the local file, performs data quantization processing on the local file through the Embdding model to obtain vector data, and stores the vector data in a pre-created node database.

[0055] In some embodiments, the node device end includes a public data area and a personal data area. When the node device end is started, it only reads local files in the public data area, and does not perform any scanning or processing on the personal data area, thereby avoiding personal data from being uploaded to the central server. The ownership of public data and personal data can be effectively clarified, thereby improving the privacy and security of personal data.

[0056] In some embodiments, both the central database and the node database can be added, deleted, checked, and modified through local tools.

[0057] Specifically, step S3 includes: S31. The node device searches the node database according to the search condition to obtain the correlation between the search condition and the vector data.

[0058] S32: If the correlation complies with a preset correlation rule, the vector data corresponding to the correlation in the node database is marked as node data.

[0059] In some embodiments, a third threshold is set in the node device end, and all vector data are sorted from high to low according to the correlation between the retrieval condition and the vector data. If the vector data sorting result is within a predetermined range (for example, the sorting result is in the top 3) and the correlation value is greater than the third threshold, then the correlation between the retrieval condition and the vector data meets the preset correlation rules.

[0060] In some embodiments, when the node device uploads the node data to the central server, it is also necessary to upload the IP address of the node device so that the central server can determine the source of the data.

[0061] S4. The central server generates second target data according to the node data and the search condition through the search enhancement generation model, and returns the second target data to the client.

[0062] Specifically, step S4 includes: S41. The central server ranks the relevance between the node data and the search condition through a Rerank model.

[0063] In some embodiments, the Rerank model can re-rank the multiple node data retrieved so that their ranking is more consistent with the relevance of the search criteria input by the user.

[0064] S42: The central server generates second target data through the search enhancement generation model according to the node data of the sorting result that meets the preset association rule and the search condition.

[0065] In some embodiments, a fourth threshold is set in the central server. If the ranking result of the correlation between the node data and the search condition is within a predetermined range (for example, the ranking result is in the top 5) and the correlation value is greater than the fourth threshold, the corresponding node data is retained and combined with the search condition to generate the second target data through the RAG model. It should be noted that the fourth threshold is a correlation value obtained after local testing based on a large amount of data. If the correlation value evaluated by the Rerank model is greater than the fourth threshold, it indicates that the correlation between the search condition and the node data is strong. In addition, the fourth threshold can be adjusted according to environmental changes.

[0066] After step S5, the method further includes: S5. The central server stores the search condition and the second target data in the central database. Specifically, the search condition and the second target data are stored in the historical question and answer database in the central database for direct query and return.

[0067] Please refer to Figure 2 , Embodiment 2 of the present invention is: A distributed retrieval method based on a retrieval enhancement generation model is different from the first embodiment in that the specific implementation method of step S102 is limited.

[0068] In step S102, the node device performs data quantization processing on the local file through the Embdding model to obtain vector data including: S1021. The node device splits the local file into basic units according to a text segmentation algorithm, and performs data quantization processing on the basic units through an embedding model to obtain vector data.

[0069] Specifically, the local files include public files and private files.

[0070] In step S102, the node device side stores the vector data in a pre-created node database including: S1022. The node device stores the vector data, the basic unit corresponding to the vector data, and the file source of the vector data in a pre-created node database.

[0071] In some embodiments, after determining the vector data (node ​​data) that needs to be uploaded, the node device packaged the vector data, the IP address of the node device, the basic unit corresponding to the vector data and the file source into a data frame, and uploaded the data frame to the central server.

[0072] Before the node device sends the node data to the central server, the method further includes: S33: The node device detects the file source of the node data.

[0073] S34. If the file source of the node data is a public file, the node device sends the node data to the central server.

[0074] S35. If the file source of the node data is a private file, the node device sends the passive data frame of the node data to the central server.

[0075] In some embodiments, for node data in private files, when the node device sends data, it will not package the basic units and send them to the central server, but will only indicate that the relevant data has been found and that the relevant data is a private file, so that the private data is always stored on the node device, effectively improving data security.

[0076] Please refer to Figures 4 to 5 , Embodiment 3 of the present invention is: A distributed retrieval system 100 based on a retrieval enhancement generation model includes a central server end 101 and multiple node device ends 102, wherein the central server end 101 includes a first memory 1011, a first processor 1012, and a first computer program stored in the first memory 1011 and running on the first processor 1012; the node device end 102 includes a second memory 1021, a second processor 1022, and a second computer program stored in the second memory 1021 and running on the second processor 1022; when the first processor 1012 executes the first computer program, the steps executed by the central server end of a distributed retrieval method based on a retrieval enhancement generation model in embodiments one and two are implemented; when the second processor 1022 executes the second computer program, the steps executed by the node device end of a distributed retrieval method based on a retrieval enhancement generation model in embodiments one and two are implemented.

[0077] In summary, the present invention provides a distributed retrieval method and system based on a retrieval enhancement generation model. When the system receives the retrieval conditions sent by the client, it first retrieves and generates data in the central database of the central server. If the user believes that the current data does not meet the retrieval conditions, the central server distributes the retrieval conditions to the node device for retrieval. If the relevant node data is retrieved at the node device, the node data is first returned to the central server, and the central server generates data based on the node data and then returns it to the client. The present invention implements distributed retrieval based on the central server and the node device, making full use of the storage resources of the node device, so that the central server only needs to store part of the data to realize retrieval, effectively reducing the cost of data storage and management. Moreover, in the process of creating a database, the quantitative processing of text data is dispersed in each node device, thereby effectively utilizing the computing power resources of the node device and reducing the data quantification pressure of the central server. In addition, different upload modes are used for private files and public files in the node device to avoid private files from being uploaded to the central server, while ensuring data privacy, data query performance can be taken into account.

[0078] The above descriptions are merely embodiments of the present invention and are not intended to limit the patent scope of the present invention. Any equivalent transformations made using the contents of the present invention's specification and drawings, or directly or indirectly applied in related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A distributed retrieval method based on a retrieval enhancement generation model, characterized in that: include: The central server receives the search condition sent by the client, and searches and generates first target data corresponding to the search condition in the central database through the search enhancement generation model; The central server returns the first target data to the client, and if the search condition is received again, distributes the search condition to all node devices; The node device side retrieves the node data corresponding to the search condition in the node database, and sends the node data to the central service side; The central server generates second target data according to the node data and the search condition through the search enhancement generation model, and returns the second target data to the client.

2. A distributed retrieval method based on a retrieval enhancement generation model according to claim 1, characterized in that: Before the central server receives the search conditions sent by the client, it also includes: The central server reads a preset local question-and-answer file and stores the local question-and-answer file in a pre-created central database, wherein the local question-and-answer file includes a plurality of standard data; After the central server receives the search conditions sent by the client, it also includes: The central server searches the central database according to the search condition to obtain matching values ​​between the search condition and the plurality of standard data; If there is a first matching value greater than the first threshold value among the matching values, the central server determines first standard data corresponding to the first matching value, and returns the first standard data to the client; If the first matching value does not exist in the matching values, the central server retrieves and generates first target data corresponding to the search condition in the central database through the retrieval enhancement generation model.

3. A distributed retrieval method based on a retrieval enhancement generation model according to claim 2, characterized in that: The central server retrieves and generates the first target data corresponding to the search condition in the central database through the search enhancement generation model, including: The central server detects whether there is a second matching value greater than a second threshold value and not greater than the first threshold value in the matching values; If so, the central server determines second standard data corresponding to the second matching value, the first threshold is greater than the second threshold, and performs enhancement processing on the second standard data by retrieving the enhancement generation model to generate first target data; If it does not exist, the central server will distribute the search conditions to all node devices.

4. The distributed retrieval method based on the retrieval enhancement generation model according to claim 1, characterized in that: Before the central server receives the search conditions sent by the client, it also includes: The node device reads the local file, performs data quantization processing on the local file through the Embdding model to obtain vector data, and stores the vector data in a pre-created node database.

5. A distributed retrieval method based on a retrieval enhancement generation model according to claim 4, characterized in that: The node device side retrieves the node data corresponding to the search condition in the node database, including: The node device side searches the node database according to the search condition to obtain the correlation between the search condition and the vector data; If the correlation complies with a preset correlation rule, the vector data corresponding to the correlation in the node database is marked as node data.

6. A distributed retrieval method based on a retrieval enhancement generation model according to claim 4, characterized in that: The node device performs data quantization processing on the local file through the Embdding model to obtain vector data including: The node device splits the local file according to the text segmentation algorithm to obtain basic units, and performs data quantization processing on the basic units through the Embdding model to obtain vector data.

7. A distributed retrieval method based on a retrieval enhancement generation model according to claim 6, characterized in that: The local files include public files and private files; The node device side stores the vector data in a pre-created node database including: The node device side stores the vector data, the basic unit corresponding to the vector data, and the file source of the vector data in a pre-created node database; Before the node device sends the node data to the central server, the method further includes: The node device detects the file source of the node data; If the file source of the node data is a public file, the node device sends the node data to the central server; If the file source of the node data is a private file, the node device sends the passive data frame of the node data to the central server.

8. The distributed retrieval method based on the retrieval enhancement generation model according to claim 1, characterized in that: The central server generates the second target data by using the retrieval enhancement generation model according to the node data and the retrieval condition, including: The central server ranks the relevance between the node data and the search condition through the Rerank model; The central server generates second target data through the retrieval enhancement generation model according to the node data of the sorting result that meets the preset association rule and the retrieval condition.

9. The distributed retrieval method based on the retrieval enhancement generation model according to claim 1, characterized in that: After the central server returns the second target data to the client, the method further includes: The central server stores the search condition and the second target data in the central database.

10. A distributed retrieval terminal based on a retrieval enhancement generation model, comprising a central server and multiple node device terminals, wherein the central server terminal comprises a first memory, a first processor, and a first computer program stored in the first memory and running on the first processor; the node device terminal comprises a second memory, a second processor, and a second computer program stored in the second memory and running on the second processor; characterized in that: When the first processor executes the first computer program, it implements the steps performed by the central server end of a distributed retrieval method based on a retrieval enhancement generation model as described in any one of claims 1-9; when the second processor executes the second computer program, it implements the steps performed by the node device end of a distributed retrieval method based on a retrieval enhancement generation model as described in any one of claims 1-9.

Images