Data retrieval method, device, and terminal

Abstract

The present application relates to a data retrieval method, device, and terminal, wherein the method includes: when an MCP client receives a query request, performing label filtering and similarity search on the query statement in the query request in a vector database; wherein the vector database includes vectors corresponding to different types of knowledge data, and metadata associated with the vectors, the metadata represents the label information of the vectors, and different types of knowledge data are obtained by vectorizing them after being acquired by different application program interfaces (APIs); different APIs are developed based on the MCP protocol; the vectors obtained by similarity search and label filtering are sent to a locally deployed large model; the results obtained by the large model through inductive reasoning on the knowledge data corresponding to the vectors are received, and the results are displayed. Through this application, the problem of low accuracy of retrieval results in the prior art based on RAG for data retrieval is solved.

Description

Technical Field

[0001] The present application relates to the field of data retrieval, and in particular to a data retrieval method, device, and terminal. Background Art

[0002] Retrieval-Augmented Generation (RAG) is a hot topic in the field of large models. It combines information retrieval technology with generative models to address the challenges of large models in terms of knowledge accuracy, contextual understanding, and utilization of the latest information. The core of RAG is to first convert knowledge into vectors and import them into a vector database. Then, user-entered information is converted into vectors, and similar vectors are matched against the vector database. Finally, the large model summarizes the retrieved content. In this process, the large model merely serves as a summary, while the accuracy of the retrieved information largely depends on the similarity of the vectors. This can lead to irrelevant content in the search results, resulting in low accuracy.

[0003] There is currently no effective solution to the above technical problems in the prior art. Summary of the Invention

[0004] The present application provides a data retrieval method and device, a terminal and a storage medium to solve the problem of low accuracy of retrieval results in RAG-based data retrieval in the prior art.

[0005] In a first aspect, the present application provides a data retrieval method, which is applied to a model context protocol (MCP) client. The method includes: when the MCP client receives a query request, performing label filtering and similarity search on the query statement in the query request in a vector database; wherein the vector database includes vectors corresponding to different types of knowledge data, and metadata associated with the vectors, the metadata represents the label information of the vectors, and the different types of knowledge data are obtained by vectorizing the acquired data after being acquired by different application programming interfaces (APIs); the different APIs are developed based on the MCP protocol; the vectors obtained by similarity search and label filtering are sent to a locally deployed large model; the results obtained by the large model through inductive reasoning on the knowledge data corresponding to the vectors are received, and the results are displayed.

[0006] In a second aspect, the present application provides a data retrieval device, which is applied to an MCP client. The device includes: a first processing module, which is used to perform label filtering and similarity search on the query statement in the query request in a vector database when the MCP client receives a query request; wherein the vector database includes vectors corresponding to different types of knowledge data, and metadata associated with the vectors, the metadata represents the label information of the vectors, and the different types of knowledge data are obtained by obtaining them through different application programming interfaces (APIs) and then vectorizing them; the different APIs are developed based on the MCP protocol; a sending module, which is used to send the vectors obtained by similarity search and label filtering to a locally deployed large model; and a second processing module, which is used to receive the results obtained by the large model through inductive reasoning on the knowledge data corresponding to the vectors, and display the results.

[0007] In a third aspect, the present application provides a terminal comprising: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor connected to the at least one bus; and at least one memory connected to the at least one bus, wherein the processor is configured to execute the data retrieval method described in the first aspect of the present application.

[0008] The above technical solution provided by the embodiment of the present application has the following advantages over the prior art: the method provided by the embodiment of the present application, after obtaining knowledge data from different sources through different APIs, vectorizes the obtained knowledge data and stores it in a vector database. Based on this, after receiving the query request, the MCP client performs label filtering and similarity search on the query statement in the query request in the vector database. Since the vectors in the vector database are not vectors of knowledge data obtained through a single channel, the knowledge data obtained after vectorization through label filtering and similarity search is more accurate. Moreover, compared with the single retrieval method in the prior art, the label filtering and similarity search are not only efficient but also have high retrieval accuracy. In addition, in the present application, since the retrieval results are sent to the large model through the MCP client with a universal protocol, the various large models can be seamlessly linked so that the retrieval results can be efficiently summarized and reasoned and their reasoning results can be displayed. It can be seen that through the method in the embodiment of the present application, not only can the database be searched quickly, but the retrieval accuracy is also high, which meets the user's expectations for retrieval needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0010] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0011] One or more embodiments are exemplarily illustrated by pictures in the corresponding drawings. These exemplifications do not constitute limitations on the embodiments. Elements with the same reference numerals in the drawings are represented as similar elements. Unless otherwise stated, the figures in the drawings do not constitute proportional limitations.

[0012] Figure 1 A flowchart of a data retrieval method provided in an embodiment of the present application;

[0013] Figure 2 A flowchart of a method for improving the accuracy and efficiency of large model retrieval knowledge based on vector database index construction and MCP provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram of the structure of a data retrieval device provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of the structure of a terminal provided in an embodiment of the present application. DETAILED DESCRIPTION

[0016] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0017] The disclosure below provides many different embodiments or examples for implementing different configurations of the present invention. To simplify the disclosure of the present invention, the components and configurations of specific examples are described below. Of course, these are merely examples and are not intended to limit the present invention. In addition, the present invention may repeat reference numerals and / or letters in different examples. Such repetition is for the purpose of simplicity and clarity and does not in itself indicate the relationship between the various embodiments and / or configurations discussed.

[0018] In order to solve the problem of low accuracy of data retrieval results based on RAG in the prior art, this application provides a data retrieval method applied to the Model Context Protocol (MCP) client, such as Figure 1 As shown, the steps of the method include:

[0019] Step 101: When an MCP client receives a query request, it performs tag filtering and similarity search on the query statement in the query request in a vector database. The vector database includes vectors corresponding to different types of knowledge data, as well as metadata associated with the vectors. The metadata represents the label information of the vectors. Different types of knowledge data are obtained through different application programming interfaces (APIs) and then vectorized. Different APIs are developed based on the MCP protocol.

[0020] In the embodiments of the present application, different types of knowledge data may refer to various types of knowledge data in a certain professional field, such as different types of knowledge data in the automotive field and different types of knowledge data in the mobile phone field. Furthermore, these different types of knowledge data can be obtained through multiple channels, that is, multiple different APIs. The API can be developed in accordance with the MCP protocol. Specifically, internal enterprise knowledge base files (such as word, excel, ppt, txt, markdown, etc.) can be obtained through a file reading and parsing interface, Mysql databases can be obtained through a database query interface, online public information can be obtained through a web crawler interface, and external private data sources can be obtained through a private data interface. It can be seen that the vector database in the embodiments of the present application includes knowledge data from various aspects, including knowledge data stored in internal documents or databases, as well as private platform or public knowledge data. In other words, the knowledge data in the vector database in the present application is more comprehensive, so that the accuracy of the knowledge data subsequently retrieved based on the vector database in the present application is higher than that of the prior art.

[0021] It's important to note that before vectorizing knowledge data, it needs to be preprocessed, such as by extracting keywords and performing entity recognition. Documents are then labeled manually or automatically based on their content or source. For example, a research paper on defect detection in automotive parts production in the industrial sector might be labeled "Industrial Sector," "Automotive Parts," and "Defect Detection." This label information is then stored as metadata in a vector database along with the vector representation of the knowledge data.

[0022] Step 102: Send the vectors obtained through similarity search and label filtering to the locally deployed large model;

[0023] In this regard, since the MCP protocol is universal, the large model in this application can be any current large model, that is, through MCP, there is no need to repeatedly write adaptation code for each model to connect with different resources, which can greatly save development workload.

[0024] Step 103: Receive the results obtained by the large model through inductive reasoning on the knowledge data corresponding to the vector and display the results.

[0025] Through the above steps 101 to 103 in the embodiment of the present application, after obtaining knowledge data from different sources through different APIs, the obtained knowledge data is vectorized and stored in a vector database. Based on this, after receiving the query request, the MCP client performs label filtering and similarity search on the query statement in the query request in the vector database. Since the vectors in the vector database are not vectors of knowledge data obtained through a single channel, the knowledge data obtained after vectorization through label filtering and similarity search is more accurate. Moreover, compared with the single retrieval method in the prior art, the label filtering and similarity search are not only efficient but also have high retrieval accuracy. In addition, in the present application, since the retrieval results are sent to the large model through the MCP client with a universal protocol, the various large models can be seamlessly linked so that the retrieval results can be efficiently summarized and reasoned and their reasoning results can be displayed. It can be seen that through the method in the embodiment of the present application, not only can the retrieval in the database be performed quickly, but the retrieval accuracy is also high, which meets the user's expectations for retrieval needs.

[0026] In an optional implementation of the present application, the method of performing label filtering on the query statement in the query request in the vector database involved in step 101 may further include:

[0027] Step 11: determine a first tag of the query statement, and determine the first tag as the first metadata of the query statement;

[0028] In this regard, in a specific example, a query statement may correspond to one or more tags, and thus the corresponding metadata may also be one or more. For example, if the current query statement includes multiple tags such as "industrial field", "auto parts", and "defect detection", then these multiple tags also correspond to multiple metadata.

[0029] Step 12: determining the distance between the first metadata and the cluster centers of all clusters, wherein the vector database includes multiple clusters, and the multiple clusters are obtained by clustering the metadata; each cluster corresponds to an inverted list, and the metadata is compressed and stored in the inverted list in segments;

[0030] As can be seen, in this application, the metadata in the vector data can be clustered in advance, specifically using the K-Means clustering algorithm to generate n cluster centers for all metadata. Each cluster center corresponds to an inverted list that records all metadata belonging to the cluster.

[0031] Step 13: Based on the distance, determine from all clusters the cluster corresponding to the cluster center closest to the first metadata, and search the inverted list corresponding to the determined cluster to obtain the corresponding second metadata.

[0032] In this regard, in the specific example, the distance between the query metadata q and all cluster centers is calculated, and the nearest nprobe clusters are selected. The larger the value, the higher the recall rate but the slower the speed. Alternatively, a fine-grained search is used, and a brute force search or quantitative decoding is performed in the selected inverted list.

[0033] The criteria for selecting clusters are:

[0034] Top-nprobe of { |i=1, 2, ..., n}

[0035] In the detailed search phase, the distance between the query metadata and the quantized sub-metadata is calculated by looking up the table and accumulating:

[0036] d(q,x)=

[0037] in is the quantitative center of the j-th segment metadata.

[0038] That is, in this application, since the inverted list stores metadata in segmented compression, when performing metadata indexing, it is necessary to accumulate the distances of the queried sub-metadata to obtain the metadata before segmentation.

[0039] Thus, after clustering the metadata based on the clustering algorithm, the present application is equivalent to building a metadata index. Using the clustering-based inverted index method, a corresponding index is built for the metadata (tags) in the vector database, allowing for rapid filtering of metadata based on tags. In other words, before the MCP client performs tag filtering on the query statement in the query request in the vector database, it is necessary to build a metadata index. Based on this, the method of the embodiment of the present application further includes:

[0040] Step 21: obtaining multiple cluster centers based on a preset clustering algorithm, wherein each cluster center corresponds to an inverted list;

[0041] In this regard, in a specific example, the preset clustering algorithm may be a K-Means algorithm, the goal of which is to minimize the intra-cluster variance:

[0042]

[0043] in: represents the i-th cluster, is the cluster center, = , is the square of the Euclidean distance from the metadata x to the cluster center, n is the number of cluster centers, and must satisfy 30*n≤dataset size≤256*n.

[0044] Step 23: All metadata are assigned to the inverted lists corresponding to the nearest cluster centers.

[0045] In this regard, in a specific example, each metadata may be assigned to the inverted list corresponding to the nearest cluster center. Furthermore, the assignment rule for metadata x is: , select the cluster center with the closest Euclidean distance to the metadata x The metadata in the inverted list is segmented and compressed to reduce memory usage. The number of sub-metadata segments is m, and the dimension of each segment is D / m. After quantization, A code book.

[0046] After the metadata is clustered based on the clustering algorithm through steps 21 to 22, the metadata index is constructed. Therefore, by using the clustering-based inverted index method, a corresponding index is constructed for the metadata (label) in the vector database, which can quickly filter the metadata based on the label.

[0047] The tag filtering described in the above embodiment of the present application, i.e., after constructing the metadata index, first performs metadata indexing when performing a data query, and then performs vector indexing after narrowing the scope. Therefore, even if the vectorized knowledge data in the vector database of the present application is large, it can be retrieved quickly and accurately. Based on this, the method of performing similarity search on the query statement in the query request in the vector database involved in the above step 101 can further include:

[0048] Step 31: determining a second vector associated with the second metadata from a vector database, and navigating a corresponding third vector in a graph based on the second vector; wherein the graph includes multiple layers, each layer has multiple nodes, and each node corresponds to a vector;

[0049] Step 32: Determine the similarity between the first vector and the third vector corresponding to the query statement;

[0050] Step 33: Determine the third vector whose similarity is higher than the first threshold as the vector requested by the query request.

[0051] It should be noted that the graph involved in steps 31 to 33 above is an efficient index structure constructed based on the characteristics of the vector in the vector database for rapid vector similarity search. Each layer in the graph includes multiple nodes, each node corresponds to a vector, and the similarity of the vectors in each layer is relatively high. Based on this, since the second vector corresponding to the second metadata obtained based on label filtering is not necessarily the second vector required by the current query statement, it is necessary to find more vectors similar to the second vector in the graph based on the second vector, and then find the required third vector from the more vectors. In this way, the retrieved third vector is a more accurate retrieval result.

[0052] Based on this, the method of navigating the corresponding third vector in the graph based on the second vector in the layered manner involved in step 31 may further include:

[0053] Step 41, searching from the top layer of the graph downwards based on the second vector, and during the search process, placing vectors having a similarity with the second vector higher than a second threshold into a candidate queue;

[0054] Step 42: After the search of the graph from the top layer to the bottom layer is completed, the vector in the candidate queue is determined as the third vector.

[0055] To this end, hierarchical navigation is used to quickly narrow the scope, and a priority queue (priority determined by distance) is used to control the size of the candidate set. During the search process, the candidate queue W is maintained to meet the following requirements:

[0056] |W|≤f and ∀e∈W, distance(q,e)≤threshold

[0057] Where f represents the queue length, distance represents the distance, and threshold represents the threshold.

[0058] In this example, the search process involves a layered traversal, starting from the top-level entry point and refining the search scope layer by layer. Pruning optimization is then performed, discarding candidate nodes that are too far away and retaining the first f nearest neighbors.

[0059] The graph in the embodiment of the present application is an efficient index structure constructed based on the characteristics of vectors for fast vector similarity search. Therefore, before performing label filtering on the query statement in the query request in the vector database, it is necessary to construct a vector index graph. Therefore, the method in the embodiment of the present application may further include:

[0060] Step 51, select a vector from the vector database and insert it into the top-level node of the graph;

[0061] Step 52: distribute the remaining vectors in the vector database to nodes in different layers according to preset distribution probabilities, wherein the preset distribution probabilities corresponding to different layers are different.

[0062] Specifically, step 52 may be as follows: the remaining vectors in the vector database are allocated based on a preset allocation probability; after the remaining vectors in the vector database are allocated, the nodes corresponding to the vectors in each layer whose similarity is higher than a third threshold are determined as neighbors, and bidirectional connections are established between the neighbors.

[0063] As can be seen, in this application, starting from an empty graph, the first inserted node is used as the top-level entry point. Then, each new node is assigned to a different layer according to the above probability, with the highest layer having the lowest probability. Data points are assigned to different levels using a preset probability method, with the upper layer being sparse (for fast navigation) and the lower layer being dense (for precise search). The number of layers is assigned according to the geometric distribution, and the probability of a node entering the level layer is:

[0064] * (1 - )

[0065] in, = ; M is the maximum number of neighbors per layer, that is, the number of nodes per layer. This M is used to control the density of the graph and affects the search efficiency and accuracy.

[0066] In each layer, a new node is connected to its M nearest neighbors. Neighbor selection is based on the cosine similarity distance metric. For node q, the selection of its candidate neighbor set C satisfies:

[0067] ∀e∈C, distance(q,e)≤distance(q,f)

[0068] Among them, f is the farthest node in the current candidate set.

[0069] Specifically, we start at the top layer and insert nodes layer by layer. At each layer, we select the M nearest neighbors to establish bidirectional connections. Bidirectional connections mean that a node can connect to its neighboring nodes, and vice versa, allowing for quick indexing later.

[0070] Furthermore, heuristic pruning can be used during the neighbor selection process to screen neighbors. This involves using path length or cost estimates to estimate the distance to the target, setting a fixed threshold (such as an upper limit on the path length), and pruning branches if they exceed that threshold. Dynamic adjustments are made based on the current optimal solution. If the current path length exceeds the historical optimal solution, subsequent branches are pruned. The child nodes of the current node are sorted by heuristic value, prioritizing expansion of high-potential branches and directly pruning low-priority branches to avoid inefficient searches. The status of previously searched nodes is recorded (memorized pruning) to avoid repeated computations. In the backtracking algorithm, if the current branch fails to meet the constraints, the algorithm immediately backtracks and prunes subsequent child nodes.

[0071] The present application is explained below in conjunction with a specific embodiment of the present application. The specific embodiment provides a method for improving the accuracy and efficiency of large model retrieval knowledge based on vector database index construction and MCP, such as Figure 2 As shown, the method includes the following steps:

[0072] Step 201, prepare the knowledge source to be searched;

[0073] In this specific implementation, knowledge sources include internal enterprise knowledge base files such as Word, Excel, PPT, TXT, and Markdown, databases, publicly available online resources, and external private data sources. Specifically, the MCP server can develop corresponding knowledge acquisition interfaces, including file reading and parsing interfaces, database query interfaces, web crawler interfaces, and private data APIs, based on the MCP protocol. Because the MCP protocol is universal, there's no need to repeatedly write adaptation code for each model's integration with different resources, significantly reducing development workload.

[0074] Step 202: knowledge preprocessing and label association;

[0075] Before knowledge is vectorized, it needs to be preprocessed, such as by extracting keywords and performing entity recognition. Documents are then labeled manually or automatically based on their content or source. For example, a research paper on defect detection in automotive parts production in the industrial sector might be labeled "Industrial Sector," "Automotive Parts," and "Defect Detection." This label information is stored as metadata in a vector database along with the knowledge vector representation.

[0076] Step 203: construct a vector index;

[0077] Vector databases build efficient index structures based on the characteristics of vectors, namely graphs, which can be used to quickly perform similarity searches.

[0078] It should be noted that the layer structure in the graph in this application is similar to a tree structure. Therefore, vectors must be allocated first, that is, vectors are allocated to nodes in the graph. Specifically, in this application, data points (vectors in the vector database) are allocated to different levels using a probabilistic method, with the upper layers being sparse (for fast navigation) and the lower layers being dense (for precise search). The layer allocation follows a geometric distribution, and the probability of a node entering the level is:

[0079] * (1 - )

[0080] in, = ; M is the maximum number of neighbors per layer, which controls the density of the graph and affects the search efficiency and accuracy.

[0081] Then, the neighbors of each layer are selected, that is, in each layer, the new node is connected to the M nearest neighbors. Neighbor selection is based on the cosine similarity distance metric. For node q, the screening of its candidate neighbor set C satisfies:

[0082] ∀e∈C, distance(q,e)≤distance(q,f)

[0083] Among them, e is any node in the neighbor set, f is the farthest node in the current candidate set; each node corresponds to a vector.

[0084] Step 204: construct a metadata index;

[0085] Using the clustering-based inverted index method, the vector database builds corresponding indexes for metadata (tags) to quickly filter based on tags.

[0086] First, the K-Means clustering algorithm is used to generate n cluster centers for all metadata. Each cluster center corresponds to an inverted list, which records all metadata belonging to the cluster.

[0087] The goal of the K-Means algorithm is to minimize the intra-cluster variance:

[0088]

[0089] in: represents the i-th cluster, is the cluster center, = , is the square of the Euclidean distance from the metadata x to the cluster center, n is the number of cluster centers, and must satisfy 30*n≤dataset size≤256*n.

[0090] Then, each metadata is assigned to the inverted list corresponding to the nearest cluster center. The assignment rule for metadata x is: , select the cluster center with the closest Euclidean distance to x . Compress the metadata in the inverted list in segments to reduce memory usage.

[0091] After the metadata index is built, the indexing process is as follows: calculate the distance between the query metadata q and all cluster centers, and select the nearest nprobe clusters. The larger the nprobe value, the higher the recall rate but the slower the speed.

[0092] The criteria for selecting clusters are: Top-nprobe of { |i=1, 2, ..., n}

[0093] In the detailed search phase, the distance between the query metadata and the quantized sub-metadata is calculated by looking up the table and accumulating: d(q, x) = ,in is the quantitative center of the j-th segment metadata.

[0094] Step 205: Build query statement;

[0095] When initiating a query, you must not only provide a vector representation of the query to vectorize it, but also specify tag filtering conditions. The query might include conditions such as WHERE tag = 'contains shrinkage defects' AND category = 'image detection result is Fail'.

[0096] Step 206, hybrid search execution;

[0097] When a vector database receives a hybrid query request, it simultaneously performs a vector similarity search and metadata filtering. A common strategy is to first perform metadata filtering to narrow the search scope, then perform a vector similarity search on the filtered result set. This approach can significantly improve query efficiency, especially when dealing with large amounts of data. Another strategy is to perform a vector search and metadata filtering in parallel, then perform an intersection operation on the results.

[0098] Step 207: Return the filtered results. That is, the vector database will return knowledge that meets both the vector similarity requirements and the tag filtering conditions. The filtered results are then sent to the locally deployed large model, which summarizes them and sends them to the MCP client, where they are displayed on the MCP service host's page.

[0099] Through steps 201 to 207, the MCP client acts as the interface between the user and the locally deployed large model, receiving user instructions and retrieved external knowledge, passing them to the model for processing. After the local large model performs inductive reasoning, it returns the generated results to the MCP client via the standard MCP protocol for display. In other words, this application connects local, cloud, public, and private data sources, enabling the retrieval and summary of internal company documents, knowledge stored in databases, and other private or public data sources. New data sources can be added without code modifications, achieving plug-and-play functionality. This shortens the retrieval process and produces more accurate results than existing technologies.

[0100] Corresponding to the above Figure 1 , the embodiment of the present application also provides a data retrieval device, which is applied to the MCP client, such as Figure 3 As shown, the device includes:

[0101] The first processing module 302 is configured to perform tag filtering and similarity search on the query statement in the query request in a vector database when the MCP client receives a query request. The vector database includes vectors corresponding to different types of knowledge data and metadata associated with the vectors, wherein the metadata represents the tag information of the vectors. Different types of knowledge data are obtained by vectorizing the acquired data using different application programming interfaces (APIs). Different APIs are developed based on the MCP protocol.

[0102] A sending module 304 is used to send the vectors obtained by similarity search and label filtering to a locally deployed large model;

[0103] The second processing module 306 is used to receive the results obtained by the large model through inductive reasoning on the knowledge data corresponding to the vector, and display the results.

[0104] The device of the embodiment of the present application can obtain knowledge data from different sources through different APIs, and then vectorize the obtained knowledge data and store it in a vector database. Based on this, after receiving the query request, the MCP client performs label filtering and similarity search on the query statement in the query request in the vector database. Since the vector in the vector database is not the vector of the knowledge data obtained through a single channel, the knowledge data obtained after vectorization through label filtering and similarity search is more accurate. Moreover, compared with the single retrieval method in the prior art, the label filtering and similarity search are not only efficient but also have high retrieval accuracy. In addition, in the present application, since the retrieval results are sent to the large model through the MCP client with a universal protocol, the various large models can be seamlessly linked so that the retrieval results can be efficiently summarized and reasoned and their reasoning results can be displayed. It can be seen that through the method in the embodiment of the present application, not only can the retrieval in the database be performed quickly, but the retrieval accuracy is also high, that is, it can meet the expectations of the retrieval needs.

[0105] In an optional implementation manner of an embodiment of the present application, the first processing module in the embodiment of the present application may further include: a first processing unit, used to determine a first label of a query statement, and determine the first label as the first metadata of the query statement; a second processing unit, used to determine the distance between the first metadata and the cluster centers of all clusters, wherein the vector database includes multiple clusters, and the multiple clusters are obtained by clustering metadata; each cluster corresponds to an inverted list, and the metadata is segmented and compressed and stored in the inverted list; a third processing unit, used to determine, from all clusters, based on the distance, the cluster corresponding to the cluster center closest to the first metadata, and search the inverted list corresponding to the determined cluster to obtain the corresponding second metadata.

[0106] In an optional implementation manner of an embodiment of the present application, the first processing module in the embodiment of the present application may further include: a fourth processing unit, used to determine a second vector associated with the second metadata from a vector database, and navigate out a corresponding third vector in a layered manner in a graph based on the second vector; wherein the graph includes multiple layers, each layer has multiple nodes, and each node corresponds to a vector; a fifth processing unit, used to determine the similarity between the first vector and the third vector corresponding to the query statement; and a sixth processing unit, used to determine the third vector whose similarity is higher than the first threshold as the vector requested by the query request.

[0107] In an optional implementation manner of an embodiment of the present application, the fourth processing unit in the embodiment of the present application may further include: a first processing sub-unit, used to search downward from the top layer of the graph based on the second vector, and during the search process, put vectors whose similarity with the second vector is higher than a second threshold into a candidate queue; a second processing sub-unit, used to determine the vector in the candidate queue as the third vector after completing the search of the graph from the top layer to the bottom layer.

[0108] In an optional implementation manner of the embodiment of the present application, the device in the embodiment of the present application may also include: a third processing module, used to select a vector from the vector database and insert it into the top-level node of the graph before performing label filtering on the query statement in the query request in the vector database; a fourth processing module, used to distribute the remaining vectors in the vector database to nodes in different layers according to preset distribution probabilities, wherein the preset distribution probabilities corresponding to different layers are different.

[0109] In an optional implementation manner of an embodiment of the present application, the fourth processing module in the embodiment of the present application may further include: a seventh processing unit, used to determine the preset allocation probability of the remaining vectors in the vector database entering each layer; an eighth processing unit, used to allocate the remaining vectors in the vector database based on the preset allocation probability, and after the remaining vectors in the vector database are allocated, the nodes corresponding to the vectors in each layer whose similarity is higher than the third threshold are determined as neighbors, and bidirectional connections are established between the neighbors.

[0110] In an optional implementation manner of the embodiment of the present application, the device in the embodiment of the present application also includes: a fifth processing module, which is used to obtain multiple cluster centers based on a preset clustering algorithm before performing label filtering on the query statement in the query request in the vector database, wherein each cluster center corresponds to an inverted list; and a sixth processing module, which is used to distribute all metadata to the inverted lists corresponding to the cluster centers closest to them.

[0111] like Figure 4 As shown, an embodiment of the present application provides a terminal, including a processor 411, a communication interface 412, a memory 413 and a communication bus 414, wherein the processor 411, the communication interface 412, and the memory 413 communicate with each other through the communication bus 414.

[0112] Memory 413, for storing computer programs;

[0113] In one embodiment of the present application, the processor 411 is used to execute the program stored in the memory 413 to implement the data retrieval method provided by any of the aforementioned method embodiments, and its role is similar and will not be repeated here.

[0114] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the data retrieval method provided in any of the aforementioned method embodiments are implemented.

[0115] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of this embodiment.

[0116] Through the description of the above embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a general hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the relevant technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0117] It should be understood that the terms used herein are for the purpose of describing specific example embodiments only and are not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms "one", "an" and "said" as used herein may also be meant to include plural forms. The terms "comprise", "include", "contain" and "have" are inclusive and therefore specify the presence of stated features, steps, operations, elements and / or parts, but do not exclude the presence or addition of one or more other features, steps, operations, elements, parts, and / or combinations thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring them to be performed in the specific order described or illustrated, unless the order of execution is clearly indicated. It should also be understood that additional or alternative steps may be used.

[0118] The foregoing description is intended only to provide specific embodiments of the present invention, which will enable those skilled in the art to understand and implement the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not intended to be limited to the embodiments shown herein, but is intended to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A data retrieval method, applied to a Model Context Protocol (MCP) client, characterized in that: The method comprises: When an MCP client receives a query request, it performs tag filtering and similarity search on the query statement in the query request in a vector database. The vector database includes vectors corresponding to different types of knowledge data and metadata associated with the vectors, wherein the metadata represents the label information of the vectors. Different types of knowledge data are obtained by vectorizing them after being acquired through different application programming interfaces (APIs). Different APIs are developed based on the MCP protocol. Send the vectors obtained through similarity search and label filtering to the locally deployed large model; Receive the result obtained by the large model through inductive reasoning on the knowledge data corresponding to the vector, and display the result.

2. The method according to claim 1, characterized in that Performing label filtering on the query statement in the query request in the vector database includes: Determining a first tag of the query statement, and determining the first tag as first metadata of the query statement; Determining the distance between the first metadata and the cluster centers of all clusters, wherein the vector database includes multiple clusters, and the multiple clusters are obtained by clustering the metadata; each of the clusters corresponds to an inverted list, and the metadata is compressed and stored in segments in the inverted lists; Based on the distance, a cluster corresponding to a cluster center closest to the first metadata is determined from all clusters, and the corresponding second metadata is obtained by searching from an inverted list corresponding to the determined cluster.

3. The method according to claim 2, characterized in that Performing a similarity search on the query statement in the query request in a vector database includes: Determining a second vector associated with the second metadata from the vector database, and navigating a corresponding third vector in a graph based on the second vector, wherein the graph includes multiple layers, each layer has multiple nodes, and each node corresponds to a vector; Determining the similarity between the first vector corresponding to the query statement and the third vector; A third vector having a similarity higher than a first threshold is determined as the vector requested by the query request.

4. The method according to claim 3, characterized in that Navigating hierarchically in the graph based on the second vector to obtain a corresponding third vector includes: Searching downwards from the top layer of the graph based on the second vector, and placing vectors having a similarity with the second vector higher than a second threshold into a candidate queue during the search process; After the search of the graph from the top layer to the bottom layer is completed, the vector in the candidate queue is determined as the third vector.

5. The method according to claim 3, characterized in that Before performing label filtering on the query statement in the query request in the vector database, the method further includes: Select a vector from the vector database and insert it into the top-level node of the graph; The remaining vectors in the vector database are distributed to nodes in different layers according to preset distribution probabilities, wherein the preset distribution probabilities corresponding to different layers are different.

6. The method according to claim 5, characterized in that Allocating the remaining vectors in the vector database to nodes in different layers according to preset allocation probabilities includes: Determining a preset allocation probability of the remaining vectors in the vector database entering each layer; The remaining vectors in the vector database are allocated based on the preset allocation probability. After the remaining vectors in the vector database are allocated, the nodes corresponding to the vectors with similarity higher than a third threshold in each layer are determined as neighbors, and bidirectional connections are established between the neighbors.

7. The method according to claim 2, characterized in that Before performing label filtering on the query statement in the query request in the vector database, the method further includes: Based on the preset clustering algorithm, multiple cluster centers are obtained, where each cluster center corresponds to an inverted list; All metadata are assigned to the inverted lists corresponding to the nearest cluster centers.

8. The method according to claim 1, characterized in that Receiving the results obtained by the large model through inductive reasoning of the knowledge, and displaying the results includes: Receiving the result returned by the large model through the MCP protocol, wherein the result is obtained by the large model performing inductive reasoning on the knowledge data corresponding to the vector; The result is displayed on the MCP service host.

9. A data retrieval device, applied to an MCP client, characterized in that: The device comprises: A first processing module is configured to, upon receiving a query request from an MCP client, perform tag filtering and similarity search on a query statement in the query request in a vector database; wherein the vector database includes vectors corresponding to different types of knowledge data and metadata associated with the vectors, wherein the metadata represents label information of the vectors. Different types of knowledge data are obtained by vectorizing the acquired data using different application programming interfaces (APIs); and different APIs are developed based on the MCP protocol. The sending module is used to send the vectors obtained through similarity search and label filtering to the locally deployed large model; The second processing module is used to receive the results obtained by the large model through inductive reasoning on the knowledge data corresponding to the vector, and display the results.

10. A terminal, characterized in that: include: at least one communication interface; at least one bus connected to the at least one communication interface; at least one processor coupled to the at least one bus; At least one memory connected to the at least one bus, wherein the processor is configured to execute the data retrieval method according to any one of claims 1 to 8.

Images