Vector data retrieval method and device, equipment and storage medium
By extracting the filter conditions in the vector data retrieval request and predicting the sparseness of the result set, the vector data retrieval process is optimized, and the problem of low vector data retrieval efficiency in the prior art is solved, achieving more efficient data recall.
Patent Information
- Application Number
- CN202310988868.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-07
- Publication Date
- 2025-02-18
AI Technical Summary
In the prior art, vector data retrieval efficiency is not high and the established data cannot be effectively recalled.
When a vector data retrieval request is received, the filter condition is extracted, and when the filter condition is uncertain, the sparseness of the result set in the data set is predicted, and vector data retrieval is performed based on the sparseness and filter conditions.
By predicting the sparseness degree and optimizing the search strategy, the efficiency of vector data retrieval is improved and relevant data can be recalled more quickly.
Smart Images
Figure CN119474290A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer technology, and in particular to a vector data retrieval method, device, equipment and storage medium. Background Art
[0002] In the process of optimizing the quality of large model dialogue, it is necessary to obtain sample data that is more relevant to the problem in real time as the prompt word input of the large model. In the related technology, auxiliary data that can be used for large model dialogue is generally extracted from the knowledge base through vector retrieval. Vector retrieval is used to retrieve sample data similar to the target vector in the vector knowledge base. Considering the performance problem of vector retrieval, the retrieval method in the related technology is to first obtain the predicted distribution corresponding to the target vector through a pre-trained distribution prediction model, and then search according to the predicted distribution. Although the above-mentioned related technology has narrowed the traversal range when performing vector retrieval, the distribution prediction model can only predict the predicted distribution corresponding to the target vector. To complete the vector retrieval, it is still necessary to traverse and search the vectors in the predicted distribution, and then return the topN results with similar distances, and then filter through the topN results. The above-mentioned vector retrieval method has technical problems such as inability to recall established data and low retrieval efficiency.
[0003] The above contents are only used to assist in understanding the technical solution of the present invention and do not constitute an admission that the above contents are prior art. Summary of the invention
[0004] The main purpose of the present invention is to provide a vector data retrieval method, device, equipment and storage medium, aiming to solve the technical problem of low efficiency of vector data retrieval in the prior art.
[0005] To achieve the above object, the present invention provides a vector data retrieval method, which comprises the following steps:
[0006] When receiving a vector data retrieval request, extracting a filtering condition in the vector data retrieval request;
[0007] When the filtering condition is uncertainty filtering, predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition;
[0008] Vector data is retrieved from the data set based on the sparsity level and the filtering condition.
[0009] Optionally, the step of performing vector data retrieval in the vector data set based on the sparsity degree and the filtering condition includes:
[0010] When the sparsity level is non-sparse, obtaining an expected recall quantity;
[0011] Determining the vector index recall quantity according to the expected recall quantity and sparsity;
[0012] Vector data is retrieved from the data set based on the vector index recall quantity and the filter condition.
[0013] Optionally, the step of performing vector data retrieval in the data set based on the vector index recall quantity and the filtering condition includes:
[0014] Based on the filtering condition, performing non-vector data search in the data set to obtain a search result;
[0015] Vector data is retrieved in the search results according to the vector index recall quantity.
[0016] Optionally, the step of performing vector data retrieval in the vector data set based on the sparsity degree and the filtering condition includes:
[0017] When the sparsity degree is sparse, filtering non-vector data in the data set according to the filtering condition to obtain initial filtered data;
[0018] Vector data is retrieved from the initial filtered data based on the filtering condition.
[0019] Optionally, after the step of extracting the filtering condition in the vector data retrieval request when the vector data retrieval request is received, the method further includes:
[0020] Determining whether the filtering condition is a deterministic filtering;
[0021] When the filtering condition is a deterministic filtering, determining the retrieval data set according to the filtering condition;
[0022] Data is retrieved in the retrieval data set according to the filtering condition.
[0023] Optionally, when the filtering condition is a deterministic filtering, the step of determining the retrieval data set according to the filtering condition includes:
[0024] When the filtering condition is a deterministic filtering, determining a data set corresponding to each target filtering condition in the filtering condition;
[0025] A search data set is constructed according to the data set.
[0026] Optionally, the step of predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition includes:
[0027] Determining a filter condition type of the filter condition;
[0028] Constructing an inverted index of the result set according to the filtering condition type;
[0029] The sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the inverted index and the filtering condition.
[0030] Optionally, the step of predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the inverted index and the filtering condition includes:
[0031] Determine a retrieval data set according to the inverted index and the filtering condition;
[0032] Extracting a target retrieval sample set from the retrieval data set;
[0033] Performing vector data retrieval in the target retrieval sample set based on the filtering condition to obtain a retrieval result;
[0034] The sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the retrieval result and the target retrieval sample set.
[0035] Optionally, the step of constructing an inverted index of the result set according to the filtering condition type includes:
[0036] When the filtering condition type is a numerical index, determine the data that hits the same numerical value, and use the numerical value as a leaf node;
[0037] According to a preset construction strategy, the leaf nodes are combined to construct a parent node, and the parent node is used as a target node;
[0038] Build upward according to the target node and the preset building strategy to obtain an inverted index of the result set.
[0039] Optionally, the step of constructing an inverted index of the result set according to the filtering condition type includes:
[0040] When the filtering condition type is a text index, each data in the result set is segmented to obtain a segmentation result;
[0041] An inverted index of the result set is constructed according to the word segmentation results.
[0042] Optionally, the step of performing vector data retrieval in the data set based on the sparsity degree and the filtering condition includes:
[0043] Determining the number of threads corresponding to the sparsity level;
[0044] Concurrent threads are constructed according to the number of threads, and vector data retrieval is performed based on the constructed concurrent threads.
[0045] In addition, to achieve the above object, the present invention also provides a vector data retrieval device, the device comprising:
[0046] An extraction module, configured to extract a filtering condition in a vector data retrieval request when receiving the vector data retrieval request;
[0047] A prediction module, used for predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition when the filtering condition is uncertainty filtering;
[0048] A retrieval module is used to retrieve vector data in the data set based on the sparsity level and the filtering condition.
[0049] In addition, to achieve the above-mentioned purpose, the present invention also proposes a vector data retrieval device, which includes: a memory, a processor, and a vector data retrieval program stored in the memory and executable on the processor, wherein the vector data retrieval program is configured to implement the steps of the vector data retrieval method described above.
[0050] In addition, to achieve the above-mentioned purpose, the present invention also proposes a storage medium, on which a vector data retrieval program is stored, and when the vector data retrieval program is executed by a processor, the steps of the vector data retrieval method described above are implemented.
[0051] When receiving a vector data retrieval request, the present invention extracts the filtering condition in the vector data retrieval request; when the filtering condition is an uncertainty filter, the sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the filtering condition; and vector data is retrieved in the data set based on the sparsity and the filtering condition. Because the present invention predicts the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition when the filtering condition is an uncertainty filter; and vector data is retrieved in the data set based on the sparsity and the filtering condition. Compared with the existing method of traversing the entire data set for retrieval, the above method of the present invention can improve the efficiency of vector retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] Figure 1 It is a structural schematic diagram of a vector data retrieval device in a hardware operating environment involved in an embodiment of the present invention;
[0053] Figure 2 It is a schematic diagram of the flow chart of the first embodiment of the vector data retrieval method of the present invention;
[0054] Figure 3 It is a flowchart of the second embodiment of the vector data retrieval method of the present invention;
[0055] Figure 4 It is a schematic diagram of the flow chart of the third embodiment of the vector data retrieval method of the present invention;
[0056] Figure 5 This is a structural block diagram of the first embodiment of the vector data retrieval device of the present invention.
[0057] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION
[0058] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.
[0059] Reference Figure 1 , Figure 1 It is a schematic diagram of the structure of a vector data retrieval device in the hardware operating environment involved in the embodiment of the present invention.
[0060] like Figure 1 As shown, the vector data retrieval device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a wireless fidelity (Wireless-Fidelity, WI-FI) interface). The memory 1005 may be a high-speed random access memory (Random Access Memory, RAM), or a stable non-volatile memory (Non-Volatile Memory, NVM), such as a disk storage. The memory 1005 may also be a storage device independent of the aforementioned processor 1001.
[0061] Those skilled in the art will understand that Figure 1 The structure shown in does not constitute a limitation on the vector data retrieval device, and may include more or less components than those shown in the figure, or a combination of certain components, or a different arrangement of components.
[0062] like Figure 1 As shown, the memory 1005 as a storage medium may include an operating system, a network communication module, a user interface module, and a vector data retrieval program.
[0063] exist Figure 1 In the vector data retrieval device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the vector data retrieval device of the present invention can be set in the vector data retrieval device, and the vector data retrieval device calls the vector data retrieval program stored in the memory 1005 through the processor 1001, and executes the vector data retrieval method provided by the embodiment of the present invention.
[0064] Based on the above-mentioned vector data retrieval device, an embodiment of the present invention provides a vector data retrieval method, referring to Figure 2 , Figure 2 Schematic diagram of the flow chart of the first embodiment of the vector data retrieval method of the present invention.
[0065] In this embodiment, the vector data retrieval method includes the following steps:
[0066] Step S10: when a vector data search request is received, a filter condition in the vector data search request is extracted.
[0067] It should be noted that the execution subject of this embodiment may be a vector data retrieval device, in which a data retrieval program (also referred to as an intelligent assistant or intelligent big model) is provided, and the vector data retrieval device may be a computing service device with data processing, network communication and program running functions, such as a mobile phone, a tablet computer, a personal computer, etc., or an electronic device capable of realizing the above functions. The following takes the vector data retrieval device as an example to illustrate this embodiment and the following embodiments.
[0068] It should be noted that the filtering conditions may include deterministic exhaustible filtering conditions and / or uncertain filtering conditions. The deterministic exhaustible filtering conditions may be filtering conditions with certain parameter values such as gender, height, grades, etc. The uncertain filtering conditions may be retrieval conditions that cannot accurately classify data according to the filtering conditions, for example, recalling data by calculating face similarity or recalling data by text similarity.
[0069] It should be noted that the application scenario of this embodiment is that each condition in the filter condition is a deterministic filter condition, or there are both deterministic filter conditions and uncertain filter conditions. For vector filtering with all uncertain filter conditions, the search is performed directly by traversing the entire data set.
[0070] Furthermore, in order to improve the efficiency of vector data retrieval, after step S10, the method further includes: determining whether the filtering condition is a deterministic filtering;
[0071] When the filtering condition is a deterministic filtering, determining the retrieval data set according to the filtering condition;
[0072] Data is retrieved in the retrieval data set according to the filtering condition.
[0073] It should be noted that the retrieval data set may be a data set that satisfies the filtering condition and corresponds to the filtering condition. For example, if the deterministic filtering is that the height is greater than 1.7 meters, the retrieval data set may be a data set of height greater than 1.7 meters. If the deterministic filtering is that the height is greater than 1.7 meters and the gender is male, the retrieval data set may include a data set of height greater than 1.7 meters and a data set of male gender. The data retrieval in the retrieval data set according to the filtering condition may be data retrieval in the retrieval data set according to the filtering condition in a vector index-based manner.
[0074] It should be understood that when the filtering condition is a deterministic filter, the data set can be divided into multiple data sets through a combination of filtering conditions, and a data file is constructed for each subset. The vector index of which set is to be retrieved is determined through the filtering condition, thereby reducing the data scale and improving the retrieval efficiency. While meeting the hard filtering conditions, the most similar result set that meets the recall number can also be recalled through vector recall.
[0075] Furthermore, in order to improve data retrieval efficiency, when the filtering condition is a deterministic filtering, the step of determining the retrieval data set according to the filtering condition includes:
[0076] When the filtering condition is a deterministic filtering, determining a data set corresponding to each target filtering condition in the filtering condition;
[0077] A search data set is constructed according to the data set.
[0078] It should be noted that the filtering condition may include at least one condition, for example, the filtering condition is height greater than 1.7 meters, gender is male, and grade is A. The target filtering condition may be one or a group of conditions of the filtering conditions, for example, it may be a height greater than 1.7 meters, or it may be a combination of conditions of gender is male and grade is A. The data set corresponding to the target filtering condition may be a data set that satisfies the target filtering condition. The constructing of the retrieval data set according to the data set may be combining the data sets corresponding to the respective target filtering conditions as the retrieval data set.
[0079] Step S20: When the filtering condition is uncertainty filtering, predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition.
[0080] It should be noted that the data set may be a vector knowledge base storing a large amount of vector data, and the vector knowledge base may be a database specifically used for storing, managing, querying, and retrieving vectors, and is mainly used in the fields of artificial intelligence, machine learning, data mining, etc. The method of predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering conditions may be to select a certain number of samples from the data set, perform a search in the selected samples according to the filtering conditions, and determine the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the retrieval results in the samples and the number of samples. Specifically, it may be: sparsity = number of data in the retrieval results / number of sample data.
[0081] Step S30: performing vector data retrieval in the data set based on the sparsity level and the filtering condition.
[0082] It should be noted that the vector data retrieval in the data set based on the sparsity and the filtering condition can be that when the sparsity is lower than the preset sparsity threshold, the data set is filtered by the non-vector filtering condition according to the filtering condition to obtain the filtering result, the data in the filtering result is traversed, the correlation between the traversed data and the uncertainty filtering condition in the filtering condition is calculated respectively, and the result of the number of users expected to recall is returned according to the correlation. When the sparsity is greater than or equal to the preset sparsity threshold, the data set is filtered by the non-vector filtering condition according to the filtering condition to obtain the filtering result, and then the vector retrieval is performed in the filtering result according to the uncertainty filtering condition in the filtering condition to obtain the vector retrieval result, the data in the vector retrieval result is traversed, the correlation between the traversed data and the uncertainty filtering condition in the filtering condition is calculated respectively, and the result of the number of users expected to recall is returned according to the correlation. Among them, the preset sparsity threshold can be a pre-set sparsity threshold, which can be customized according to the actual scenario, and this embodiment is not limited here.
[0083] Further, if the sparsity of the result set in the data set is relatively sparse, that is, the number of result sets is small, then there may be a small number of recalled data retrieval results or no recalled data may be obtained. Therefore, in order to recall the given data, the step S30 may include: when the sparsity is sparse, filtering non-vector data in the data set according to the filtering condition to obtain initial filtered data;
[0084] Vector data is retrieved from the initial filtered data based on the filtering condition.
[0085] It should be noted that whether the sparseness is sparse can be determined by a preset sparseness threshold. When the sparseness is less than the preset sparseness threshold, the sparseness is determined to be non-sparse.
[0086] It should be noted that the filtering of non-vector data in the data set according to the filtering conditions to obtain the initial filtered data may be filtering of non-vector data in the data set according to the deterministic filtering conditions in the filtering conditions to obtain the initial filtered data. That is, when the sparsity is sparse, the initial filtered data is directly recalled based on the non-vector filtering conditions instead of being recalled by vector indexing. The vector data retrieval in the initial filtered data based on the filtering conditions may be calculating the similarity between each data in the initial filtered data and the uncertain filtering conditions in the filtering conditions, and determining the data retrieval result based on the similarity and the number of recalls expected by the user.
[0087] It should be understood that in the process of training and / or optimizing the model, sample data needs to be collected, and the quality of the sample data can affect the training and optimization results of the model, especially in the process of optimizing the model, it is necessary to select answer information that is more matched with the input question and answer information, and then optimize the model based on the question and answer information and the selected answer information, so that the answer information output by the model is more matched with the question and answer information. This embodiment can be applied to the scenario of retrieving answer information that is more matched with the question and answer information from multiple answer information during model training. Specifically, it can be to generate a target answer that matches the question and answer information based on the question and answer information, retrieve answer information that is more matched with the question and answer information from multiple answer information output by the model (or vector knowledge base) based on the target answer, and optimize the model based on the retrieved answer information.
[0088] When receiving a vector data retrieval request, this embodiment extracts the filtering condition in the vector data retrieval request; when the filtering condition is uncertainty filtering, the sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the filtering condition; and vector data is retrieved in the data set based on the sparsity and the filtering condition. Since this embodiment predicts the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition when the filtering condition is uncertainty filtering; and vector data is retrieved in the data set based on the sparsity and the filtering condition. Compared with the existing method of traversing the entire data set for retrieval, the above method of this embodiment can improve the efficiency of vector retrieval.
[0089] refer to Figure 3 , Figure 3 Schematic diagram of the flow chart of the second embodiment of the vector data retrieval method of the present invention.
[0090] Based on the above first embodiment, in this embodiment, step S30 includes:
[0091] Step S301: when the sparsity level is non-sparse, obtaining the expected recall quantity.
[0092] It should be noted that the sparsity level can be determined by a preset sparsity level threshold to determine whether the sparsity level is non-sparse. When the sparsity level is greater than or equal to the preset sparsity level threshold, the sparsity level is determined to be non-sparse. The expected recall quantity can be the number of data to be recalled specified by the user.
[0093] Step S302: determining the vector index recall quantity according to the expected recall quantity and sparsity.
[0094] It should be noted that determining the vector index recall quantity according to the expected recall quantity and the sparsity level may be obtaining the vector index recall quantity by dividing the expected recall quantity by the sparsity level.
[0095] Step S303: performing vector data retrieval in the data set based on the vector index recall quantity and the filtering condition.
[0096] It should be noted that the vector data retrieval in the data set based on the vector index recall number and the filtering condition may be filtering the data set according to the deterministic filtering condition in the filtering condition to obtain a filtering result, and then retrieving vector data in the filtering result based on the uncertain filtering condition in the filtering condition and the vector index recall number.
[0097] Further, in order to ensure that the number of recalled data meets the number of recalled data expected by the user, the step S303 may include: performing a non-vector data search in the data set based on the filtering condition to obtain a search result;
[0098] Vector data is retrieved in the search results according to the vector index recall quantity.
[0099] It should be noted that, in this embodiment, when the filtering condition is an uncertain filtering condition, it indicates that the filtering condition contains both an uncertain filtering condition and a deterministic filtering condition. The non-vector data retrieval in the data set based on the filtering condition to obtain the retrieval result can be filtering the data in the data set that meets the deterministic filtering condition by non-vector data filtering based on the deterministic filtering condition in the filtering condition to obtain the retrieval result. The vector data retrieval in the retrieval result based on the vector index recall quantity can be recalling the vector index recall quantity data in the retrieval result by the vector index based on the uncertain filtering condition in the filtering condition to obtain the recall result, and then returning the expected recall quantity data, i.e., the vector retrieval result, based on the similarity between each data in the recall result and the filtering condition.
[0100] In this embodiment, when the sparsity level is non-sparse, the expected recall number is obtained; the vector index recall number is determined according to the expected recall number and the sparsity level; non-vector data is searched in the data set based on the filtering conditions to obtain a search result; and vector data is searched in the search result according to the vector index recall number. In this embodiment, when the sparsity level is non-sparse, in order to ensure that the number of vector data retrieval recalls meets the user's expected data recall number and improve data retrieval efficiency, data is first filtered through deterministic filtering conditions, and then vector data is searched in the filtering results according to the uncertain filtering conditions in the filtering conditions and the vector index recall number. Determining the number of recalls meets the user's expectations and also avoids searching for all the data in the data set through vectors, thereby improving data retrieval efficiency.
[0101] refer to Figure 4 , Figure 4 It is a flowchart diagram of the third embodiment of the vector data retrieval method of the present invention.
[0102] Based on the above embodiments, in this embodiment, step S20 includes:
[0103] Step S201: Determine the filtering condition type of the filtering condition.
[0104] It should be noted that the filtering condition type includes text and / or numerical type.
[0105] Step S202: construct an inverted index of the result set according to the filtering condition type.
[0106] It should be noted that the inverted index of the result set constructed according to the filtering condition type may be constructed by extracting feature information of each data in the data set according to the filtering condition type, and taking data with the same features as leaf nodes according to the extracted feature information to construct the inverted index of the result set.
[0107] Further, in order to improve the efficiency of data retrieval, the step S202 may include: when the filtering condition type is a numerical index, determining data that hits the same numerical value, and using the numerical value as a leaf node;
[0108] According to a preset construction strategy, the leaf nodes are combined to construct a parent node, and the parent node is used as a target node;
[0109] Build upward according to the target node and the preset building strategy to obtain an inverted index of the result set.
[0110] When the filtering condition type is a text index, each data in the result set is segmented to obtain a segmentation result;
[0111] An inverted index of the result set is constructed according to the word segmentation results.
[0112] It should be noted that the combining of the leaf nodes to construct the parent node according to the preset construction strategy may be combining multiple leaf nodes within a certain range to construct the parent node. For example, the leaf nodes include leaf nodes with values 2, 3, 6, 7, and 8. Combine leaf nodes 2 and 3 to construct a parent node with a value range of 2-3. Combine leaf nodes 6, 7, and 8 to construct a parent node with a value range of 6-8. The leaf nodes may be combined according to the size of the data corresponding to each leaf node, or the parent node may be constructed according to the preset number of leaf nodes. The combination method of the leaf nodes may be customized according to the specific scenario, and this embodiment is not limited here. For the parent node constructed after the leaf nodes are combined, the parent node may be further combined as the target node to construct the parent node of the target node, thereby constructing upward to form a multi-branch tree, in which each node points to an inverted chain data containing a node value.
[0113] It should be noted that when the filter condition type is a text index, it means that each data in the data set is in text form. At this time, the data can be segmented to obtain the segmentation results, and the inverted index of the result set can be constructed based on the data that hits the same words in the segmentation results.
[0114] Step S203: predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the inverted index and the filtering condition.
[0115] It should be noted that the prediction of the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the inverted index and the filtering condition may be to determine the data set to be retrieved corresponding to the filtering condition according to the inverted index and the filtering condition, select a portion of the sample data set from the data set to be retrieved, perform data retrieval based on the sample data set and the filtering condition, obtain the retrieval result, and determine the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the retrieval result and the sample data set. For example, the leaf nodes of the inverted index include leaf nodes with values 2, 3, 6, 7, and 8. The parent nodes corresponding to leaf nodes 2 and 3 are 2-3, and the parent nodes corresponding to leaf nodes 6, 7, and 8 are 6-8. If the filtering condition is to retrieve data with a value of 6, then the data set to be retrieved determined according to the inverted index and the filtering condition may be the data corresponding to leaf node 6. If the filtering condition is to retrieve data with a value of 6-8, then the data set to be retrieved determined according to the inverted index and the filtering condition may be the data set corresponding to parent nodes 6-8. In order to improve the retrieval efficiency, in the above embodiments, when searching in a data set according to a filtering condition, the data set to be retrieved corresponding to the filtering condition can be found according to the inverted index corresponding to the data set, and the data set to be retrieved can be used as the data set, and data filtering can be performed in the data set to be retrieved.
[0116] Further, in order to improve data retrieval efficiency, the step S203 may include: determining a retrieval data set according to the inverted index and the filtering condition;
[0117] Extracting a target retrieval sample set from the retrieval data set;
[0118] Performing vector data retrieval in the target retrieval sample set based on the filtering condition to obtain a retrieval result;
[0119] The sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the retrieval result and the target retrieval sample set.
[0120] It should be noted that the target retrieval sample set may be a portion of data selected from the retrieval data set, used to predict the sparsity of the result set in the data set. The vector data retrieval in the target retrieval sample set based on the filtering condition to obtain the retrieval result may be to retrieve the data satisfying the filtering condition in the target retrieval sample set to obtain the retrieval result. The prediction of the sparsity of the result set corresponding to the vector data retrieval request in the data set based on the retrieval result and the target retrieval sample set may be sparsity = number of data items in the retrieval result / number of data items in the target retrieval sample set.
[0121] Furthermore, in order to improve the speed of data retrieval, the step S30 may include: determining the number of threads corresponding to the sparsity level; constructing concurrent threads according to the number of threads, and performing vector data retrieval based on the constructed concurrent threads.
[0122] It should be noted that when performing data retrieval, the number of concurrent threads can be dynamically adjusted according to the size of the result set, and data retrieval can be performed simultaneously through multiple concurrent threads to improve the efficiency of data retrieval. The sparsity reflects the size of the result set to a certain extent. Therefore, the number of threads corresponding to different sparsity levels can be pre-set, and then vector data retrieval can be performed according to concurrent threads. Furthermore, in order to accurately determine the number of threads, the number of result sets can be predicted according to the sparsity, the target retrieval sample set and the retrieval data set, and the number of concurrent threads can be determined according to the number of result sets.
[0123] This embodiment determines the filter condition type of the filter condition; constructs an inverted index of the result set according to the filter condition type; and predicts the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the inverted index and the filter condition. This embodiment constructs an inverted index of the data set according to the filter condition type of the filter condition, and can select the data set to be retrieved corresponding to the filter condition from the data set through the inverted index of the data set during data retrieval, thereby reducing the size of the data set during retrieval and improving data retrieval efficiency.
[0124] Reference Figure 5 , Figure 5 This is a structural block diagram of the first embodiment of the vector data retrieval device of the present invention.
[0125] like Figure 5 As shown, the vector data retrieval device proposed in the embodiment of the present invention includes:
[0126] The extraction module 10 is used to extract the filtering condition in the vector data retrieval request when receiving the vector data retrieval request;
[0127] A prediction module 20, configured to predict the sparsity of a result set corresponding to the vector data retrieval request in a data set according to the filtering condition when the filtering condition is uncertainty filtering;
[0128] The retrieval module 30 is used to retrieve vector data in the data set based on the sparsity level and the filtering condition.
[0129] When receiving a vector data retrieval request, this embodiment extracts the filtering condition in the vector data retrieval request; when the filtering condition is uncertainty filtering, the sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the filtering condition; and vector data is retrieved in the data set based on the sparsity and the filtering condition. Since this embodiment predicts the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition when the filtering condition is uncertainty filtering; and vector data is retrieved in the data set based on the sparsity and the filtering condition. Compared with the existing method of traversing the entire data set for retrieval, the above method of this embodiment can improve the efficiency of vector retrieval.
[0130] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of the present invention. In practical applications, technicians in this field can select part or all of them according to actual needs to achieve the purpose of the present embodiment, and no limitation is made here.
[0131] In addition, for technical details that are not described in detail in this embodiment, reference can be made to the vector data retrieval method provided in any embodiment of the present invention, and will not be repeated here.
[0132] Based on the above-mentioned first embodiment of the vector data retrieval device of the present invention, a second embodiment of the vector data retrieval device of the present invention is proposed.
[0133] In this embodiment, the retrieval module 30 is further used to obtain the expected recall quantity when the sparsity level is non-sparse;
[0134] Determining the vector index recall quantity according to the expected recall quantity and sparsity;
[0135] Vector data is retrieved from the data set based on the vector index recall quantity and the filter condition.
[0136] Furthermore, the retrieval module 30 is further used to perform non-vector data retrieval in the data set based on the filtering condition to obtain a retrieval result;
[0137] Vector data is retrieved in the search results according to the vector index recall quantity.
[0138] Furthermore, the retrieval module 30 is further configured to filter non-vector data in the data set according to the filtering condition to obtain initial filtered data when the sparsity level is sparse;
[0139] Vector data is retrieved from the initial filtered data based on the filtering condition.
[0140] Furthermore, the extraction module 10 is also used to determine whether the filtering condition is a deterministic filtering;
[0141] When the filtering condition is a deterministic filtering, determining the retrieval data set according to the filtering condition;
[0142] Data is retrieved in the retrieval data set according to the filtering condition.
[0143] Furthermore, the extraction module 10 is also used to determine the data set corresponding to each target filtering condition in the filtering condition when the filtering condition is a deterministic filtering;
[0144] A search data set is constructed according to the data set.
[0145] Furthermore, the prediction module 20 is also used to determine the filtering condition type of the filtering condition;
[0146] Constructing an inverted index of the result set according to the filtering condition type;
[0147] The sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the inverted index and the filtering condition.
[0148] Furthermore, the prediction module 20 is further used to determine the retrieval data set according to the inverted index and the filtering condition;
[0149] Extracting a target retrieval sample set from the retrieval data set;
[0150] Performing vector data retrieval in the target retrieval sample set based on the filtering condition to obtain a retrieval result;
[0151] The sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the retrieval result and the retrieval data set.
[0152] Furthermore, the prediction module 20 is further configured to determine data that hits the same value when the filtering condition type is a numerical index, and use the value as a leaf node;
[0153] According to a preset construction strategy, the leaf nodes are combined to construct a parent node, and the parent node is used as a target node;
[0154] Build upward according to the target node and the preset building strategy to obtain an inverted index of the result set.
[0155] Furthermore, the prediction module 20 is also used to perform word segmentation on each data in the result set to obtain a word segmentation result when the filtering condition type is a text index;
[0156] An inverted index of the result set is constructed according to the word segmentation results.
[0157] Furthermore, the retrieval module 30 is also used to determine the number of threads corresponding to the sparsity level;
[0158] Concurrent threads are constructed according to the number of threads, and vector data retrieval is performed based on the constructed concurrent threads.
[0159] Other embodiments or specific implementations of the vector data retrieval device of the present invention can refer to the above-mentioned method embodiments and will not be described in detail here.
[0160] In addition, an embodiment of the present invention further proposes a storage medium, on which a vector data retrieval program is stored. When the vector data retrieval program is executed by a processor, the steps of the vector data retrieval method described above are implemented.
[0161] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.
[0162] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.
[0163] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as a read-only memory / random access memory, a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in each embodiment of the present invention.
[0164] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.
[0165] The present invention discloses A1, a vector data retrieval method, the vector data retrieval method comprising the following steps:
[0166] When receiving a vector data retrieval request, extracting a filtering condition in the vector data retrieval request;
[0167] When the filtering condition is uncertainty filtering, predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition;
[0168] Vector data is retrieved from the data set based on the sparsity level and the filtering condition.
[0169] A2. The vector data retrieval method as described in A1, wherein the step of performing vector data retrieval in the vector data set based on the sparsity degree and the filtering condition comprises:
[0170] When the sparsity level is non-sparse, obtaining an expected recall quantity;
[0171] Determining the vector index recall quantity according to the expected recall quantity and sparsity;
[0172] Vector data is retrieved from the data set based on the vector index recall quantity and the filter condition.
[0173] A3. The vector data retrieval method as described in A2, wherein the step of performing vector data retrieval in the data set based on the vector index recall quantity and the filtering condition comprises:
[0174] Based on the filtering condition, performing non-vector data search in the data set to obtain a search result;
[0175] Vector data is retrieved in the search results according to the vector index recall quantity.
[0176] A4. The vector data retrieval method as described in A1, wherein the step of performing vector data retrieval in the vector data set based on the sparsity degree and the filtering condition comprises:
[0177] When the sparsity degree is sparse, filtering non-vector data in the data set according to the filtering condition to obtain initial filtered data;
[0178] Vector data is retrieved from the initial filtered data based on the filtering condition.
[0179] A5. The vector data retrieval method according to any one of A1 to A4, further comprising: after the step of extracting the filtering condition in the vector data retrieval request when the vector data retrieval request is received:
[0180] Determining whether the filtering condition is a deterministic filtering;
[0181] When the filtering condition is a deterministic filtering, determining the retrieval data set according to the filtering condition;
[0182] Data is retrieved in the retrieval data set according to the filtering condition.
[0183] A6. The vector data retrieval method as described in A5, wherein when the filtering condition is a deterministic filtering, the step of determining the retrieval data set according to the filtering condition comprises:
[0184] When the filtering condition is a deterministic filtering, determining a data set corresponding to each target filtering condition in the filtering condition;
[0185] A search data set is constructed according to the data set.
[0186] A7. In the vector data retrieval method as described in any one of A1 to A4, the step of predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition comprises:
[0187] Determining a filter condition type of the filter condition;
[0188] Constructing an inverted index of the result set according to the filtering condition type;
[0189] The sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the inverted index and the filtering condition.
[0190] A8. The vector data retrieval method as described in A7, wherein the step of predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the inverted index and the filtering condition comprises:
[0191] Determine a retrieval data set according to the inverted index and the filtering condition;
[0192] Extracting a target retrieval sample set from the retrieval data set;
[0193] Performing vector data retrieval in the target retrieval sample set based on the filtering condition to obtain a retrieval result;
[0194] The sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the retrieval result and the target retrieval sample set.
[0195] A9. In the vector data retrieval method described in A7, the step of constructing an inverted index of the result set according to the filtering condition type comprises:
[0196] When the filtering condition type is a numerical index, determine the data that hits the same numerical value, and use the numerical value as a leaf node;
[0197] According to a preset construction strategy, the leaf nodes are combined to construct a parent node, and the parent node is used as a target node;
[0198] Build upward according to the target node and the preset building strategy to obtain an inverted index of the result set.
[0199] A10. In the vector data retrieval method described in A7, the step of constructing an inverted index of the result set according to the filtering condition type comprises:
[0200] When the filtering condition type is a text index, each data in the result set is segmented to obtain a segmentation result;
[0201] An inverted index of the result set is constructed according to the word segmentation results.
[0202] A11. In the vector data retrieval method as described in any one of A1 to A4, the step of performing vector data retrieval in the data set based on the sparsity degree and the filtering condition comprises:
[0203] Determining the number of threads corresponding to the sparsity level;
[0204] Concurrent threads are constructed according to the number of threads, and vector data retrieval is performed based on the constructed concurrent threads.
[0205] The present invention discloses B12, a vector data retrieval device, the vector data retrieval device comprising:
[0206] An extraction module, configured to extract a filtering condition in a vector data retrieval request when receiving the vector data retrieval request;
[0207] A prediction module, used for predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition when the filtering condition is uncertainty filtering;
[0208] A retrieval module is used to retrieve vector data in the data set based on the sparsity level and the filtering condition.
[0209] B13. The vector data retrieval device as described in B12, wherein the retrieval module is further used to obtain the expected recall quantity when the sparsity level is non-sparse;
[0210] Determining the vector index recall quantity according to the expected recall quantity and sparsity;
[0211] Vector data is retrieved from the data set based on the vector index recall quantity and the filter condition.
[0212] B14, the vector data retrieval device as described in B12, wherein the retrieval module is further used to filter non-vector data in the data set according to the filtering condition to obtain initial filtered data when the sparsity level is sparse;
[0213] Vector data is retrieved from the initial filtered data based on the filtering condition.
[0214] B15. The vector data retrieval device as described in B14, wherein the retrieval module is further used to obtain a filtering condition vector of the filtering condition;
[0215] Determining the similarity between each data vector in the initial filtering data and the filtering condition vector;
[0216] Vector data retrieval is performed based on the similarity.
[0217] B16. The vector data retrieval device as described in any one of B12-B15, wherein the extraction module is further used to determine whether the filtering condition is a deterministic filtering;
[0218] When the filtering condition is a deterministic filtering, determining the retrieval data set according to the filtering condition;
[0219] Data is retrieved in the retrieval data set according to the filtering condition.
[0220] B17, as described in B16, the vector data retrieval device, the extraction module is further used to determine the data set corresponding to each target filtering condition in the filtering condition when the filtering condition is a deterministic filtering;
[0221] A search data set is constructed according to the data set.
[0222] B18. The vector data retrieval device as described in any one of B12-B15, wherein the prediction module is further used to determine the filtering condition type of the filtering condition;
[0223] Constructing an inverted index of the result set according to the filtering condition type;
[0224] The sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the inverted index and the filtering condition.
[0225] The present invention discloses C19, a vector data retrieval device, which includes: a memory, a processor, and a vector data retrieval program stored in the memory and executable on the processor, wherein the vector data retrieval program is configured to implement the steps of the vector data retrieval method described in any one of A1 to A11.
[0226] The present invention discloses D20, a storage medium, on which a vector data retrieval program is stored. When the vector data retrieval program is executed by a processor, the steps of the vector data retrieval method as described in any one of A1 to A11 are implemented.
Claims
1. A vector data retrieval method, characterized in that: The vector data retrieval method comprises the following steps: When receiving a vector data retrieval request, extracting a filtering condition in the vector data retrieval request; When the filtering condition is uncertainty filtering, predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition; Vector data is retrieved from the data set based on the sparsity level and the filtering condition.
2. The vector data retrieval method according to claim 1, characterized in that: The step of performing vector data retrieval in the vector data set based on the sparsity degree and the filtering condition comprises: When the sparsity level is non-sparse, obtaining an expected recall quantity; Determining the vector index recall quantity according to the expected recall quantity and sparsity; Vector data is retrieved from the data set based on the vector index recall quantity and the filter condition.
3. The vector data retrieval method according to claim 2, characterized in that: The step of performing vector data retrieval in the data set based on the vector index recall quantity and the filtering condition comprises: Based on the filtering condition, performing non-vector data search in the data set to obtain a search result; Vector data is retrieved in the search results according to the vector index recall quantity.
4. The vector data retrieval method according to claim 1, characterized in that: The step of performing vector data retrieval in the vector data set based on the sparsity degree and the filtering condition comprises: When the sparsity degree is sparse, filtering non-vector data in the data set according to the filtering condition to obtain initial filtered data; Vector data is retrieved from the initial filtered data based on the filtering condition.
5. The vector data retrieval method according to any one of claims 1 to 4, characterized in that: After the step of extracting the filtering condition in the vector data retrieval request when the vector data retrieval request is received, the method further comprises: Determining whether the filtering condition is a deterministic filtering; When the filtering condition is a deterministic filtering, determining the retrieval data set according to the filtering condition; Data is retrieved in the retrieval data set according to the filtering condition.
6. The vector data retrieval method according to claim 5, characterized in that: When the filtering condition is a deterministic filtering, the step of determining the retrieval data set according to the filtering condition includes: When the filtering condition is a deterministic filtering, determining a data set corresponding to each target filtering condition in the filtering condition; A search data set is constructed according to the data set.
7. The vector data retrieval method according to any one of claims 1 to 4, characterized in that: The step of predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition comprises: Determining a filter condition type of the filter condition; Constructing an inverted index of the result set according to the filtering condition type; The sparsity of the result set corresponding to the vector data retrieval request in the data set is predicted according to the inverted index and the filtering condition.
8. A vector data retrieval device, characterized in that: The vector data retrieval device comprises: An extraction module, configured to extract a filtering condition in a vector data retrieval request when receiving the vector data retrieval request; A prediction module, used for predicting the sparsity of the result set corresponding to the vector data retrieval request in the data set according to the filtering condition when the filtering condition is uncertainty filtering; A retrieval module is used to retrieve vector data in the data set based on the sparsity level and the filtering condition.
9. A vector data retrieval device, characterized in that: The device comprises: a memory, a processor, and a vector data retrieval program stored in the memory and executable on the processor, wherein the vector data retrieval program is configured to implement the steps of the vector data retrieval method according to any one of claims 1 to 7.
10. A storage medium, characterized in that: The storage medium stores a vector data retrieval program, which, when executed by a processor, implements the steps of the vector data retrieval method according to any one of claims 1 to 7.