Vector database processing method and device, storage medium and program product

Abstract

The invention provides a vector database processing method and a related device, and relates to the technical field of vector databases. Wherein a first index parameter is determined according to a first vector data parameter of the vector database, a first vector index is created, and the first vector index and the first index parameter meet the target recall rate; performing retrieval by using the first vector index and the first index parameter; the vector data parameter of the vector database is changed from the first vector data parameter to a second vector data parameter, and determining that the first vector index and the first index parameter do not meet the target recall rate; if yes, a second index parameter is determined according to the second vector data parameter, a second vector index is created, and the second vector index and the second index parameter meet the target recall rate; and before the second vector index is created, performing retrieval by using the first vector index and the first index parameter or the second index parameter, and after the second vector index is created, performing retrieval by using the second vector index and the second index parameter.

Description

Technical Field

[0001] This application relates to the technical field of vector databases, and in particular, to a processing method, device, storage medium, and program product of a vector database. Background Art

[0002] With the rapid development of database technology, for data models such as large language models, various technologies related to database retrieval have been proposed in the industry, such as Retrieval-Augmented Generation (RAG) technology, vector databases, and Approximate Nearest Neighbor (ANN) technology, etc.

[0003] For data models, they face a well-known challenge at the beginning of their generation, that is, once trained, they cannot understand the latest or untrained knowledge in professional fields. The RAG technology can solve this problem. One of the cores of the RAG technology is the vector database. The vector database is specifically used to store and retrieve vector data. For some unstructured real-time information and sensitive private domain data, when the vector database is generated, these real-time information and private domain data can be converted into vector data and stored in the vector database. In addition, after the vector database is generated, before the data model executes the inference task, when needed, these information can be supplemented in the vector database according to the user's input (such as inputting some real-time information and private domain data), so as to greatly expand the knowledge scope of the data model and improve the question-and-answer effect of the inference task.

[0004] The core of the vector database is to use a special vector index to effectively organize and retrieve vector data. The vector database can provide queries of vector retrieval algorithms such as ANN while processing high-dimensional data. Through the query of the vector retrieval algorithm, data items similar to the user's query vector can be quickly found from a large amount of vector data, so that the RAG system can quickly retrieve the information most relevant to the user's query vector.

[0005] The current relatively mainstream ANN technologies mainly include two vector retrieval algorithms: Hierarchical Navigable Small World (HNSW) and Inverted File with Flat Quantization (IVFFlat). These retrieval algorithms can well optimize the retrieval efficiency, retrieval performance, and retrieval effect of vectors.

[0006] For a data model, recall is a very important metric. A decrease in recall indicates a decline in the data model's ability to identify positive class samples. In some application scenarios (such as medical diagnosis, fraud detection, etc.), missing a positive class sample may lead to very serious consequences.

[0007] In fact, in the context of complex requirements, the growth of data can easily exceed expectations. Even for the same scenario, as the data volume increases and / or decreases, the data distribution will change, resulting in a decrease in the recall rate of vector retrieval. Current technologies do not focus on how to handle the possible decrease in recall rate when the data volume in the vector database varies during actual production use. However, the importance of recall cannot be ignored. Accordingly, how to improve recall is of crucial engineering significance and is an urgent problem to be solved currently. Summary of the Invention

[0008] To solve the above problems, the present application provides a method, apparatus, storage medium, and program product for processing a vector database to improve the recall rate of the vector database.

[0009] In a first aspect, the present application provides a method for processing a vector database. Exemplarily, this method can be applied to a first device, and the first device can be an electronic device with relevant processing capabilities.

[0010] Optionally, the method for processing the vector database includes the following processes:

[0011] First, determine a first index parameter according to the first vector data parameter of the vector database, and create a first vector index according to the first index parameter. The first recall rate corresponding to the first vector index and the first index parameter at a first time meets the target recall rate;

[0012] After that, the first vector index and the first index parameter can be used to retrieve the vector database;

[0013] In the case where the vector database changes, etc., the second recall rate corresponding to the first vector index and the first index parameter at a second time does not meet the target recall rate, where the vector data parameter of the vector database changes from the first vector data parameter to a second vector data parameter;

[0014] Determine a second index parameter according to the second vector data parameter, and create a second vector index according to the second index parameter. The third recall rate corresponding to the second vector index and the second index parameter at a third time meets the target recall rate;

[0015] Before the creation of the second vector index is completed, continue to use the first vector index and the first index parameter, or use the first vector index and the second index parameter to retrieve the vector database, and after the creation of the second vector index is completed, use the second vector index and the second index parameter to retrieve the vector database.

[0016] Through the above implementation manner in the embodiments of the present application, when the vector data in the vector database undergoes additions, deletions, etc., resulting in the first vector index and the first index parameter no longer meeting the target recall rate, the second index parameter is re-determined and the second vector index is created, so that when the recall rate decreases, the recall rate can be timely increased. Moreover, in the process of re-creating the vector index in the above implementation manner, continue to use the first vector index and the first index parameter for retrieval, or use the first vector index and the second index parameter for retrieval, and after the re-creation is completed, use the second vector index and the second index parameter for retrieval, so that in the case of improving the recall rate, the influence of re-creating the vector index on the retrieval function of the vector database can be minimized as much as possible, improving the user experience.

[0017] Optionally, the method may further include:

[0018] Determine a parameter effect data model based on pre-training, where the parameter effect data model includes the correspondence between index parameters and vector data parameters;

[0019] The above-mentioned determination of the first index parameter according to the first vector data parameter of the vector database includes:

[0020] According to the first vector data parameter, determine a first parameter recommended value including index parameters through the parameter effect data model, and determine the first index parameter according to the first parameter recommended value;

[0021] The above-mentioned determination of the second index parameter according to the second vector data parameter includes:

[0022] According to the second vector data parameter, determine a second parameter recommended value including index parameters through the parameter effect data model, and determine the second index parameter according to the second parameter recommended value, where the first parameter recommended value is different from the second parameter recommended value.

[0023] In the above implementation manner, the first and second index parameters are obtained through the parameter effect data model, that is, the index parameters can be recommended through the parameter effect mathematical model and reasoning, so that appropriate index parameters can be configured more efficiently for the vector database.

[0024] Optionally, the data set corresponding to the parameter effect data model includes multiple sample sets;

[0025] The information in the parameter effect data model includes: the vector index corresponding to each sample set and the index parameter, and the recall rate corresponding to the vector index and index parameter corresponding to each sample set is greater than the target recall rate.

[0026] From the above description, it can be seen that the index parameters in the parameter effect data model all meet the target recall rate, so that the embodiments of the present application can quickly obtain the first index parameter and the second index parameter that meet the target recall rate according to the parameter effect data model.

[0027] Optionally, the determination that the second recall rate corresponding to the first vector index and the first index parameter at the second time does not meet the target recall rate includes:

[0028] Determine the first vector data response set actually corresponding to the first vector data request;

[0029] According to the first vector index and the first index parameter, retrieve the second vector data response set corresponding to the first vector data request from the vector database;

[0030] Determine the second recall rate according to the first vector data response set and the second vector data response set;

[0031] Determine that the second recall rate does not meet the target recall rate.

[0032] Through the above implementation manner, the embodiments of the present application can timely determine that the second recall rate does not meet the target recall rate.

[0033] Optionally, the determination of the first vector data response set actually corresponding to the first vector data request includes:

[0034] Perform brute-force search on the vector database according to the first vector data request to obtain the first vector data response set; or

[0035] Perform brute-force search on the vector data in each partition after partitioning the vector database according to the first vector data request to obtain the first vector data response set.

[0036] For obtaining the first vector database response set, the embodiments of the present application provide two implementation schemes. The first is to perform brute-force search on the vector database, which is more suitable for vector databases with not particularly large data volumes. The second is to perform brute-force search after partitioning the vector database, which is more efficient for massive vector databases to complete brute-force search, and thus can quickly determine that the recall rate does not meet the target recall rate.

[0037] Optionally, the determination that the second recall rate corresponding to the first vector index and the first index parameter at the second time does not meet the target recall rate includes one of the following:

[0038] Determine that at least one of the recall rates corresponding to the first vector index and the first index parameter within a set period does not meet the target recall rate;

[0039] Determine that the average value of multiple recall rates corresponding to the first vector index and the first index parameter within a set period does not meet the target recall rate;

[0040] Determine that the number of recall rates that do not meet the target recall rate among multiple recall rates corresponding to the first vector index and the first index parameter within a set period is greater than or equal to a set threshold.

[0041] The embodiments of the present application provide various implementation manners for determining that the second recall rate does not meet the target recall rate, making the determination manner of the reduced recall rate diversified. In addition, when it is determined that the recall rate is reduced, an abnormal alarm mechanism can be triggered in a timely manner. In the case where an operator needs to set, the operator can initiate an adjustment operation of the recall rate as soon as possible.

[0042] Optionally, the above retrieval of the vector database using the first vector index and the first index parameter includes:

[0043] Retrieve the vector database using the first vector index, the first index parameter, and a first user parameter, where the first user parameter includes a first retrieval parameter for determining a retrieval range, and both the first recall rate and the second recall rate correspond to the first vector index, the first index parameter, and the first user parameter;

[0044] Before determining the second index parameter and creating the second vector index, the method further includes:

[0045] Adjust the first user parameter to a second user parameter, where the second retrieval parameter for determining the retrieval range in the second user parameter is greater than the first retrieval parameter in the first user parameter;

[0046] After determining that the fourth recall rate corresponding to the first vector index, the first index parameter, and the second user parameter at a fourth time meets the target recall rate, retrieve the vector database using the first vector index, the first index parameter, and the second user parameter;

[0047] After determining that the fourth recall rate does not meet the target recall rate, directly perform the process of determining the second index parameter and creating the second vector index; or, perform the following process:

[0048] Adjust the second user parameter to a third user parameter, where the third retrieval parameter for determining the retrieval range in the third user parameter is greater than the second retrieval parameter;

[0049] After determining that the fifth recall rate corresponding to the first vector index, the first index parameter, and the third user parameter meets the target recall rate, retrieve the vector database using the first vector index, the first index parameter, and the third user parameter;

[0050] After determining that the fifth recall rate does not meet the target recall rate, perform the process of determining the second index parameter and creating the second vector index.

[0051] The above processing method provides a way to adjust the recall rate. However, in some cases, the recall rate cannot be improved. For example, if the vector data for some knowledge points in the vector database is deleted, even if the search range is expanded to the entire database, the recall rate cannot be improved. Therefore, a scheme of adjusting user parameters can be considered when a large amount of vector data is added to the vector database.

[0052] In addition, after the above adjustment of the third user parameter still does not meet the recall rate, in principle, the user parameter can be continuously adjusted. However, since adjusting the user parameter expands the search range, the search efficiency may be reduced, and in some scenarios, the search efficiency may be reduced significantly, such as exceeding the set threshold. Therefore, the implementation method of adjusting the user parameter can be used as a means. When this implementation method cannot improve the recall rate and other situations, other means need to be used to improve the recall rate. For example, it can be considered to change to perform the process of determining the second index parameter and creating the second vector index.

[0053] Optionally, before determining the second index parameter and creating the second vector index, continue to retrieve the vector database using the first vector index and the first index parameter, including:

[0054] Retrieve the vector database using the first vector index, the first index parameter, and the second user parameter or the third user parameter;

[0055] After determining the second index parameter and creating the second vector index, retrieve the vector database using the second vector index and the second index parameter, including:

[0056] Query the vector database using the second vector index, the second index parameter, and the first user parameter.

[0057] The above implementation method can also temporarily intervene in the recall rate by temporarily modifying the user parameter to quickly adjust the recall rate.

[0058] In the case where a second user parameter is provided or a third user parameter is further provided, when retrieving using the first vector index and the first index parameter, one of these two parameters can be used for retrieval. After obtaining the second index parameter and the second vector index, the second index parameter, the second vector index, and the first user parameter can be used for retrieval. In this way, on the one hand, the recall rate can be improved as much as possible before creating the second vector index, and on the other hand, after creating the second vector index, the original user parameters are still used for retrieval, thus avoiding the impact on retrieval efficiency and the like caused by increasing user parameters.

[0059] Optionally, the embodiments of the present application also provide several configuration methods for user parameters.

[0060] For example, the algorithm for retrieving the vector database is HNSW, and the parameter for the retrieval range in the user parameters corresponding to this algorithm includes ef search ; or

[0061] The algorithm for retrieving the vector database is IVFFlat, and the parameters for the retrieval range in the user parameters corresponding to this algorithm include the parameter nprobe for indicating the number of clustering centers to be found at one time and the parameter max_codes for indicating the number of vectors to be searched at one time;

[0062] The second retrieval parameter for determining the retrieval range in the second user parameter is greater than the first retrieval parameter in the first user parameter, including:

[0063] The nprobe in the second user parameter is greater than the nprobe in the first user parameter, and / or, the max_codes in the second user parameter is greater than the max_codes in the first user parameter;

[0064] The third retrieval parameter for determining the retrieval range in the third user parameter is greater than the second retrieval parameter, including:

[0065] The nprobe in the third user parameter is greater than the nprobe in the second user parameter, and / or, the max_codes in the third user parameter is greater than the max_codes in the second user parameter.

[0066] Optionally, the embodiments of the present application also provide possible configuration methods for index parameters.

[0067] For example, the index parameters of the above vector database include M and ef construction , where M indicates the maximum number of neighbors of a node, and the ef construction indicates the search range during vector index construction;

[0068] The M in the first index parameter is less than the M in the second index parameter; and / or

[0069] The ef in the first index parameter construction is less than the ef in the second index parameter construction .

[0070] Optionally, the first vector index and the first index parameter are identified by a first version number; the first vector index and the second index parameter are identified by a second version number; the second vector index and the second index parameter are identified by a third version number.

[0071] The above identification of the vector index and the corresponding index parameter enables the user to select the vector index and index parameter for retrieval according to the version number, providing more choices for the user. For the scenario of adding a large number of vector data to the vector database, selection can also be based on the version number, so that the retrieval of the newly added vector data can be performed in the way of using the old vector index plus the new index parameter, and the optimization of the retrieval effect for the newly added vector data can be achieved as early as possible.

[0072] In a second aspect, an embodiment of the present application further provides an apparatus for processing a vector database. The apparatus includes a memory and a processor. The memory is used to store the vector database, and the processor is used to execute the above various methods. Optionally, the apparatus may be an electronic device.

[0073] In a third aspect, an embodiment of the present application further provides a computer-readable storage medium. Instructions are stored on the readable storage medium, and when the instructions are executed on an electronic device, the electronic device is caused to execute the above various methods.

[0074] In a fourth aspect, an embodiment of the present application further provides a computer program product, including: instructions, when the instructions run on an electronic device, causing the electronic device to execute the above various methods. Description of the Drawings

[0075] Figure 1 is a schematic flowchart of a process for retrieving a vector database provided by an embodiment of the present application;

[0076] Figure 2 is a schematic flowchart of a process for processing data written into a vector database provided by an embodiment of the present application;

[0077] Figure 3 is a schematic diagram of an implementation process for adjusting the recall rate provided by an embodiment of the present application;

[0078] Figure 4 is a schematic diagram of a parameter effect data model generation and use provided by an embodiment of the present application. Detailed Embodiments

[0079] For the sake of clear and concise description of the following embodiments, the vocabulary involved in the embodiments of the present application will be described first. It should be understood that this description is for a clearer understanding of the embodiments of the present application and does not necessarily constitute a limitation on the embodiments of the present application.

[0080] To enable those skilled in the art to understand the solution of the present application more clearly, the application scenario of the technical solution of the present application will be described first.

[0081] Data models are currently widely used in various fields. A very common model at present is the large language model called ChatGPT. The core of the large language model is to train the model using a large-scale data set so that it can generate natural language text or understand the meaning of language text. Compared with traditional natural language processing models, large language models can better understand and generate natural text, and at the same time show certain logical thinking and reasoning abilities.

[0082] Currently, there are also corresponding data models for some professional fields. The vector data stored in the vector database of such data models corresponds to the private domain data of the relevant professional fields. For example, in the medical field, for the data model used to provide expert system functions, the vector data stored in its vector database corresponds to various cases and their medical means. For various data models, a large amount of data growth and deletion will be involved. For example, after generating the vector database, new data will be continuously added, and old data may also be continuously eliminated. Therefore, after the vector database runs for a period of time, its distribution characteristics will change. After building a mathematical model such as a large language model, even if the problem of data growth has been fully considered when building the mathematical model, however, in a complex demand background, the growth of data is still likely to exceed expectations, and it is very easy to have the problem of a decrease in the recall rate. For the case where data is eliminated, deleting the data from the vector database will cause holes in the vector index, resulting in a decrease in retrieval efficiency and a reduction in the recall rate. Correspondingly, when the recall rate decreases, it is very important to take timely measures to improve the recall rate.

[0083] For the case where the vector database may involve a large amount of data deletion, the usual implementation solution is to adjust it through the A / B index method. Or rather, for this case, an easy-to-think-of solution is that the vector database or the operation and maintenance personnel choose to trigger an operation for data processing according to a certain condition, and the vector database can create a new vector index in the background. This processing method can solve the index hole problem, but it is very likely that it cannot solve the problem of the reduction in the recall rate. If it is by deleting the current index and adding the index again, this method will also cause the database to be inaccessible during the operation.

[0084] For the retrieval of a mathematical model, a possible implementation is to set multiple vector indexes based on multiple shards of vector data. Specifically, the vector data is split into multiple shards through hashing or range partitioning, and vector indexes are set respectively for these shards to obtain multiple vector indexes, thereby achieving concurrent construction and concurrent query of indexes. This implementation solves the problem of long time consumption for index construction under a large amount of data, but does not pay attention to the problem of reduced actual recall rate.

[0085] Regarding how to solve the problem of reduced recall rate, a possible implementation is to periodically rebuild the vector index. However, for this implementation, if the amount of vector data in the vector database is extremely large, the resources and time required for rebuilding the index will be relatively large. Moreover, the recall rate cannot be optimized during the period of rebuilding the index. On the other hand, even if the index is rebuilt to adjust the recall rate, it may not be possible to adjust the recall rate to the expected target.

[0086] Therefore, to solve the problem of decreased recall rate, the embodiments of the present application provide a processing method and related device for a vector database. In this method, after determining the first index parameter according to the first vector data parameter of the vector database and creating the first vector index, the first vector index and the first index parameter are used for retrieval, where the first recall rate corresponding to the created first vector index and the first index parameter at the first time meets the target recall rate, and this first recall rate is the current recall rate at the first time; when the vector database changes (the vector data parameter corresponding to the vector database will change, for example, from the first vector data parameter to the second vector data parameter), the recall rate may decrease. At this time, after determining that the second recall rate corresponding to the first vector index and the first index parameter at the second time does not meet the target recall rate, where this second recall rate is the current recall rate at the second time, the second index parameter is determined according to the second vector data parameter, and the second vector index is created, that is, the third recall rate of the second vector index and the corresponding second index parameter at the third time meets the target recall rate; before the second vector index is created, the first vector index and the first index parameter are continued to be used to retrieve the vector database, or the first vector index and the second index parameter can also be used for retrieval, and after the second vector index and the second index parameter are created, the second vector index and the second index parameter are used to retrieve the vector database.

[0087] Next, take Figure 1 as an example to describe in detail the technical solution provided by the embodiments of the present application. As Figure 1 shown, the embodiments of the present application include the following steps.

[0088] Step 101: Determine a first index parameter according to the first vector data parameter of the vector database, and create a first vector index.

[0089] For this first vector index and the first index parameter, the first vector index and the first index parameter should satisfy that the first recall rate corresponding to them at the first time meets the target recall rate, that is, the current recall rate needs to meet the target recall rate.

[0090] The first vector data parameter corresponding to the vector database can be used to indicate information such as the number of vector data stored in the vector database. For example, it can be the vector dimension, vector size, etc. When determining the first index parameter, this first vector data parameter needs to be considered. To make the recall rate meet the target recall rate, the recall rate can also be considered when determining the first index parameter.

[0091] Step 102: Retrieve the vector database using this first vector index and the first index parameter.

[0092] Step 103: Determine that the second recall rate corresponding to the first vector index and the first index parameter at the second time does not meet the target recall rate, where the first vector data parameter of the vector database becomes the second vector data parameter.

[0093] There are often situations such as addition / deletion of vector data in the vector database. When the vector database changes, the corresponding vector data parameter will also change. For example, it changes from the first vector data parameter to the second vector data parameter.

[0094] When there are situations such as addition / deletion of vector data in the vector database, it may lead to the situation that the second recall rate corresponding to the previously created first vector index and the first index parameter at the second time does not meet the target recall rate. That is, it is necessary to determine whether the current recall rate still meets the target recall rate.

[0095] If the recall rate is met, the first vector index and the first index parameter can continue to be used. If the recall rate is not met, perform the following Step 104.

[0096] Step 104: Determine a second index parameter according to this second vector data parameter, and create a second vector index.

[0097] Similar to the first vector index and the first index parameter, the third recall rate corresponding to the second vector index and the second index parameter at the third time, that is, the current recall rate, also needs to meet the target recall rate.

[0098] Step 105: Before the second vector index is created, continue to retrieve the vector database using the first vector index and the first index parameter, or retrieve using the first vector index and the second index parameter. After the second vector index and the second index parameter are created, retrieve the vector database using the second vector index and the second index parameter.

[0099] Since it takes some time to determine the second index parameter and create the second vector index, to provide the retrieval function of the vector database as much as possible, before obtaining the second vector index and the second index parameter, continue to retrieve the vector database using the first vector index and the first index parameter. Since creating a vector index takes a long time and the corresponding second index parameter is determined before creating the vector index, therefore, an optional implementation is that when the second index parameter is determined, the first vector index and the second index parameter can be used to retrieve the vector database. After obtaining the second vector index and the second index parameter, switch to using the second vector index and the second index parameter to retrieve the vector database, so as to improve the recall rate as soon as possible.

[0100] When creating the second vector index, to enable the user to continue retrieving using the first vector index, corresponding identifiers can be set for different vector indexes, such as setting a version number. Identify the first vector index as version V1 and the second vector index as version V2. And these version identifiers can be provided to the user so that when the user needs to continue retrieving, the first vector index can be selected for retrieval according to the identifier of version V1. Optionally, when the first vector index can also be used together with the second index parameter for retrieval, the version number can also be set as follows: identify the first vector index and the first index parameter as V1, identify the first vector index and the second index parameter as V1', and identify the second vector index and the second index parameter as V.

[0101] Through the above implementation, in the embodiment of the present application, when the vector data in the vector database undergoes operations such as addition and deletion, resulting in the first vector index and the first index parameter no longer meeting the target recall rate, the second index parameter is re-determined and the second vector index is created, so that when the recall rate decreases, the recall rate can be timely increased. Moreover, in the process of re-creating the vector index, the above implementation continues to retrieve using the first vector index and the first index parameter, or retrieves using the first vector index and the second index parameter. After re-creation is completed, the second vector index and the second index parameter are used for retrieval, so that while improving the recall rate, this solution can minimize the impact of re-creating the vector index on the retrieval function of the vector database and improve the user experience.

[0102] The implementation process of the embodiments of this application can also be achieved through Figure 3 manifested. As Figure 3 shown, for the knowledge base, first generate a parameter effect data model. Then, determine the current recall rate through sampling and other methods, and issue a warning and perform effect tuning when the recall rate does not meet the target recall rate. The specific implementation process of effect tuning can adopt the implementation method as Figure 1 shown. During the implementation process, the vector data can be sharded and the ANN vector index can be used.

[0103] In the embodiments of this application, two index parameters are respectively determined in the above steps 101 and 104. There are various implementation methods for generating index parameters. One possible implementation method is to first obtain a parameter effect data model based on pre-training, and then obtain the corresponding index parameters based on this parameter effect data model.

[0104] Specifically, this parameter effect data model reflects the recall effects under different index parameters and data magnitudes. Therefore, a relatively reasonable index parameter can be quickly obtained based on this parameter effect mathematical model. For example, this parameter effect mathematical model provides the corresponding relationships between various index parameters, data magnitudes, and recall rates. For different data magnitudes of the vector database (specifically indicated by the above first vector data parameter and second vector data parameter), the index parameter corresponding to this data magnitude can be found. Even if the above first vector data parameter and / or second vector data parameter are not included in this parameter effect mathematical model, this parameter effect mathematical model can also deduce a relatively reasonable vector index and index parameter.

[0105] Optionally, when creating a vector index and index parameters, especially when there are no vector index and index parameters directly corresponding to the vector data parameters in this parameter effect data model, this parameter effect data model can provide multiple optional values to select an optimal value from these optional values.

[0106] The following describes the generation method of this parameter effect data model.

[0107] The parameter effect data model can be determined based on multiple sample sets. For each sample set, there is an index parameter set during initialization. The index parameters used by each sample set can be the same or different. Each sample set can be processed to obtain a vector index corresponding to the index parameter. If the vector index and the index parameter meet the requirements of the target recall rate, it is determined that the vector index and the corresponding index parameter correspond to the sample set. If the recall rate requirements are not met, the index parameter is adjusted until the target recall rate is satisfied. According to the index parameters obtained by processing each sample set, after each sample set determines the index parameter and the corresponding vector index, it always uses the index parameter and the vector index. Thus, a data model with approximate parameter effects can be obtained based on these sample sets.

[0108] Figure 4 An implementation of the parameter effect data model is shown. For example, a feasible implementation is that, in the pre-training stage, for each sample set, according to M indicating the maximum number of neighbors of a node, ef indicating the search range when constructing the vector index construction the vector dimension, the data volume, the index size, etc., pre-training is performed to output the corresponding M, ef construction and the number of queries per second (QPS), so as to obtain the parameter effect data model. After that, according to the expected recall rate, the data volume in the vector database, etc., through the parameter effect data model, or according to the corresponding inference mathematical model, the recommended index parameters can be obtained, such as including M and ef construction .

[0109] Specifically, taking the HNSW index as an example, the following processing can be performed for each known sample set.

[0110] First, multiple sample sets can be determined. For each sample set, there is an index parameter set during initialization, such as including M and ef construction . M indicates the maximum number of neighbors of a node, and ef construction indicates the search range when constructing the vector index. And the M and ef construction used by each sample set construction can be the same or different. For example, sample set 1 uses M1 and ef construction 1, and sample set 2 uses M2 and ef construction 2. M1 can be the same as or different from M2. Similarly, ef construction 1 can be the same as or different from ef construction, make adjustments until the target recall rate is met. Thus, the M and ef that meet the recall rate for each sample set can be obtained. construction And the corresponding relationship between the recall rate. Then, based on these corresponding relationships, the parameter effect data model is generated.

[0111] The following Tables 1-1 to 1-3 show the recall rate indicators corresponding to different parameters. Among them, 768D1M, 1536D5M, and 768D10M represent different vector dimensions and data magnitudes.

[0112] 768D1M M＝16 M＝32 M＝64 M＝128 efC = 128 0.965 0.9804 0.9843 0.988 efC = 256 0.9855 0.9941 0.9966 0.9976 efC = 512 0.9967 0.9997 0.9994 0.9998

[0113] Table 1-1

[0114] 1536D5M M＝16 M＝32 M＝64 M＝128 efC = 128 0.9823 0.9888 0.9922 0.9914 efC = 256 0.9924 0.9973 0.9983 0.9981 efC = 512 0.998 0.9993 0.9998 0.9999

[0115] Table 1-2

[0116] 768D10M M＝16 M＝32 M＝64 M＝128 efC = 128 0.9244 0.9671 0.9757 0.9809 efC = 256 0.9675 0.988 0.9929 0.9944 efC = 512 0.9861 0.9963 0.9972 0.9984

[0117] Table 1-3

[0118] Taking the creation of a vector index as an example for the above sample set, the embodiments of the present application also calculate the size of the vector index and the duration of building the index, etc. Correspondingly, the parameter effect data model can further include information such as the size of the vector index and the duration of building the index. The following Tables 2-1 to 2-3 show the duration of building the index corresponding to different parameters.

[0119] 768D1M M＝16 M＝32 M＝64 M＝128 efC = 128 155.9 332.9 377.1 415.7 efC = 256 432.4 583.9 692.1 749.7 efC = 512 738.0 1051.4 1332.8 1476.0

[0120] Table 2-1

[0121] 1536D5M M＝16 M＝32 M＝64 M＝128 efC = 128 988.3 1216.1 1373.2 1429.1 efC = 256 1460.1 2170.6 2625.7 2827.3 efC = 512 2467.2 3861.9 5045.9 5820.5

[0122] Table 2-2

[0123] 768D10M M＝16 M＝32 M＝64 M＝128 efC = 128 686.7 1985.5 1564.7 1833.3 efC = 256 1843.8 2548.8 3009.6 3647.0 efC = 512 3191.6 4968.4 6335.3 6525.9

[0124] Table 2-3

[0125] The data model of approximate parameter effects obtained through the above method can reflect the recall rate under different index parameters, different data magnitudes, and different vector dimensions. Based on this mathematical model, relatively reasonable index parameters can be quickly obtained.

[0126] Due to the limitations of the sample set, it is impossible for the test data to cover all possible scenarios. Therefore, the inference ability of the "inference model" can be utilized to fit the limited test results into a larger sample space. For example, there is no test data with a dimension of 2560 in the test samples, but there is a large amount of test data with dimensions of 256, 512, 768, 960, and 1024. Here, the "inference model" can be used to infer a relatively reasonable parameter based on the known conclusions. In such scenarios, the inference model can also be used to provide multiple optional values so as to select an optimal value from these optional values. For example, when the index parameters include M and ef construction in the case of, then it is possible to make a selection based on M and / or ef construction so that the values of M and ef construction are as small as possible.

[0127] As mentioned above, for the data model, it is necessary to determine whether the current retrieval of the vector database is accurate, that is, to judge whether the target recall rate is satisfied.

[0128] A possible implementation is not to directly calculate the recall rate, but to judge whether the recall rate is satisfied based on the user's feedback. For example, if the number of times of receiving feedback that the retrieval result is inaccurate from the user within a set time exceeds the set threshold, it is considered that the current recall rate does not meet the requirement. The accuracy of this implementation is not very high.

[0129] In an embodiment of the present application, a possible implementation is to determine the recall rate to determine the accuracy of the vector database retrieval.

[0130] When determining the recall rate, a possible implementation is to retrieve all vector data in the direct vector database. For example, retrieve by means of brute-force search to determine the recall rate. Since there are often a large amount of vector data stored in the vector database of the data model, calculating the recall rate for all vector data will be relatively inefficient.

[0131] The embodiment of the present application provides a method for determining the recall rate. Specifically, first, the vector database is sampled by a sampling system. For example, first determine the first vector data response set actually corresponding to the first vector data request through sampling; then, each time the recall rate needs to be calculated, according to the vector index and index parameters, retrieve the second vector data response set corresponding to the first vector data request in the vector database; then determine the recall rate according to the first vector data response set and the second vector data response set.

[0132] The response information in the above-mentioned first vector data response set is considered to be true, accurate, and comprehensive response information. Therefore, dividing the second vector data response set by the first vector data response set can obtain the recall rate corresponding to the first vector data request.

[0133] For determining the first vector data response set actually corresponding to the first vector data request, a possible implementation is to retrieve it from an open-source data set. Some standard data is stored in this open-source data set, and this data remains unchanged. Therefore, by retrieving from this open-source data set, the first vector data response set can be obtained. However, since this open-source data set is unchanged, it often cannot meet the requirements of different types of vector databases, especially for the private vector databases where data is frequently added and deleted.

[0134] Another possible implementation is to obtain the first vector data response set by performing brute-force search on the vector database. The results retrieved in this way are comprehensive, but the efficiency is relatively low. Especially for vector databases with a large amount of data, the low retrieval efficiency is a big problem.

[0135] In an implementation manner of this application, it is also possible to partition the vector database and perform brute-force search on these partitions, for example, perform brute-force search using a Flat vector index of the brute-force search type. The above-mentioned Flat vector index of the brute-force search type is an index created only for part of the data according to the partitioning method. For example, the vector database is divided into multiple partitions, and for each partition, an index is created for part of the data therein. Among them, for the partitioning method, for example, the vector database can be partitioned by a certain hash algorithm. For this implementation manner, compared with full-database search, the search efficiency is higher, and compared with searching using an open-source data set, the accuracy is higher.

[0136] After obtaining the recall rate through the above various implementation manners, it is necessary to determine whether the recall rate meets the target recall rate. Specifically, it can be to calculate the recall rate at least once within a set period and make a judgment based on the calculated recall rate. The period for calculating the recall rate, for example, can be to monitor the recall rate every day.

[0137] After calculating the recall rate, one implementation manner is to determine whether there is a situation where the calculated recall rate does not meet the target recall rate. If so, it is considered that the recall rate has decreased and the recall rate needs to be adjusted. For the case of calculating the recall rate multiple times, it can be to determine whether the average value of the multiple recall rates meets the target recall rate, or it can be to determine how many times the recall rate does not meet the target recall rate among the multiple recall rates. If the number of times not meeting the target recall rate exceeds the set threshold, it is considered that the target recall rate is not met.

[0138] The above-mentioned period for calculating the recall rate, for example, can be to monitor the recall rate every day.

[0139] Take the example of judging whether the target recall rate is satisfied according to the average value of the recall rate. Let the target recall rate be α, and monitor multiple actual recall rates P within the time window T recall , based on these actual recall rates P recal Calculate the average recall rate And In the case where the average recall rate Is less than or equal to α, that is Determine that the recall rate has decreased.

[0140] In the case where it is determined that the recall rate has decreased, an alarm can also be triggered, so that the operator can be prompted to increase the recall rate of the data model.

[0141] The above embodiments describe various implementation methods for improving the recall rate. In the above embodiments, it is mentioned that the index parameters may include M and ef construction . When adjusting the index parameters, specifically, M in the first index parameter can be made larger, that is, M in the second index parameter is greater than M in the first index parameter. It can also be that ef in the first index parameter construction Is made larger, that is, ef in the second index parameter construction Is greater than ef in the first index parameter construction . Of course, it can also be that both M and ef construction Are made larger.

[0142] Next, a detailed analysis is carried out on the parameters M and ef construction .

[0143] The parameter M is used to control the maximum number of neighbors of each node. The parameter ef construction Is used to control the search range of candidate neighbors during the index construction phase. In the embodiments of the present application, it is first demonstrated whether the recall rate can be improved by adjusting the parameters M and ef construction .

[0144] Take the HNSW algorithm as an example. For the HNSW graph including multiple nodes, the navigation path of the HNSW graph depends on the connectivity of the graph. The length of the navigation path and the recall rate can be associated through the following formula (1).

[0145] P recall ∝1 - e -α.L Formula (1)

[0146] Among them, P recall Is the recall rate, L is the length of the navigation path, and α is the graph connection density parameter.

[0147] Among the above parameters, α is positively correlated with M. Therefore, if M is increased, the connection density of the graph will be increased, that is, α is increased, and increasing α can increase the path coverage area, thereby improving the recall rate.

[0148] In the HNSW graph, the coverage of neighbor nodes for each node is affected by M and the data distribution dimension d. The coverage of neighbor nodes can be approximately represented by the following formula:

[0149]

[0150] where C(M, d) is the average coverage of a node, and d is the dimension of the vector.

[0151] It can be seen from this that increasing M can increase C(M, d), so that each node can be connected to a wider range of data nodes, thereby reducing the number of path jumps and increasing the recall rate.

[0152] Therefore, through the analysis of the above formulas (1) and (2), it can be known that increasing M can increase the path coverage area and reduce the number of path jumps, thereby increasing the recall rate.

[0153] In addition, in the above formula (1), L is determined by ef constructiob That is, ef construction determines the search range of graph construction, thus affecting the overall quality of the graph.

[0154] The construction time T of the graph build and the relationship between ef construction and M are shown in the following formula (3).

[0155]

[0156] where N is the number of data points.

[0157] Although there will be a problem of increasing the construction time (as can be seen from the above formula (3)), increasing ef construction can enhance the diversity of the graph, thereby increasing the recall rate.

[0158] From this, through the above analysis, it can be known that by increasing one or all of the parameters M and ef construction in it, the recall rate can be increased.

[0159] Specifically, increasing M can increase the connection density of the graph, provide more choices for the navigation path, optimize the path quality, and increase the probability of finding the target node in the query stage. However, increasing M will lead to an increase in the memory occupation of construction and query.

[0160] While increasing ef construction , can make the exploration range wider in the construction stage, select better neighbor nodes, the constructed graph has higher quality and better navigability. However, increasing ef constructionIt will also lead to more build time, especially when the amount of data is large. Therefore, M can be adjusted first, or which parameter to adjust first can be determined according to the specific scenario. For example, for some scenarios that are not sensitive to the memory occupation of queries, increasing M can be considered first. For some scenarios, the increase in build time does not have a great impact, so increasing ef can be considered first. construction 。

[0161] From the above demonstration, it can be seen that the recall rate can be improved by adjusting at least one of the parameters M and ef construction .

[0162] Taking the adjustment of M as an example below, the implementation method of improving the recall rate by adjusting the parameters M and ef construction will be described in detail. ef construction As an index parameter, it has a similar role in creating a vector index, so no separate example will be given for illustration.

[0163] After determining the first index parameter including M1 and ef construction 1 according to the parameter effect data model and creating the first vector index, when creating the second vector index according to the parameter effect data model, M1 can be adjusted to a larger value M2 first according to the parameter effect data model, and then the second vector index can be created based on the adjusted M2. Thus, the created second vector index is more likely to meet the target recall rate.

[0164] If M adjustment is not considered and the second vector index is created just like the first vector index, it will be found during the creation of the second vector index that if multiple optional index parameters recommended by the parameter effect data model are used to retrieve in the vector database after the vector data increases or decreases, the recall rate of the retrieval result may still not meet the target recall rate. However, if M is adjusted first, among the multiple optional index parameters recommended by the parameter effect data model, more index parameters will meet the target recall rate, so that the recall rate can be improved by re-determining the index parameters and creating the vector index.

[0165] In addition, as mentioned above, before obtaining the second vector index and the second index parameter, the second index parameter is determined first. Correspondingly, when there is newly written vector data in the vector database, the newly written vector data can be retrieved first using the first vector index and the second index parameter. To facilitate the user to select the vector index and the corresponding index parameter during retrieval, the first vector index and the second index parameter can be marked as version V1’, and the user can choose to retrieve using version V1 or version V1’ according to needs. Similarly, the created second vector index and the corresponding second index parameter can also be marked as version V2.

[0166] The following takes the case where new vector data is written into the vector database as an example for detailed description.

[0167] As described above, the vector index and its index parameters can be versioned. For example, index parameters for a new version can be added in advance. If an index reconstruction operation is triggered subsequently, the new index uses the index parameters of the new version.

[0168] As Figure 2 shown, an implementation method of setting multiple versions of vector indexes and index parameters may include the following steps.

[0169] Step 201: Set a first vector index and a first index parameter for the vector database.

[0170] The first index parameter and the first vector index correspond to version V1. The recall rate corresponding to the first vector index and the first index parameter can meet the target recall rate.

[0171] Step 202: When vector data is written into the vector database for the first time, use the set first vector index and first index parameter.

[0172] For the situation of Step 202, too much vector data is written into the vector database, resulting in the recall rate not meeting the target recall rate. Therefore, the second index parameter is first adjusted.

[0173] For example, in this step, the parameter M and ef construction in one of them can be adjusted, or both can be adjusted. For the situation of temporarily adjusting the parameter M and ef construction in this case, when retrieving the newly written data, this parameter can be used, while for the previously written data, this parameter cannot be used for retrieval. Therefore, for the vector data written into the vector database in Step 202, the first index parameter is still used for retrieval.

[0174] The first vector index and the second index parameter correspond to version V1'.

[0175] Step 203: When vector data is written into the vector database for the second time, use the first vector index and the second index parameter.

[0176] In this step, for the vector data newly written into the vector database, this first vector index and this second index parameter can be used for retrieval, so that the recall rate can be improved.

[0177] Step 204: When vector data is written into the vector database for the third time, if an operation to reconstruct the vector index is triggered, reconstruct the second vector index and use the second index parameter.

[0178] The second vector index and the second index parameter correspond to version V2.

[0179] In this step, since the amount of vector data written is too large, using the first vector index and the second index parameter cannot meet the recall rate either. Therefore, it is necessary to reconstruct the vector index. For example, use this second index parameter and create this second vector index.

[0180] For the embodiments of the present application, during the process of reconstructing the vector index, since the newly written vector data can be retrieved using the first vector index and the second index parameter, the recall rate can also be optimized to a certain extent during this process.

[0181] Through the above operations, multiple vector indexes and their corresponding index parameters are identified, so that users can determine the vector index and index parameters used for retrieval according to identification information such as version numbers.

[0182] In addition, in order to better adjust the recall rate of the embodiments of the present application on the premise of minimizing the impact as much as possible, a possible implementation manner is to first adjust the user parameters, and then, when the adjustment of the user parameters still cannot meet the recall rate, adjust the index parameters and vector indexes.

[0183] First, describe how to adjust the recall rate by adjusting the user parameters. As mentioned above, for the ANN technology, the current mainstream algorithms include two vector retrieval algorithms, HNSW and IVFFlat. The ef search parameter in the HNSW algorithm and the nprobe and max_codes parameters in the IVFFlat algorithm are used to determine the search range. Among them, nprobe is specifically a parameter used to indicate the number of clustering centers searched at one time, and max_codes is specifically a parameter used to indicate the number of vectors searched at one time. The embodiments of the present application can improve the recall rate by adjusting at least one of the ef search or nprobe and max_codes parameters to expand the search range of vector retrieval.

[0184] Specifically, when retrieving the vector database in the aforementioned step 102, the first vector index, the first index parameter, and the first user parameter can be specifically used for retrieval. The first user parameter includes a first retrieval parameter for determining the retrieval range. Correspondingly, the first recall rate and the second recall rate in the foregoing embodiments both correspond to the first vector index, the first index parameter, and the first user parameter.

[0185] Before the aforementioned step 104, that is, before determining the second index parameter and creating the second vector index, the first user parameter can be adjusted to the second user parameter first. That is, the second retrieval parameter used to determine the retrieval range in the second user parameter is greater than the first retrieval parameter in the first user parameter. By this operation, it is possible to increase the recall rate.

[0186] After determining that the fourth recall rate corresponding to the first vector index, the first index parameter, and the second user parameter meets the target recall rate, retrieve using the first vector index, the first index parameter, and the second user parameter.

[0187] After determining that the fourth recall rate does not meet the target recall rate, the process of determining the second index parameter and creating the second vector index can be directly executed. Alternatively, the user parameter can continue to be adjusted. For example, the second user parameter is adjusted to the third user parameter, and the third retrieval parameter used to determine the retrieval range in the third user parameter is greater than the second retrieval parameter. If the fifth recall rate based on the third user parameter, the first vector index, and the first index parameter meets the target recall rate, retrieve using the first vector index, the first index parameter, and the third user parameter. If the fifth recall rate still does not meet the target recall rate, the process of determining the second index parameter and creating the second vector index can be executed.

[0188] After the above adjustment of the third user parameter still does not meet the recall rate, in principle, the user parameter can continue to be adjusted. However, since adjusting the user parameter expands the retrieval range, the retrieval efficiency will be reduced. In some scenarios, the retrieval efficiency may be reduced significantly, such as exceeding the set threshold. Therefore, the implementation method of adjusting the user parameter can be used as an implementation means. When this implementation method cannot improve the recall rate, other means need to be used to improve the recall rate. For example, it can be considered to change to execute the process of determining the second index parameter and creating the second vector index.

[0189] In addition, as mentioned above, for the scheme of adjusting the user parameter, this scheme may be able to increase the recall rate. However, in some cases, the recall rate cannot be improved. For example, the vector data for some knowledge points in the vector database is deleted, and even if the search range is expanded to the entire database, the recall rate cannot be improved. Therefore, it can be considered to use the scheme of adjusting the user parameter when a large amount of vector data is added to the vector database.

[0190] The various implementation manners of the embodiments of the present application are introduced in detail above. The implementation manners of the embodiments of the present application are introduced by way of example in the scenario of a RAG knowledge base. When a certain RAG knowledge base needs to be created, it is estimated that the data volume of the knowledge base is 1 million, the vector dimension is 512, the expected recall rate is 95%, and QPS≥500. The vector data in this type of database only involves the problem of addition, and there will be no vector data deleted. According to business requirements, the HNSW index type is recommended, and the recommended parameters obtained according to the parameter effect data model are M = 16, ef construction = 256.

[0191] After the service is launched, the data growth far exceeds expectations, and the data volume has grown to 50 million. Users feedback that the retrieval results often do not meet expectations. According to traditional technical means, in response to this situation, the vector index can be rebuilt periodically. However, even if the index is rebuilt, the problem of the deteriorating recall effect may not be solved.

[0192] For the above situation, according to the technical solution of the embodiments of the present application, the recall rate can be calculated first to confirm whether the target recall rate is indeed not met. And after confirmation, request intervention can be applied, such as expanding the search range of each retrieval request ef search , so as to reduce the frequency of the problem of "the retrieval results do not meet expectations". When the modification still ef search cannot solve the recall rate problem well, the index parameters can be re-evaluated according to the parameter effect data model, or further through reasoning, to obtain more reasonable index parameters, such as M = 16, ef construction = 512, that is, M remains unchanged, and ef construction is increased. Then a new vector index is created based on the index parameters, and the recall rate of the new index is detected. After the new vector index and the new index parameters meet the target recall rate, the new vector index and the new index parameters can be used for retrieval. Different vector indexes and index parameters correspond to different version numbers. During the process of creating a new vector index, the user can also select the vector index and index parameters needed by himself according to the version number. For example, the user can select the old vector index and the old index parameters, or the old vector index and the new index parameters.

[0193] Through the above processing, after the recall rate of the RAG knowledge base decreases, the recall rate can be quickly increased, thus greatly improving the retrieval experience of users using the RAG knowledge base.

[0194] The embodiments of the present application also provide a computer-readable storage medium storing a computer program, and when the computer program is executed by an electronic device, it can implement one or more steps in the above various methods.

[0195] A computer-readable storage medium may be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0196] Another embodiment of the present application further provides a computer program product including instructions. When the computer program product is executed by an electronic device, one or more steps in the above-mentioned various methods can be implemented.

[0197] The electronic device, computer-readable storage medium, and computer program product provided in this embodiment are all used to execute the methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be elaborated here.

[0198] The terms "first", "second", "third", etc. in the description, claims, and drawings of the present application are used to distinguish different objects, rather than to limit a specific order. The term "and / or" in the description and claims of the present application is used to represent multiple implementation manners. For example, for the expression "A and / or B", it can be understood as including the following several situations: A alone, B alone, and A and B.

[0199] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations, or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, the use of words such as "exemplary" or "for example" is intended to present relevant concepts in a specific manner.

[0200] As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that: they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements on some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for processing a vector database, characterized in that: The method comprises: Determining a first index parameter according to a first vector data parameter of the vector database, and creating a first vector index according to the first index parameter, wherein a first recall rate corresponding to the first vector index and the first index parameter at a first time satisfies a target recall rate; Search the vector database using the first vector index and the first index parameter; Determining that a second recall rate corresponding to the first vector index and the first index parameter at a second time does not meet the target recall rate, wherein the vector data parameter of the vector database is changed from the first vector data parameter to a second vector data parameter; Determine a second index parameter according to the second vector data parameter, and create a second vector index according to the second index parameter, wherein a third recall rate corresponding to the second vector index and the second index parameter at a third time satisfies the target recall rate; Before the second vector index is created, continue to use the first vector index and the first index parameter, or use the first vector index and the second index parameter, to search the vector database, and after the second vector index is created, use the second vector index and the second index parameter to search the vector database.

2. The method according to claim 1, characterized in that The method further comprises: Determine a pre-trained parameter effect data model, wherein the parameter effect data model includes a correspondence between index parameters and vector data parameters; The determining the first index parameter according to the first vector data parameter of the vector database comprises: According to the first vector data parameter, determine a first parameter recommended value including an index parameter through the parameter effect data model, and determine the first index parameter according to the first parameter recommended value; The determining the second index parameter according to the second vector data parameter comprises: According to the second vector data parameter, a second parameter recommended value including an index parameter is determined through the parameter effect data model, and the second index parameter is determined according to the second parameter recommended value, and the first parameter recommended value is different from the second parameter recommended value.

3. The method according to claim 2, characterized in that The data set corresponding to the parameter effect data model includes multiple sample sets; The information in the parameter effect data model includes: a vector index and an index parameter corresponding to each sample set, and the recall rate corresponding to the vector index and the index parameter corresponding to each sample set is greater than the target recall rate.

4. The method according to any one of claims 1 to 3, characterized in that: The determining that a second recall rate corresponding to the first vector index and the first index parameter at a second time does not meet the target recall rate includes: Determine a first vector data response set actually corresponding to the first vector data request; Obtaining, by searching from the vector database according to the first vector index and the first index parameter, a second vector data response set corresponding to the first vector data request; Determining the second recall rate according to the first vector data response set and the second vector data response set; It is determined that the second recall rate does not meet the target recall rate.

5. The method according to claim 4, characterized in that The determining a first vector data response set actually corresponding to the first vector data request includes: Perform a burst search on the vector database according to the first vector data request to obtain the first vector data response set; or According to the first vector data request, a burst search is performed on the vector data in each partition after the vector database is partitioned to obtain the first vector data response set.

6. The method according to any one of claims 1 to 5, characterized in that: The determining that a second recall rate corresponding to the first vector index and the first index parameter at a second time does not meet the target recall rate includes one of the following: Determining that at least one of the at least one recall rates corresponding to the first vector index and the first index parameter within a set period does not meet the target recall rate; Determining that an average of multiple recall rates corresponding to the first vector index and the first index parameter within a set period does not meet the target recall rate; It is determined that the number of recall rates that do not meet the target recall rate among multiple recall rates corresponding to the first vector index and the first index parameter within a set period is greater than or equal to a set threshold.

7. The method according to any one of claims 1 to 6, characterized in that: The using the first vector index and the first index parameter to search the vector database includes: Searching the vector database using the first vector index, the first index parameter, and a first user parameter, wherein the first user parameter includes a first search parameter for determining a search range, and the first recall rate and the second recall rate both correspond to the first vector index, the first index parameter, and the first user parameter; Before determining the second index parameter and creating the second vector index, the method further includes: adjusting the first user parameter to a second user parameter, wherein a second search parameter used to determine a search range in the second user parameter is greater than the first search parameter in the first user parameter; After determining that a fourth recall rate corresponding to the first vector index, the first index parameter, and the second user parameter at a fourth time satisfies the target recall rate, searching the vector database using the first vector index, the first index parameter, and the second user parameter; After determining that the fourth recall rate does not meet the target recall rate, directly performing the process of determining the second index parameter and creating the second vector index; or performing the following process: adjusting the second user parameter to a third user parameter, wherein a third search parameter used to determine a search range in the third user parameter is greater than the second search parameter; After determining that a fifth recall rate corresponding to the first vector index, the first index parameter, and the third user parameter meets the target recall rate, searching the vector database using the first vector index, the first index parameter, and the third user parameter; After determining that the fifth recall rate does not meet the target recall rate, the process of determining the second index parameter and creating the second vector index is performed.

8. The method according to claim 7, characterized in that Before determining the second index parameter and creating the second vector index, continuing to use the first vector index and the first index parameter to search the vector database includes: Searching the vector database using the first vector index, the first index parameter, and the second user parameter or the third user parameter; After determining the second index parameter and creating the second vector index, searching the vector database using the second vector index and the second index parameter includes: The vector database is queried using the second vector index, the second index parameter, and the first user parameter.

9. The method according to claim 7 or 8, characterized in that: The algorithm used to search the vector database is HNSW, and the parameters used for the search range in the user parameters corresponding to the algorithm include ef search ;or The algorithm used to search the vector database is IVFFlat, and the parameters for the search range in the user parameters corresponding to the algorithm include a parameter nprobe for indicating the number of cluster centers to be searched at one time and a parameter max_codes for indicating the number of vectors to be searched at one time; The second search parameter used to determine the search range in the second user parameter is greater than the first search parameter in the first user parameter, including: The nprobe in the second user parameter is greater than the nprobe in the first user parameter, and / or the max_codes in the second user parameter is greater than the max_codes in the first user parameter; The third search parameter used to determine the search scope in the third user parameter is greater than the second search parameter, including: The nprobe in the third user parameter is greater than the nprobe in the second user parameter, and / or the max_codes in the third user parameter is greater than the max_codes in the second user parameter.

10. The method according to any one of claims 1 to 9, characterized in that: The index parameters of the vector database include M and ef construction , the M indicates the maximum number of neighbors of the node, the ef construction Indicates the search range when constructing a vector index; The M in the first index parameter is less than the M in the second index parameter; and / or The first index parameter in ef construction is less than the second index parameter in ef construction .

11. The method according to any one of claims 1 to 10, characterized in that: The first vector index and the first index parameter are identified by a first version number; The first vector index and the second index parameter are identified by a second version number; The second vector index and the second index parameter are identified by a third version number.

12. A device for processing a vector database, characterized in that: The device comprises a memory and a processor, wherein the memory is used to store a vector database, and the processor is used to execute the method according to any one of claims 1 to 11.

13. A readable storage medium, characterized in that: The readable storage medium stores instructions, which, when executed on an electronic device, enable the electronic device to execute the method according to any one of claims 1 to 11.

14. A program product, characterized in that The method comprises: instructions, which, when executed on an electronic device, cause the electronic device to execute the method according to any one of claims 1 to 11.

Images