Vector database retrieval method, device and equipment and readable storage medium
By introducing auxiliary databases and record databases into vector databases, infrequently used vector data are stored in auxiliary databases, the problem of excessive hardware resource utilization of vector databases is solved, and the effect of increasing the number of retrieval and reducing hardware requirements is achieved.
Patent Information
- Application Number
- CN202411948573.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-22
AI Technical Summary
When increasing the number of searches, existing vector databases need to occupy a large amount of CPU and memory resources, resulting in high hardware resource requirements, and adjusting configuration or using disk indexes may slow down the search speed.
By adding auxiliary databases and record databases, infrequently used vector data are stored in auxiliary databases, reducing the hardware pressure of vector databases, thereby increasing the number of searches of vector databases and reducing hardware requirements.
It realizes that while increasing the search amount of vector data, it reduces the hardware demand of vector databases, avoids the problem of excessive hardware resource utilization, and improves the search efficiency.
Smart Images
Figure CN119862215A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of data retrieval, and in particular to a retrieval method, device, equipment and readable storage medium for a vector database. Background Art
[0002] Vector databases are designed specifically for processing large-scale vector data and are widely used in the fields of artificial intelligence and machine learning, especially in scenarios that require efficient processing and retrieval of vector data. The higher the number of retrievals in the vector database, the more beneficial it is to increasing the intelligence of artificial intelligence and the efficiency of machine learning.
[0003] In order to increase the number of searches in the vector database, the industry usually adopts the method of adding nodes. However, this method also has bottlenecks. While increasing the number of searches in the vector database, it requires large CPU resources and memory resources. For example, when the number of searches in the vector database is 10 million, the number of CPU cores required is 55 and the memory required is 177GB. In other words, this method has very high requirements for hardware resources.
[0004] In order to reduce the demand for hardware resources, this can be achieved by adjusting the vector database configuration, but this method also has limitations and may reduce the speed of vector retrieval; if the disk index method is used, it will also have certain limitations on the speed of vector retrieval. Therefore, there is an urgent need for a vector database retrieval method that can overcome the above defects. Summary of the invention
[0005] The purpose of the present invention is to provide a retrieval method, device, equipment and readable storage medium for a vector database. By adding an auxiliary database and a record database, infrequently used vector data are stored in the auxiliary database, thereby reducing the hardware pressure of the vector database, thereby achieving the effect of increasing the number of retrievals of the vector database and reducing the hardware requirements of the vector database.
[0006] In order to achieve the above object, the present invention provides the following technical solutions:
[0007] In a first aspect, the present invention provides a method for searching a vector database, the method comprising:
[0008] Determine target attribute information of target vector data according to input text uploaded by the user;
[0009] Judging whether the target vector data is in the temporary delisting directory in the record database according to the target attribute information;
[0010] If the target vector data is in the temporary delisting directory in the record database, the target vector data is restored from the auxiliary database to the vector database, and the target vector data is deleted from the temporary delisting directory;
[0011] The target vector data is retrieved from the vector database, and the last usage time of the target vector data is updated.
[0012] In some embodiments, the method further comprises:
[0013] If the target vector data is not in the temporary delisting directory in the record database, the target vector data is retrieved from the vector database, and the last use time of the target vector data is updated.
[0014] In some embodiments, the method further comprises:
[0015] The candidate vector data are stored in the vector database and the auxiliary database respectively;
[0016] Deleting expired vector data in the candidate vector data from the vector database according to the last use time and preset time threshold of the candidate vector data;
[0017] The expired vector data is recorded in a temporary delisting directory in the record database.
[0018] In some embodiments, according to the last use time of the candidate vector data and a preset time threshold, deleting expired vector data in the candidate vector data from the vector database includes:
[0019] For each candidate vector data, if the last use time of the candidate vector data is earlier than a preset time threshold, the candidate vector data is determined to be expired vector data;
[0020] Delete expired vector data in the vector database.
[0021] In some embodiments, storing the candidate vector data in the vector database and the auxiliary database respectively includes:
[0022] Create vector table and auxiliary data table;
[0023] According to the vector table and the auxiliary data table, the candidate vector data are stored in the vector database and the auxiliary database respectively.
[0024] In some embodiments, updating the last usage time of the target vector data includes:
[0025] Get the current usage time;
[0026] Using the current usage time, update the last usage time of the target vector data.
[0027] In a second aspect, the present invention further provides a vector database retrieval device, the device comprising:
[0028] An information confirmation module is used to determine target attribute information of target vector data according to the input text uploaded by the user;
[0029] A location determination module, used to determine whether the target vector data is in a temporary delisting directory in the record database according to the target attribute information;
[0030] A data recovery module, used for recovering the target vector data from the auxiliary database to the vector database if the target vector data is in the temporary delisting directory in the record database, and deleting the target vector data from the temporary delisting directory;
[0031] The data retrieval module is used to retrieve target vector data from the vector database and update the last use time of the target vector data.
[0032] In a third aspect, the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the vector database retrieval method provided in the first aspect when executing the computer program.
[0033] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the vector database retrieval method provided in the first aspect is implemented.
[0034] In a fifth aspect, the present invention further provides a computer program product, comprising a computer program, which, when executed by a processor, implements the vector database retrieval method provided in the first aspect.
[0035] The beneficial effects of the present invention are:
[0036] The scheme in the present application first determines the target attribute information of the target vector data based on the input text uploaded by the user; then determines whether the target vector data is in the temporary delisting directory in the record database based on the target attribute information; if the target vector data is in the temporary delisting directory in the record database, the target vector data is restored from the auxiliary database to the vector database, and the target vector data is deleted from the temporary delisting directory; finally, the target vector data is retrieved from the vector database, and the last use time of the target vector data is updated. The above scheme restores the target vector data from the auxiliary database to the vector database only when the target vector data is in the temporary delisting directory in the record database, that is, the vector data that is not in the temporary delisting directory in the record database will only be stored in the auxiliary database, which greatly reduces the hardware pressure of the vector database, and uses the vector database for vector data retrieval, which greatly increases the retrieval volume of vector data compared to other databases, that is, this scheme reduces the hardware requirements of the vector database while increasing the retrieval volume of vector data.
[0037] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention and implement it according to the contents of the specification, the following is a detailed description of the preferred embodiments of the present invention in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0038] Figure 1 A schematic diagram of a flow chart of a vector database search method according to an embodiment of the present invention;
[0039] Figure 2 A schematic flow chart of another vector database retrieval method according to an embodiment of the present invention;
[0040] Figure 3 A schematic flow chart of another method for searching a vector database according to an embodiment of the present invention;
[0041] Figure 4 A schematic diagram of the structure of a vector database retrieval device according to an embodiment of the present invention;
[0042] Figure 5 It is a schematic diagram of the structure of another vector database retrieval device shown in one embodiment of the present invention;
[0043] Figure 6 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION
[0044] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0045] It should be noted that references to "one embodiment", "embodiment", "example embodiment", etc. in this specification refer to the embodiment being described which may include specific features, structures or characteristics, but not every embodiment must include these specific features, structures or characteristics. In addition, such expressions do not refer to the same embodiment. Furthermore, when describing specific features, structures or characteristics in conjunction with an embodiment, whether or not there is an explicit description, it is indicated that incorporating such features, structures or characteristics into other embodiments is within the knowledge of those skilled in the art.
[0046] In addition, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.
[0047] In some embodiments, Figure 1 As shown, a method for searching a vector database is provided, the method comprising:
[0048] S101, determining target attribute information of target vector data according to input text uploaded by a user.
[0049] The input text is the text that the user inputs to the server, and the target attribute information may include the vector table name, index type, etc.
[0050] Specifically, when the user needs to perform a database search, he will input text into the server. After obtaining the input text uploaded by the user, the vector table name and index type of the target vector data can be directly extracted from the input text; the data set ID can also be extracted from the input text, and the target vector data and its target attribute information can be searched according to the data set ID.
[0051] S102, judging whether the target vector data is in a temporary delisting directory in the record database according to the target attribute information.
[0052] The record database may be a redis database, which is used to record the vector data deleted from the vector database.
[0053] Specifically, the target attribute information is used as a keyword to search in the temporary delisted directory of the record database. If a certain vector data is found that is consistent with the target attribute information of the target vector data, it means that the target vector data is in the temporary delisted directory of the record database, that is, the target vector data has been deleted from the vector database; otherwise, it means that the target vector data is not in the temporary delisted directory of the record database, that is, the target vector data is still stored in the vector database.
[0054] S103: If the target vector data is in the temporary delisted directory in the record database, the target vector data is restored from the auxiliary database to the vector database, and the target vector data is deleted from the temporary delisted directory.
[0055] The auxiliary database may be a mongo database, which is used to store all vector data, and the vector database may be a milvus database, which is used to store commonly used vector data.
[0056] Specifically, if the target vector data is in the temporary delisting directory in the record database, it means that the target vector data is no longer in the vector database, so it is necessary to find the target vector data from the auxiliary database and copy it to the vector database. Since the target vector data is already in the vector database at this time, it is necessary to delete the target vector data from the temporary delisting directory in the record database.
[0057] S104, retrieving target vector data from the vector database, and updating the last use time of the target vector data.
[0058] The last usage time of the target vector data is the time when the target vector data was last retrieved.
[0059] Specifically, since the target vector data is already in the vector database, the target vector data can be directly retrieved from the vector database. At this time, the last use time of the target vector data has changed, and the last use time of the target vector data needs to be updated to the current time.
[0060] Optionally, if the target vector data is not in the temporary delisting directory in the record database, the target vector data is retrieved from the vector database and the last usage time of the target vector data is updated.
[0061] Specifically, if the target vector data is not in the temporary delisting directory in the record database, it means that the target vector data is still in the vector database. At this time, no additional processing is required, and the target vector data can be directly retrieved from the vector database. At this time, the last use time of the target vector data has changed, and the last use time of the target vector data needs to be updated to the current time.
[0062] Optionally, the method for updating the time may be: obtaining the current usage time; and updating the last usage time of the target vector data using the current usage time.
[0063] Specifically, the target attribute information includes the last use time, and the current time can be obtained and used to replace the last use time in the target attribute information, that is, the update of the last use time is completed.
[0064] In the above embodiment, the target attribute information of the target vector data is first determined according to the input text uploaded by the user; then, it is determined whether the target vector data is in the temporary delisting directory in the record database according to the target attribute information; if the target vector data is in the temporary delisting directory in the record database, the target vector data is restored from the auxiliary database to the vector database, and the target vector data is deleted from the temporary delisting directory; finally, the target vector data is retrieved from the vector database, and the last use time of the target vector data is updated. The above scheme restores the target vector data from the auxiliary database to the vector database only when the target vector data is in the temporary delisting directory in the record database, that is, the vector data that is not in the temporary delisting directory in the record database will only be stored in the auxiliary database, which greatly reduces the hardware pressure of the vector database, and uses the vector database for vector data retrieval, which greatly increases the retrieval volume of vector data compared to other databases, that is, this scheme reduces the hardware requirements of the vector database while increasing the retrieval volume of vector data.
[0065] In another embodiment, Figure 2 As shown, the retrieval method of the vector database also includes:
[0066] S201, storing the candidate vector data in a vector database and an auxiliary database respectively.
[0067] Specifically, after the administrator has organized the documents to be published, he slices and segments the documents, and then calls the interface to publish them. When publishing the documents, the documents will first be embedded vectorized to obtain candidate vector data, and then the candidate vector data will be inserted into the vector database, and then inserted into the auxiliary database as a backup.
[0068] Optionally, the above method may also be: creating a vector table and an auxiliary data table; and storing the candidate vector data in a vector database and an auxiliary database respectively according to the vector table and the auxiliary data table.
[0069] Specifically, before storing the candidate vector data, it is necessary to create a vector table in the vector database. The key configuration required for creating the vector table can be set, including vector dimension, vector model, vector index, scalar index and other information. After setting, the vector table is created, and at the same time, an auxiliary data table is created for the auxiliary database with the same table name, the same field (plus the last used time field), and index (scalar index as auxiliary database index). Among them, the vector table and the auxiliary data table are both used to record and query vector data, and then the candidate vector data are stored in the vector database and the auxiliary database respectively according to the vector table and the auxiliary data table, so as to facilitate the subsequent search for target vector data in the vector database and the auxiliary database according to the vector table and the auxiliary data table.
[0070] S202: Deleting expired vector data in the candidate vector data from the vector database according to the last use time of the candidate vector data and a preset time threshold.
[0071] The expired vector data is candidate vector data whose last usage time is earlier than a preset time threshold.
[0072] Optionally, for each candidate vector data, if the last usage time of the candidate vector data is earlier than a preset time threshold, the candidate vector data is determined to be expired vector data; and the expired vector data in the vector database is deleted.
[0073] The preset time threshold is a preset time threshold, which can be adjusted according to actual needs, for example, set to two weeks.
[0074] Specifically, if the last usage time of a candidate vector data is earlier than a preset time threshold, it means that the candidate vector data is an infrequently used vector data, that is, expired vector data. In order to reduce its occupancy of the vector database memory, it can be deleted from the vector database.
[0075] Exemplarily, if the preset time threshold is set to two weeks, if the current time is 00:00 on December 26, then the preset time threshold is 00:00 on December 12. If the last use time of a candidate vector data is earlier than 00:00 on December 12, for example, 10:00 on December 10, then the candidate vector data is determined to be expired vector data, and the expired vector data is deleted from the vector database. If the last use time of a candidate vector data is later than 00:00 on December 12, for example, 10:00 on December 15, then the candidate data vector is not processed.
[0076] S203, recording the expired vector data in a temporary delisting directory in the record database.
[0077] Specifically, after deleting the expired vector data from the vector database, it is also necessary to record the expired vector data in a temporary delisting directory in the record database according to the attribute information of the expired vector data, so that the expired vector data can be found in the temporary delisting directory when the expired vector data is subsequently retrieved.
[0078] In the above embodiment, the candidate vector data is first stored in the vector database and the auxiliary database respectively; then, according to the last use time of the candidate vector data and the preset time threshold, the expired vector data in the candidate vector data is deleted from the vector database; finally, the expired vector data is recorded in the temporary delisting directory in the record database. The above method can delete the infrequently used vector data from the vector database, greatly reducing the memory usage of the vector database and alleviating the memory pressure on the database.
[0079] In order to more comprehensively demonstrate the present solution, this embodiment provides an optional method for searching a vector database, such as Figure 3 As shown:
[0080] S301, create a vector table and an auxiliary data table.
[0081] S302, according to the vector table and the auxiliary data table, storing the candidate vector data in the vector database and the auxiliary database respectively.
[0082] S303: Deleting expired vector data in the candidate vector data from the vector database according to the last use time of the candidate vector data and a preset time threshold.
[0083] Specifically, for each candidate vector data, if the last use time of the candidate vector data is earlier than a preset time threshold, the candidate vector data is determined to be expired vector data; and the expired vector data in the vector database is deleted.
[0084] S304, recording the expired vector data in a temporary delisting directory in the record database.
[0085] S305: Determine target attribute information of target vector data according to the input text uploaded by the user.
[0086] S306: Determine whether the target vector data is in the temporary delisting directory in the record database according to the target attribute information.
[0087] S307: If the target vector data is in the temporary delisted directory in the record database, the target vector data is restored from the auxiliary database to the vector database, and the target vector data is deleted from the temporary delisted directory.
[0088] S308, retrieving target vector data from the vector database, and updating the last usage time of the target vector data.
[0089] S309: If the target vector data is not in the temporary delisting directory in the record database, the target vector data is retrieved from the vector database, and the last use time of the target vector data is updated.
[0090] Specifically, updating the last usage time of the target vector data includes: obtaining the current usage time; and updating the last usage time of the target vector data using the current usage time.
[0091] The specific process of the above S301-S309 can refer to the description of the above method embodiment, and its implementation principle and technical effect are similar, which will not be repeated here.
[0092] Based on the same inventive concept, the embodiment of the present application also provides a vector database retrieval device for implementing the above-mentioned vector database retrieval method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in the embodiments of the retrieval device for one or more vector databases provided below can refer to the limitations of the vector database retrieval method above, and will not be repeated here.
[0093] In one embodiment, Figure 4 As shown, a retrieval device for a vector database is provided, the device comprising:
[0094] The information confirmation module 40 is used to determine the target attribute information of the target vector data according to the input text uploaded by the user;
[0095] A location determination module 41 is used to determine whether the target vector data is in a temporary delisting directory in the record database according to the target attribute information;
[0096] A data recovery module 42 is used for recovering the target vector data from the auxiliary database to the vector database and deleting the target vector data from the temporary delisting directory if the target vector data is in the temporary delisting directory in the record database;
[0097] The data retrieval module 43 is used to retrieve the target vector data from the vector database and update the last use time of the target vector data.
[0098] In another embodiment, the above Figure 4 The retrieval device of the vector database is also used for: if the target vector data is not in the temporary delisting directory in the record database, retrieving the target vector data from the vector database and updating the last use time of the target vector data.
[0099] In another embodiment, Figure 5 As shown above Figure 4 The retrieval device of the vector database in further comprises:
[0100] A data storage module 44, used to store the candidate vector data in the vector database and the auxiliary database respectively;
[0101] A data deletion module 45, configured to delete expired vector data in the candidate vector data from the vector database according to the last use time of the candidate vector data and a preset time threshold;
[0102] The data recording module 46 is used to record the expired vector data in a temporary delisting directory in the record database.
[0103] In another embodiment, the above Figure 5The data deletion module 45 is specifically used to: for each candidate vector data, if the last use time of the candidate vector data is earlier than a preset time threshold, determine that the candidate vector data is expired vector data; and delete the expired vector data in the vector database.
[0104] In another embodiment, the above Figure 5 The data storage module 44 is specifically used to: create a vector table and an auxiliary data table; and store the candidate vector data in the vector database and the auxiliary database respectively according to the vector table and the auxiliary data table.
[0105] In another embodiment, the above Figure 4 The data retrieval module 43 is specifically used to: obtain the current usage time; and update the last usage time of the target vector data using the current usage time.
[0106] The present application also provides an electronic device, in some implementations, referring to Figure 6 As shown, the electronic device 700 includes an input unit 710, a memory 720, a processor 730, and an output unit 740. The memory 720 stores program instructions that can be run on the processor 730, and the processor 730 calls the program instructions to execute the retrieval method and / or technical solution based on the vector database in the above embodiment. The electronic device 700 can be a mobile terminal device such as a mobile phone, a computer, etc.
[0107] In addition, the embodiment of the present application also provides a computer-readable storage medium for storing a computer program for executing the retrieval method of the vector database. For example, a computer program instruction, when executed by a computer, can call or provide the method and / or technical solution according to the present application through the operation of the computer. The program instruction for calling the method of the present application may be stored in a fixed or removable storage medium, and / or transmitted through a data stream in a broadcast or other signal-bearing medium and / or stored in a storage medium that runs according to the program instruction.
[0108] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.
[0109] The technical features of the above embodiments may be arbitrarily integrated. To make the description concise, not all possible integrations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the integration of these technical features, they should be considered to be within the scope of this specification.
[0110] The above embodiments only express several implementation methods of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the patent of the present invention shall be subject to the attached claims.
Claims
1. A method for searching a vector database, characterized in that: The method comprises: Determine target attribute information of target vector data according to input text uploaded by the user; Determining whether the target vector data is in a temporary delisting directory in a record database according to the target attribute information; If the target vector data is in the temporary delisted directory in the record database, restore the target vector data from the auxiliary database to the vector database, and delete the target vector data from the temporary delisted directory; The target vector data is retrieved from the vector database, and the last usage time of the target vector data is updated.
2. The vector database retrieval method according to claim 1, characterized in that: The method further comprises: If the target vector data is not in the temporary delisting directory in the record database, the target vector data is retrieved from the vector database, and the last use time of the target vector data is updated.
3. The vector database retrieval method according to claim 1, characterized in that: The method further comprises: storing the candidate vector data in the vector database and the auxiliary database respectively; Deleting expired vector data in the candidate vector data from the vector database according to the last use time of the candidate vector data and a preset time threshold; The expired vector data is recorded in a temporary delisting directory in the record database.
4. The vector database retrieval method according to claim 3, characterized in that: Deleting expired vector data in the candidate vector data from the vector database according to the last use time of the candidate vector data and a preset time threshold, comprising: For each candidate vector data, if the last use time of the candidate vector data is earlier than the preset time threshold, the candidate vector data is determined to be expired vector data; Deleting expired vector data in the vector database.
5. The vector database retrieval method according to claim 3, characterized in that: Storing the candidate vector data in the vector database and the auxiliary database respectively includes: Create vector table and auxiliary data table; According to the vector table and the auxiliary data table, the candidate vector data are stored in the vector database and the auxiliary database respectively.
6. The vector database search method according to any one of claims 1 to 2, characterized in that: Update the last use time of the target vector data, including: Get the current usage time; The last usage time of the target vector data is updated using the current usage time.
7. A retrieval device for a vector database, characterized in that: The device comprises: An information confirmation module is used to determine target attribute information of target vector data according to the input text uploaded by the user; A location determination module, used for determining whether the target vector data is in a temporary delisting directory in a record database according to the target attribute information; A data recovery module, used for recovering the target vector data from the auxiliary database to the vector database if the target vector data is in the temporary delisted directory in the record database, and deleting the target vector data from the temporary delisted directory; A data retrieval module is used to retrieve the target vector data from the vector database and update the last use time of the target vector data.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the vector database retrieval method described in any one of claims 1 to 6 is implemented.
9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the vector database retrieval method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the vector database search method described in any one of claims 1 to 6 is implemented.