High-dimensional vector search method for target database and related equipment

Abstract

An embodiment of the present invention provides a high-dimensional vector search method for a target database. Through shared memory, a high-dimensional auxiliary process integrated and encapsulated with a high-dimensional vector search library is communicated with the target database. When a high-dimensional vector search is required, the search is performed through the high-dimensional auxiliary process. Since the high-dimensional auxiliary process is integrated and encapsulated, the search capability of the high-dimensional vector search library for high-dimensional vectors can be fully utilized to improve the search speed for high-dimensional vectors in the target database. In addition, low-dimensional vector search or high-dimensional vector search can be flexibly performed. By expanding the search function of the target database for high-dimensional vectors, the original database query function of the target database and the high-dimensional vector search function are combined, thereby improving the vector search efficiency and reducing the development cost.

Description

Technical Field

[0001] The present invention relates to the field of data search, and in particular to a high-dimensional vector search method for a target database and related equipment. Background Art

[0002] In the Internet era, the scale of unstructured data is growing rapidly, such as video, images, voice, text, etc. With the rise of deep learning methods in various fields, most unstructured data can be converted into high-dimensional vectors for structured representation through deep learning, and then these vector data can be effectively used through vector search methods. However, in actual applications, these data are stored in databases. Due to the large time span of database data resources, most databases are developed based on low-dimensional vectors, so they only support lower-dimensional vector searches. For example, the imgsmlr extension and cube extension of PostgreSQL store high-dimensional structured data in the PostgreSQL database, and it is not possible to directly use existing vector search methods for high-dimensional vector searches. Summary of the invention

[0003] The embodiment of the present invention provides a high-dimensional vector search method for a target database, which can realize communication between a high-dimensional auxiliary process integrated and encapsulated with a high-dimensional vector search library and a target database through shared memory. When a high-dimensional vector search is required, the search is performed through the high-dimensional auxiliary process. Since the high-dimensional auxiliary process is integrated and encapsulated, the search capability of the high-dimensional vector search library for high-dimensional vectors can be fully utilized to improve the search speed for high-dimensional vectors in the target database, and low-dimensional vector search or high-dimensional vector search can be flexibly performed. By expanding the search function of the target database for high-dimensional vectors, the original database query function (mainly scalar data) of the target database is combined with the high-dimensional vector search function, thereby improving the vector search efficiency and reducing the development cost.

[0004] In a first aspect, an embodiment of the present invention provides a method for searching a high-dimensional vector in a target database, the method comprising:

[0005] When the high-dimensional auxiliary process reads the request corresponding to the service process in the shared memory, it returns a response result corresponding to the request and sends it to the shared memory;

[0006] The service process reads the response result in the shared memory and returns it to the client;

[0007] The high-dimensional auxiliary process integrates and encapsulates a high-dimensional vector search library, and the service process is used to process requests between the target database and the client.

[0008] Optionally, before the step of returning a response result corresponding to the request and sending it to the shared memory when the high-dimensional auxiliary process reads the request corresponding to the service process in the shared memory, the method further includes:

[0009] The service process sends the request to the shared memory through a preset index extension, accesses the shared memory in a timed or real-time manner through the preset index extension, reads the state of the shared memory, and determines whether there is a response result corresponding to the request;

[0010] The high-dimensional auxiliary process accesses the shared memory periodically or in real time, reads the state of the shared memory, and determines whether there is the request corresponding to the service process;

[0011] The access of the service process and the high-dimensional auxiliary process to the shared memory is synchronized through a lock.

[0012] Optionally, the index table of the target database is stored in a disk, and the method further includes:

[0013] Each time the high-dimensional auxiliary process is initialized, the index table is loaded into the shared memory, and the index table is used to record the query request corresponding to the service process;

[0014] When the index table in the shared memory changes, the changed index table is sent to the disk for updating.

[0015] Optionally, the request includes an operation request for the high-dimensional vector search library, the response result includes an operation result of the high-dimensional vector search library, and before the step of updating the row when the index table in the shared memory changes and sending the changed index table to the disk, the method further includes:

[0016] When the high-dimensional auxiliary process reads the operation request corresponding to the service process in the shared memory, it operates the high-dimensional vector search library according to the operation request and sends the corresponding operation result to the shared memory;

[0017] The service process reads the operation result in the shared memory and returns it to the client.

[0018] Optionally, the method further includes:

[0019] By processing the abnormal disconnection of the client connection and the request interruption, deadlock of the lock in the shared memory is prevented.

[0020] Optionally, the preset index extension includes parameters, access methods and operator classes, wherein the parameters are used to support the index creation process and index search process of the index extension, different access methods correspond to different response results, and the operator class is used to identify the specific data type corresponding to the access method.

[0021] Optionally, the access method includes at least one of index creation, index insertion, index deletion, and index query, and the service process sends the request to the shared memory through a preset index extension, including:

[0022] Obtaining the access method of the client through the service process;

[0023] Encapsulating the request corresponding to the access method to obtain the request;

[0024] The request is sent to the shared memory.

[0025] In a second aspect, an embodiment of the present invention provides a high-dimensional vector search device for a target database, the device comprising:

[0026] The reading module is used to return a response result corresponding to the request and send it to the shared memory when the high-dimensional auxiliary process reads the request corresponding to the service process in the shared memory;

[0027] A return module, used for reading the response result in the shared memory through the service process and returning it to the client;

[0028] The high-dimensional auxiliary process integrates and encapsulates a high-dimensional vector search library, and the service process is used to process requests between the target database and the client.

[0029] In a third aspect, an embodiment of the present invention provides a high-dimensional vector search system for a target database, the system comprising: a high-dimensional auxiliary process, a service process, a shared memory, and a client, wherein the high-dimensional auxiliary process integrates and encapsulates a high-dimensional vector search library, and the service process is used to process requests between the target database and the client;

[0030] The high-dimensional auxiliary process is used to return a response result corresponding to the request and send it to the shared memory when reading the request corresponding to the service process in the shared memory;

[0031] The service process is used to read the response result in the shared memory and return it to the client;

[0032] The shared memory is used to receive the response result sent by the high-dimensional auxiliary process, and is also used to receive the request corresponding to the service process;

[0033] The client is used to send the request to the service process, and is also used to receive the response result sent by the service process.

[0034] In a fourth aspect, an embodiment of the present invention provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps in the high-dimensional vector search method for the target database provided in the embodiment of the present invention are implemented.

[0035] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps in the high-dimensional vector search method for a target database provided in an embodiment of the invention are implemented.

[0036] In the embodiment of the present invention, when the high-dimensional auxiliary process reads the request corresponding to the service process in the shared memory, the response result corresponding to the request is returned to the shared memory; the response result in the shared memory is read by the service process and returned to the client; wherein the high-dimensional auxiliary process is integrated and encapsulated with a high-dimensional vector search library, the service process is used to process the request between the target database and the client, and the target database only supports the search of low-dimensional vectors. Through the shared memory, the high-dimensional auxiliary process integrated and encapsulated with the high-dimensional vector search library can communicate with the target database, and when a high-dimensional vector search is required, the search is performed through the high-dimensional auxiliary process. Since the high-dimensional auxiliary process is integrated and encapsulated, the search capability of the high-dimensional vector search library for high-dimensional vectors can be fully utilized to improve the search speed of the high-dimensional vector in the target database, and low-dimensional vector search or high-dimensional vector search can be flexibly performed. By expanding the search function of the target database for high-dimensional vectors, the original database query function (mainly scalar data) of the target database and the high-dimensional vector search function are combined to improve the vector search efficiency and reduce the development cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0038] Figure 1 It is an architecture diagram of a high-dimensional vector search system for a target database provided by an embodiment of the present invention;

[0039] Figure 1aThis is another architecture diagram of a high-dimensional vector search system for a target database provided by an embodiment of the present invention.

[0040] Figure 2 is a flow chart of a high-dimensional auxiliary process provided by an embodiment of the present invention;

[0041] Figure 3 is a flow chart of a high-dimensional vector search method for a target database provided by an embodiment of the present invention;

[0042] Figure 4 is a flow chart of another high-dimensional vector search method for a target database provided by an embodiment of the present invention;

[0043] Figure 5 It is a structural schematic diagram of a high-dimensional vector search device for a target database provided by an embodiment of the present invention;

[0044] Figure 6 It is a structural schematic diagram of another high-dimensional vector search device for a target database provided by an embodiment of the present invention;

[0045] Figure 7 It is a structural schematic diagram of another high-dimensional vector search device for a target database provided by an embodiment of the present invention;

[0046] Figure 8 It is a structural schematic diagram of another high-dimensional vector search device for a target database provided by an embodiment of the present invention;

[0047] Fig. 9 is a structural diagram of a sending module provided by an embodiment of the present invention;

[0048] Fig.10 It is a structural schematic diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] See also Figure 1 , Figure 1 is an architecture diagram of a high-dimensional vector search system for a target database provided by an embodiment of the present invention, such as Figure 1 As shown, the high-dimensional vector search system of the target database includes: a target database 1, a high-dimensional auxiliary process 2, a service process 3, a shared memory 4, and a client 5.

[0051] The target database only supports searching of low-dimensional vectors, and the target database is a PostgreSQL database.

[0052] Among them, the above-mentioned high-dimensional auxiliary process 2 integrates and encapsulates a high-dimensional vector search library, and the above-mentioned service process 3 is used to process the request between the above-mentioned target database 1 and the above-mentioned client 5; the above-mentioned high-dimensional auxiliary process 2 is used to read the request corresponding to the service process 3 in the shared memory 4, and return the response result corresponding to the above-mentioned request to the above-mentioned shared memory 4; the above-mentioned service process 3 is used to read the above-mentioned response result in the above-mentioned shared memory and return it to the client 5; the above-mentioned shared memory 4 is used to receive the above-mentioned response result sent by the above-mentioned high-dimensional auxiliary process 2, and is also used to receive the above-mentioned request corresponding to the above-mentioned service process 3; the above-mentioned client 5 is used to send the above-mentioned request to the above-mentioned service process 3, and is also used to receive the above-mentioned response result sent by the above-mentioned service process 3.

[0053] Furthermore, the target database 1 is connected to the client 5 by signal, and the user operates the client 5 to make a query request to the low-dimensional vector search library, and a vector search is performed in the low-dimensional vector search library, and the corresponding result is returned to the client 5 after the search. Specifically, the general target database 1 will create a new service process 3 for each connection of the client 5 to process the corresponding request.

[0054] In an embodiment of the present invention, the high-dimensional auxiliary process 2 integrated and encapsulated with the high-dimensional vector search library can communicate with the target database 1 through the shared memory 4. When a high-dimensional vector search is required, the search is performed through the high-dimensional auxiliary process 2. Since the high-dimensional auxiliary process 2 is integrated and encapsulated, the search capability of the high-dimensional vector search library for high-dimensional vectors can be fully utilized, thereby improving the search speed for high-dimensional vectors in the target database 1, and low-dimensional vector searches or high-dimensional vector searches can be flexibly performed. Therefore, there is no need to develop a search method for the target database 1. The current vector search method can be used to implement high-dimensional vector searches in the target database 1, thereby improving vector search efficiency and reducing development costs.

[0055] In a possible embodiment, the high-dimensional vector search system of the target database 1 further includes an index extension 6. The service process 3 sends the request to the shared memory 4 through the preset index extension 6, and accesses the shared memory 4 in a regular or real-time manner through the preset index extension 6, reads the state of the shared memory 4, and determines whether there is a response result corresponding to the request; the high-dimensional auxiliary process 2 accesses the shared memory 4 in a regular or real-time manner, reads the state of the shared memory 4, and determines whether there is the request corresponding to the service process 3; wherein the access of the service process 3 and the high-dimensional auxiliary process 2 to the shared memory 4 is synchronized through a lock.

[0056] To better illustrate the embodiment of the present invention, PostgreSQL is used as the target database and the faiss library is used as the high-dimensional vector search library to further illustrate the system architecture. The faiss library is a framework that provides efficient similarity search and clustering for dense vectors. The faiss library can provide multiple search methods and has a fast search speed.

[0057] For details, see Figure 1a , Figure 1a is an architecture diagram of another high-dimensional vector search system for a target database provided by an embodiment of the present invention, such as Figure 1a As shown, the high-dimensional vector search system of the target database includes: PostgreSQL library 11, PGFaiss (high-dimensional auxiliary process) 12, Postgres (service process) 13, shared memory 14, and client 15.

[0058] Among them, the above-mentioned PGFaiss12 high-dimensional auxiliary process integrates and encapsulates the faiss library, and the above-mentioned Postgres high-dimensional auxiliary process is used to process the request between the above-mentioned PostgreSQL library 11 and the above-mentioned client 15; the above-mentioned Postgres service process 13 is used to read the request corresponding to the service process Postgres13 in the shared memory 14, and return the response result corresponding to the above-mentioned request to the above-mentioned shared memory 14; the above-mentioned Postgres service process 13 is used to read the above-mentioned response result in the above-mentioned shared memory and return it to the client 15; the above-mentioned shared memory 14 is used to receive the above-mentioned response result sent by the above-mentioned PGFaiss high-dimensional auxiliary process 12, and is also used to receive the above-mentioned request corresponding to the above-mentioned Postgres service process 13; the above-mentioned client 15 is used to send the above-mentioned request to the above-mentioned Postgres service process 13, and is also used to receive the above-mentioned response result sent by the above-mentioned Postgres service process 13.

[0059] Furthermore, the above-mentioned PostgreSQL library 11 is connected to the client 15 signal, and the user operates the client 15 to make a query request to the PostgreSQL library 11, and a vector search is performed in the PostgreSQL library 11. After the corresponding result is searched, it is returned to the client 15. Specifically, the general PostgreSQL library 11 will create a new Postgres service process 13 for each client 15 connection to handle the corresponding request. It should be noted that in the application of high-dimensional vector search, the data scale is usually large, and each instance data is also high-dimensional. In order to ensure the speed of the search, it is usually a memory-based search, that is, the data features and index structure are loaded into the memory for search. The PostgreSQL library 11 will create a new Postgres service process 13 for each client 15 connection to handle the corresponding request. In order to meet the search situation of the index of the PostgreSQL library 11 when facing multiple connections, the embodiment of the present invention creates a PGFaiss high-dimensional auxiliary process 12 for assistance. The PGFaiss high-dimensional auxiliary process 12 receives the request of the Postgres service process 13 and returns the corresponding response result.

[0060] In a possible embodiment, the high-dimensional vector search system of the target database further includes pg_faiss (index extension) 6. The Postgres service process 13 sends the request to the shared memory 14 through a preset index extension, and accesses the shared memory 14 in a timed or real-time manner through the preset index extension, reads the state of the shared memory 14, and determines whether there is a response result corresponding to the request; the PGFaiss high-dimensional auxiliary process 12 accesses the shared memory 14 in a timed or real-time manner, reads the state of the shared memory 14, and determines whether there is the request corresponding to the Postgres service process 13; wherein the access of the Postgres service process 13 and the PGFaiss high-dimensional auxiliary process 12 to the shared memory 14 is synchronized through a lock. The pg_faiss index extension 6 is mainly adapted to the index extension interface of the PostgreSQL library 11, and establishes communication between the Postgres service process 13 and the PGFaiss high-dimensional auxiliary process 12 through the pg_faiss index extension 6 and the shared memory 14, and completes the index and query request connected by the client 15.

[0061] Optional, see Figure 2 , Figure 2 is a flow chart of a high-dimensional auxiliary process provided by an embodiment of the present invention, such as Figure 2 As shown, the high-dimensional auxiliary process includes the following steps:

[0062] 201. Information processing function registration.

[0063] In the embodiment of the present invention, the information processing function may also be called a signal processing function, which is mainly used to determine how a process processes a signal. The signal processing function may use a signal() function or a signal set function group.

[0064] 202. Index loading for high-dimensional vector search library.

[0065] In the embodiment of the present invention, the high-dimensional vector search library may be any high-dimensional vector search library. Preferably, the high-dimensional vector search library may be a faiss library.

[0066] 203. Connect to shared memory.

[0067] In the embodiment of the present invention, the above-mentioned shared memory refers to the memory that can be accessed by different central processing units (CPUs) in a multi-processor computer system. Multiple unrelated processes can be allowed to access the same memory. Specifically, the above-mentioned shared memory can be connected through a bus interface.

[0068] 204. Read the shared memory status.

[0069] In the high-dimensional auxiliary process, the status of the shared memory is continuously read to obtain whether the index table in the shared memory is updated, such as whether the corresponding request is written.

[0070] 205. Determine whether there is a request.

[0071] In the embodiment of the present invention, the shared memory is loaded with an index table, and the index table can record the request and response results in a written form. In the high-dimensional auxiliary process, if there is a request in the shared memory, it proceeds to step 206, and if there is no request, it proceeds to step 204.

[0072] 206. Request processed.

[0073] In the embodiment of the present invention, the corresponding high-dimensional vector is searched through the high-dimensional vector search library, and the high-dimensional vector and / or the corresponding data obtained by the search are used as the response result.

[0074] 207. Write the response to the shared memory and update the shared memory status.

[0075] In the embodiment of the present invention, the response result is sent to the shared memory, and according to the index table in the shared memory, the index table records the corresponding response result. It should be noted that the operation of sending the response result to the shared memory may also be referred to as writing the response result to the shared memory.

[0076] See also Figure 3 , Figure 3is a flow chart of a high-dimensional vector search method for a target database provided by an embodiment of the present invention. Figure 3 As shown, based on the architecture of the high-dimensional vector search system of the target database, the high-dimensional vector search method of the target database includes the following steps:

[0077] 301. When the high-dimensional auxiliary process reads the request corresponding to the service process in the shared memory, it returns the response result corresponding to the request and sends it to the shared memory.

[0078] In an embodiment of the present invention, the above-mentioned high-dimensional auxiliary process integrates and encapsulates a high-dimensional vector search library and a high-dimensional auxiliary process. It should be noted that in the application of high-dimensional vector search, the data scale is usually relatively large, and each instance data is also high-dimensional. In order to ensure the speed of the search, it is usually a memory-based search, that is, the data features (high-dimensional vectors) and the index structure are loaded into the memory for searching. The target database will create a new high-dimensional auxiliary process for each client connection to handle the corresponding request. In order to meet the search situation of the target database index when facing multiple client connections, the embodiment of the present invention creates a high-dimensional auxiliary process, which receives the request of the service process and returns the corresponding response result.

[0079] Further, the service process receives the request from the client and establishes a connection from the service process to the shared memory so that the service process can access the shared memory. The service process writes the request to the shared memory for the high-dimensional auxiliary process to read. When the high-dimensional auxiliary process reads the request in the shared memory, it searches the high-dimensional vector search library according to the request, obtains the corresponding response result and writes it into the shared memory for the service process to read. In an embodiment of the present invention, the access of the service process and the high-dimensional auxiliary process to the shared memory is synchronized through a lock.

[0080] Optionally, in an embodiment of the present invention, the above-mentioned request includes request types such as operation request and query request, and the above-mentioned operation request includes operation requests such as add, delete, modify, etc. of the client connection. The above-mentioned add, delete, modify operations can be insertion and deletion operations on the high-dimensional vector search library. Performing insertion and deletion operations at the same time can realize the modification of the high-dimensional vector search library.

[0081] Specifically, PostgreSQL is used as the target database and faiss is used as the high-dimensional vector search library. The above request types mainly include BUILD (create index), INSERT (insert a piece of data into the index), SEARCH (query through the index), REMOVE (delete a piece of data from the index), SAVE (save index), and DROP (delete index). BUILD, INSERT, SEARCH, and REMOVE correspond to the creation, insertion, query, and deletion operations in the faiss library, while SAVE and DROP correspond to saving index information to disk and deleting index information from disk in the PostgreSQL library.

[0082] Optionally, the index information of the target database is stored in a disk, and each time the high-dimensional auxiliary process is initialized, the index information is loaded into a shared memory, and the index information is used to record the query request corresponding to the service process; when the index information in the shared memory changes, the changed index information is sent to the disk for update. The index information may be an index table of the target database.

[0083] Specifically, in the PostgreSQL library, the index information includes index tables, user-created tables, and system tables. The above-mentioned index tables, user-created tables, and system tables all use a unified organization method. After updating the index tables, user-created tables, and system tables, these updates will be refreshed from the buffer to the disk at a specific time, and then loaded from the disk to the buffer when needed. In an embodiment of the present invention, the index is resident in the memory, specifically, it is resident in the shared memory. The Postgres service process communicates with the PGFaiss high-dimensional auxiliary process to complete the relevant functions of the index (search and find), so it is necessary to provide additional index saving and loading functions. In the process of initializing the PGFaiss high-dimensional auxiliary process each time the database is started, the index information on the disk is loaded into the memory, and then the access request of the Postgres high-dimensional auxiliary process is waited. Every time the index information changes, the change of the index information is updated to the index file on the disk.

[0084] In an embodiment of the present invention, the structured high-dimensional vectors are also stored in the disk corresponding to the target database. The high-dimensional auxiliary process integrates and encapsulates the search support of the high-dimensional vector search library and has the high-dimensional vector search capability of the high-dimensional vector search library.

[0085] Optionally, in the embodiment of the present invention, the abnormal disconnection of the client connection and the request interruption may be processed to prevent the lock in the shared memory from being deadlocked.

[0086] It should be noted that each process in the above-mentioned target database provides a processing mechanism for various signals, and the high-dimensional auxiliary process newly added in the embodiment of the present invention also needs to process related signals.

[0087] Furthermore, in the embodiment of the present invention, the signals that the high-dimensional auxiliary process needs to process mainly include abnormal disconnection of the client connection and request interruption. Since the communication between the high-dimensional auxiliary process and the service process is mainly carried out through shared memory, and the synchronization of the shared memory is carried out through locks, in order to prevent the locks in the shared memory from deadlocking, the abnormal disconnection of the client connection and request interruption can be processed as the signal processing mechanism of the high-dimensional auxiliary process.

[0088] Specifically, the signals that the PGFaiss high-dimensional auxiliary process needs to handle mainly include abnormal disconnection of client connections and request interruption. Since the communication between the PGFaiss high-dimensional auxiliary process and the Postgres service process is mainly through shared memory, and the synchronization of shared memory is performed through locks, in order to prevent the locks in the shared memory from deadlocking, abnormal disconnection of client connections and request interruption can be handled.

[0089] 302. The service process reads the response result in the shared memory and returns it to the client.

[0090] In the embodiment of the present invention, after the high-dimensional auxiliary process returns the response result to the shared memory according to the request, the service process reads the response result in the shared memory and returns the response result to the client.

[0091] In an embodiment of the present invention, a high-dimensional auxiliary process integrated and encapsulated with a high-dimensional vector search library can communicate with a target database through shared memory. When a high-dimensional vector search is required, the search is performed through the high-dimensional auxiliary process. Since the high-dimensional auxiliary process is integrated and encapsulated, the search capability of the high-dimensional vector search library for high-dimensional vectors can be fully utilized to improve the search speed for high-dimensional vectors in the target database, and low-dimensional vector searches or high-dimensional vector searches can be flexibly performed. By expanding the search function of the target database for high-dimensional vectors, the original database query function (mainly scalar data) of the target database and the high-dimensional vector search function are combined, thereby improving the vector search efficiency and reducing the development cost.

[0092] Optional, see Figure 4 , Figure 4 FIG. 1 is a flow chart of another method for searching a high-dimensional vector of a target database provided by an embodiment of the present invention. Figure 4 As shown, in Figure 3 Before step 301 of the embodiment, the method further includes:

[0093] 401. The service process sends a request to the shared memory through a preset index extension.

[0094] In the embodiment of the present invention, the above-mentioned index extension may be an index interface provided by the target database. Furthermore, most functions of the above-mentioned index extension are completed by sending requests to the high-dimensional auxiliary process.

[0095] Optionally, the above-mentioned preset index extension includes parameters, access methods and operator classes, wherein the above-mentioned parameters are used to support the index creation process and index search process of the above-mentioned index extension, different access methods correspond to different response results, and the above-mentioned operator class is used to identify the specific data type corresponding to the above-mentioned access method.

[0096] Optionally, the access method includes at least one of index creation, index insertion, index deletion and index query. In an embodiment of the present invention, the access method of the client can be obtained through the service process; the request corresponding to the access method is encapsulated to obtain the request; and the request is sent to the shared memory.

[0097] Specifically, the parameters of the pg_faiss index extension are mainly related parameters needed in the index creation and index search process. The parameters of the pg_faiss index extension mainly include vector dimension, index type and index parameters (faiss library).

[0098] More specifically, the PostgreSQL library provides the relevant access methods required for index extension, including index creation (ambuild), index insertion (aminsert), index deletion (ambulkdelete, amvacuumcleanup), index query (amgettuple, amgetbitmap), etc. The access method implemented in the pg_faiss index extension mainly encapsulates the requests corresponding to the access methods such as index creation, index insertion, index deletion, and index query into a unified format, then sends the corresponding requests to the shared memory, and finally obtains the response result of the PGFaiss high-dimensional auxiliary process from the shared memory and returns it to complete the function of the corresponding access method.

[0099] Furthermore, in the PostgreSQL library, the access method in the index extension does not directly understand the data type it will operate on, but an operator class is used to identify the specific data type that the access method operates on. Through the operator class, the support functions required for the internal operation of the access method in the index extension can be specified. In an embodiment of the present invention, the pg_faiss index extension defines an operator <@> of type float4[] for operating raw vectors and an operator <=> of type bytea for operating high-dimensional vectors. By calling the corresponding operator on float4[] / bytea type data, the pg_faiss index can be used to complete the fast query of high-dimensional vectors and improve the search speed.

[0100] 402. The service process accesses the shared memory in a scheduled or real-time manner through a preset index extension to read the state of the shared memory.

[0101] In an embodiment of the present invention, after the service process sends a corresponding request to the shared memory through the index extension, it waits for the response result returned by the high-dimensional auxiliary process. During this process, the shared memory can be accessed in a scheduled or real-time manner through the preset index extension to read the status of the shared memory.

[0102] 403. Determine whether there is a response result corresponding to the request.

[0103] In the embodiment of the present invention, after detecting that there is a corresponding response result in the shared memory, the service process immediately reads out the response result through index extension and returns it to the client.

[0104] 404. The high-dimensional auxiliary process accesses the shared memory periodically or in real time, reads the state of the shared memory, and determines whether there is a request corresponding to the service process.

[0105] In an embodiment of the present invention, the high-dimensional auxiliary process detects that there is a request corresponding to the service process in the shared memory, and then reads the above request for processing. The above request includes a query request or an operation request. The above query request corresponds to the indexing and searching of the data instance, and the above operation request corresponds to the operation request of adding, deleting, modifying, etc. of the high-dimensional vector search library.

[0106] It should be noted that the access of the service process and the high-dimensional auxiliary process to the shared memory may be synchronized through a lock.

[0107] Optionally, the request includes an operation request for the high-dimensional vector search library, and the above-mentioned response result includes the operation result of the above-mentioned high-dimensional vector search library. In an embodiment of the present invention, when the high-dimensional auxiliary process reads the operation request corresponding to the service process in the shared memory, the above-mentioned high-dimensional vector search library is operated according to the above-mentioned operation request, and the corresponding operation result is sent to the above-mentioned shared memory; the above-mentioned operation result in the above-mentioned shared memory is read by the above-mentioned service process and returned to the above-mentioned client.

[0108] It should be noted that the high-dimensional vector search method for a target database provided in the embodiment of the present invention can be applied to devices such as smart phones, computers, servers, etc. that can perform data search.

[0109] Optional, see Figure 5 , Figure 5 is a schematic diagram of the structure of a high-dimensional vector search device for a target database provided by an embodiment of the present invention. Figure 5 As shown, the device comprises:

[0110] The reading module 501 is used for returning a response result corresponding to the request and sending it to the shared memory when the high-dimensional auxiliary process reads a request corresponding to the service process in the shared memory;

[0111] A return module 502 is used to read the response result in the shared memory through the service process and return it to the client;

[0112] The high-dimensional auxiliary process integrates and encapsulates a high-dimensional vector search library, the service process is used to process requests between the target database and the client, and the target database only supports the search of low-dimensional vectors.

[0113] Optional, such as Figure 6 As shown, the device also includes:

[0114] The sending module 503 is used for the service process to send the request to the shared memory through a preset index extension, and to access the shared memory in a timed or real-time manner through the preset index extension, read the state of the shared memory, and determine whether there is a response result corresponding to the request;

[0115] A determination module 504 is used for the high-dimensional auxiliary process to access the shared memory periodically or in real time, read the state of the shared memory, and determine whether there is the request corresponding to the service process;

[0116] The access of the service process and the high-dimensional auxiliary process to the shared memory is synchronized through a lock.

[0117] Optional, such as Figure 7As shown, the index table of the target database is stored in a disk, and the device further includes:

[0118] A loading module 505, configured to load the index table into the shared memory each time the high-dimensional auxiliary process is initialized, wherein the index table is used to record the query request corresponding to the service process;

[0119] The updating module 506 is used for sending the changed index table to the disk for updating when the index table in the shared memory changes.

[0120] Optionally, the request includes an operation request for the high-dimensional vector search library, the response result includes an operation result for the high-dimensional vector search library, and the reading module 501 is further used to operate the high-dimensional vector search library according to the operation request when the high-dimensional auxiliary process reads the operation request corresponding to the service process in the shared memory, and send the corresponding operation result to the shared memory;

[0121] The return module 502 is further configured to read the operation result in the shared memory through the service process and return it to the client.

[0122] Optional, such as Figure 8 As shown in , the device also includes:

[0123] The signal processing module 507 is used to prevent the lock in the shared memory from deadlocking by processing the abnormal disconnection of the client connection and the request interruption.

[0124] Optionally, the preset index extension includes parameters, access methods and operator classes, wherein the parameters are used to support the index creation process and index search process of the index extension, different access methods correspond to different response results, and the operator class is used to identify the specific data type corresponding to the access method.

[0125] Optional, such as Fig. 9 As shown, the access method includes at least one of index creation, index insertion, index deletion and index query, and the sending module 503 includes:

[0126] An acquisition unit 5031 is used to acquire the access method of the client through the service process;

[0127] The encapsulation unit 5032 is used to encapsulate the request corresponding to the access method to obtain the request;

[0128] The sending unit 5033 is used to send the request to the shared memory.

[0129] It should be noted that the high-dimensional vector search device for a target database provided in the embodiment of the present invention can be applied to devices such as smart phones, computers, servers, etc. that can perform data search.

[0130] The high-dimensional vector search device for a target database provided by the embodiment of the present invention can implement each process implemented by the high-dimensional vector search method for a target database in the above method embodiment, and can achieve the same beneficial effects. To avoid repetition, it will not be described here.

[0131] See also Fig.10 , Fig.10 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention, such as Fig.10 As shown, it includes: a memory 1002, a processor 1001, and a computer program of a high-dimensional vector search method for a target database stored in the memory 1002 and executable on the processor 1001, wherein:

[0132] The processor 1001 is used to call the computer program stored in the memory 1002 and execute the following steps:

[0133] When the high-dimensional auxiliary process reads the request corresponding to the service process in the shared memory, it returns a response result corresponding to the request and sends it to the shared memory;

[0134] The service process reads the response result in the shared memory and returns it to the client;

[0135] The high-dimensional auxiliary process integrates and encapsulates a high-dimensional vector search library, the service process is used to process requests between the target database and the client, and the target database only supports the search of low-dimensional vectors.

[0136] Optionally, before the step of returning a response result corresponding to the request and sending it to the shared memory when the high-dimensional auxiliary process reads the request corresponding to the service process in the shared memory, the processor 1001 executes the method further comprising:

[0137] The service process sends the request to the shared memory through a preset index extension, accesses the shared memory in a timed or real-time manner through the preset index extension, reads the state of the shared memory, and determines whether there is a response result corresponding to the request;

[0138] The high-dimensional auxiliary process accesses the shared memory periodically or in real time, reads the state of the shared memory, and determines whether there is the request corresponding to the service process;

[0139] The access of the service process and the high-dimensional auxiliary process to the shared memory is synchronized through a lock.

[0140] Optionally, the index table of the target database is stored in a disk, and the method executed by the processor 1001 further includes:

[0141] Each time the high-dimensional auxiliary process is initialized, the index table is loaded into the shared memory, and the index table is used to record the query request corresponding to the service process;

[0142] When the index table in the shared memory changes, the changed index table is sent to the disk for updating.

[0143] Optionally, the request includes an operation request for the high-dimensional vector search library, the response result includes an operation result of the high-dimensional vector search library, and before the step of updating the row when the index table in the shared memory changes and sending the changed index table to the disk, the method executed by the processor 1001 also includes:

[0144] When the high-dimensional auxiliary process reads the operation request corresponding to the service process in the shared memory, it operates the high-dimensional vector search library according to the operation request and sends the corresponding operation result to the shared memory;

[0145] The service process reads the operation result in the shared memory and returns it to the client.

[0146] Optionally, the method executed by the processor 1001 further includes:

[0147] By processing the abnormal disconnection of the client connection and the request interruption, deadlock of the lock in the shared memory is prevented.

[0148] Optionally, the preset index extension includes parameters, access methods and operator classes, wherein the parameters are used to support the index creation process and index search process of the index extension, different access methods correspond to different response results, and the operator class is used to identify the specific data type corresponding to the access method.

[0149] Optionally, the access method executed by the processor 1001 includes at least one of index creation, index insertion, index deletion, and index query, and the service process sends the request to the shared memory through a preset index extension, including:

[0150] Obtaining the access method of the client through the service process;

[0151] Encapsulating the request corresponding to the access method to obtain the request;

[0152] The request is sent to the shared memory.

[0153] In a second aspect, an embodiment of the present invention provides a high-dimensional vector search device for a target database, the device comprising:

[0154] The reading module is used to return a response result corresponding to the request and send it to the shared memory when the high-dimensional auxiliary process reads the request corresponding to the service process in the shared memory;

[0155] A return module, used for reading the response result in the shared memory through the service process and returning it to the client;

[0156] The high-dimensional auxiliary process integrates and encapsulates a high-dimensional vector search library, and the service process is used to process requests between the target database and the client.

[0157] It should be noted that the electronic device provided in the embodiment of the present invention can be applied to devices such as smart phones, computers, servers, etc. that can perform data search.

[0158] The electronic device provided by the embodiment of the present invention can implement each process implemented by the high-dimensional vector search method of the target database in the above method embodiment, and can achieve the same beneficial effect. To avoid repetition, it will not be described here.

[0159] An embodiment of the present invention also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the computer program implements the various processes of the high-dimensional vector search method for the target database provided by the embodiment of the present invention or the high-dimensional vector search method for the application-side target database, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0160] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM).

[0161] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.

Claims

1. A high-dimensional vector search method for a target database, characterized in that: The following steps are involved: The service process sends a request to the shared memory through a preset index extension, and accesses the shared memory in a scheduled or real-time manner through the preset index extension, reads the state of the shared memory, and determines whether there is a response result corresponding to the request in the shared memory; The state of the shared memory is whether the index table in the shared memory is updated, and the index table in the shared memory records the request and response results in a written form; The index extension is an index interface provided by the target database. The preset index extension includes parameters, access methods and operator classes. The parameters are used to support the index creation process and index search process of the index extension. Different access methods correspond to different response results. The operator class is used to identify the specific data type corresponding to the access method. The high-dimensional auxiliary process accesses the shared memory periodically or in real time, reads the state of the shared memory, and determines whether the request corresponding to the service process exists in the shared memory; Wherein, the access of the service process and the high-dimensional auxiliary process to the shared memory is synchronized through a lock; When the high-dimensional auxiliary process reads the request corresponding to the service process in the shared memory, it returns a response result corresponding to the request and sends it to the shared memory; Reading the response result in the shared memory through the service process and returning it to the client; The high-dimensional auxiliary process integrates and encapsulates a high-dimensional vector search library, the service process is used to process requests between the target database and the client, and the target database only supports the search of low-dimensional vectors.

2. The method according to claim 1, characterized in that The index table of the target database is stored in a disk, and the method further comprises: Each time the high-dimensional auxiliary process is initialized, the index table is loaded into the shared memory, and the index table is used to record the request corresponding to the service process; When the index table in the shared memory changes, the changed index table is sent to the disk for updating.

3. The method according to claim 2, characterized in that The request includes an operation request for the high-dimensional vector search library, the response result includes an operation result of the high-dimensional vector search library, and before the step of sending the changed index table to the disk for updating when the index table in the shared memory changes, the method further includes: When the high-dimensional auxiliary process reads the operation request corresponding to the service process in the shared memory, it operates the high-dimensional vector search library according to the operation request and sends the corresponding operation result to the shared memory; The service process reads the operation result in the shared memory and returns it to the client.

4. The method according to claim 1, characterized in that The method further comprises: By processing the abnormal disconnection of the client connection and the request interruption, deadlock of the lock in the shared memory is prevented.

5. The method according to claim 1, characterized in that The access method includes at least one of index creation, index insertion, index deletion and index query, and the service process sends the request to the shared memory through a preset index extension, including: Obtaining the access method of the client through the service process; Encapsulating the request corresponding to the access method to obtain the request; The request is sent to the shared memory.

6. A high-dimensional vector search device for a target database, characterized in that: The device comprises: A sending module, in which the service process sends a request to the shared memory through a preset index extension, and accesses the shared memory in a scheduled or real-time manner through the preset index extension, reads the state of the shared memory, and determines whether there is a response result corresponding to the request in the shared memory; the state of the shared memory is whether the index table in the shared memory is updated, and the request and response results are recorded in the index table in the form of writing; the index extension is an index interface provided by the target database, and the preset index extension includes parameters, access methods, and operator classes, the parameters are used to support the index creation process and index search process of the index extension, different access methods correspond to different response results, and the operator class is used to identify the specific data type corresponding to the access method; A judgment module is used for the high-dimensional auxiliary process to access the shared memory in a scheduled or real-time manner, read the state of the shared memory, and judge whether the request corresponding to the service process exists in the shared memory; wherein the access of the service process and the high-dimensional auxiliary process to the shared memory is synchronized through a lock; The reading module is used to return a response result corresponding to the request and send it to the shared memory when the high-dimensional auxiliary process reads the request corresponding to the service process in the shared memory; A return module, used for reading the response result in the shared memory through the service process and returning it to the client; The high-dimensional auxiliary process integrates and encapsulates a high-dimensional vector search library, the service process is used to process requests between the target database and the client, and the target database only supports the search of low-dimensional vectors.

7. A high-dimensional vector search system for a target database, characterized in that: The system includes: a high-dimensional auxiliary process, a service process, a shared memory, and a client, wherein the high-dimensional auxiliary process integrates and encapsulates a high-dimensional vector search library, the service process is used to process requests between the target database and the client, and the target database only supports low-dimensional vector searches; the service process sends the request to the shared memory through a preset index extension, and accesses the shared memory in a timed or real-time manner through the preset index extension, reads the state of the shared memory, and determines whether there is a response result corresponding to the request in the shared memory; the state of the shared memory is whether the index table in the shared memory is updated, and the index table in the shared memory is updated through The request and response results are recorded in a written form; the index extension is an index interface provided by the target database, and the preset index extension includes parameters, access methods and operator classes, the parameters are used to support the index creation process and index search process of the index extension, different access methods correspond to different response results, and the operator class is used to identify the specific data type corresponding to the access method; the high-dimensional auxiliary process accesses the shared memory periodically or in real time, reads the status of the shared memory, and determines whether the request corresponding to the service process exists in the shared memory; wherein the access of the service process and the high-dimensional auxiliary process to the shared memory is synchronized through a lock; The high-dimensional auxiliary process is used to return a response result corresponding to the request and send it to the shared memory when reading the request corresponding to the service process in the shared memory; The service process is used to read the response result in the shared memory and return it to the client; The shared memory is used to receive the response result sent by the high-dimensional auxiliary process, and is also used to receive the request corresponding to the service process; The client is used to send the request to the service process, and is also used to receive the response result sent by the service process.

8. An electronic device, characterized in that: include: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the high-dimensional vector search method for a target database as described in any one of claims 1 to 5 when executing the computer program.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the high-dimensional vector search method for a target database according to any one of claims 1 to 5 are implemented.

Images