Index construction method and device

By listening to database write operations and building indexes in real time, the problem of time-consuming index construction in the existing technology is solved, and the online real-time index construction of big data in the database is realized, which meets the user's real-time query needs.

CN114168589BActive Publication Date: 2024-11-22ALIBABA (CHINA) CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111262763.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-10-28
Publication Date
2024-11-22
Estimated Expiration
2041-10-28

AI Technical Summary

Technical Problem

The existing technology cannot realize the online real-time index construction of big data in the database, resulting in the long-term index construction and cannot meet the user's real-time online query needs.

Method used

By listening to the user's data writing operations, obtaining keywords and sending them to the index construction thread when the index construction conditions are met, the index construction thread is used to build the index of the written data, and online real-time index construction is realized.

Benefits of technology

It realizes rapid index construction for users to write data in real time, avoids the problem of time-consuming index construction and meets users' real-time online query needs.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114168589B_ABST
    Figure CN114168589B_ABST
Patent Text Reader

Abstract

The present specification provides an index building method and device, wherein the index building method includes: monitoring a user's data writing operation to determine the write data corresponding to the data writing operation; obtaining a keyword in the write data, and if the keyword meets an index creation condition, sending the keyword to the index building thread, wherein the index creation condition is a condition corresponding to the index building thread; based on the keyword, building an index of the write data through the index building thread.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an index construction method. Background Art

[0002] With the development of Internet technology, many organizations will store their own data information in databases to facilitate subsequent processing of the data information. For example, features of unstructured data such as pictures and videos will be extracted and vectorized, and indexes will be built for the vectorized unstructured data in the database for use in approximate queries such as product recommendations and image searches.

[0003] However, the database in the prior art can only index big data offline and cannot index online in real time, which results in a long time spent on index construction and cannot meet the user's real-time online query needs. Summary of the invention

[0004] In view of this, the embodiment of this specification provides an index construction method. This specification also relates to an index construction device, a computing device, a computer-readable storage medium, and a computer program to solve the technical defects existing in the prior art.

[0005] According to a first aspect of an embodiment of this specification, there is provided an index construction method, including:

[0006] Monitoring a user's data writing operation, and determining the writing data corresponding to the data writing operation;

[0007] Acquire a keyword in the written data, and if the keyword satisfies an index creation condition, send the keyword to the index building thread, wherein the index creation condition is a condition corresponding to the index building thread;

[0008] Based on the keyword, an index of the write data is constructed by the index construction thread.

[0009] According to a second aspect of an embodiment of this specification, there is provided an index construction device, including:

[0010] A monitoring module, configured to monitor a user's data writing operation and determine the writing data corresponding to the data writing operation;

[0011] A sending module is configured to obtain a keyword in the written data, and if the keyword satisfies an index creation condition, send the keyword to the index building thread, wherein the index creation condition is a condition corresponding to the index building thread;

[0012] A construction module is configured to construct an index of the write data based on the keyword through the index construction thread.

[0013] According to a third aspect of an embodiment of this specification, a computing device is provided, including:

[0014] Memory and processor;

[0015] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, any step of the index building method is implemented.

[0016] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the steps of any of the index building methods are implemented.

[0017] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, the computer is caused to execute any step of the index building method.

[0018] The index building method provided in this specification includes monitoring the user's data writing operation, determining the write data corresponding to the data writing operation; obtaining keywords in the write data, and sending the keywords to the index building thread when the keywords meet the index creation conditions, wherein the index creation conditions are conditions corresponding to the index building thread; based on the keywords, building an index of the write data through the index building thread.

[0019] Specifically, the index building method monitors the user's data writing operation in real time, so that when the keyword in the written data corresponding to the data writing operation meets the index creation condition corresponding to the index building thread, based on the keyword, the index of the written data is built by the index building thread, thereby realizing online real-time index building and achieving the purpose of quickly building an index for the data written by the user in real time through the index building thread, avoiding the problem of time-consuming index building for big data, and meeting the needs of subsequent users for real-time online query based on the index. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a processing schematic diagram based on the Milvus solution provided in an embodiment of this specification;

[0021] Figure 2 is a flowchart of an index construction method provided in an embodiment of this specification;

[0022] Figure 3 It is a processing flow chart of an index construction method applied to a vector approximate query scenario provided in an embodiment of this specification;

[0023] Figure 4 It is a structural diagram of an index building device provided in an embodiment of this specification;

[0024] Figure 5 It is a structural block diagram of a computing device provided in one embodiment of this specification. DETAILED DESCRIPTION

[0025] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0026] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0027] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0028] First, the terms involved in one or more embodiments of this specification are explained.

[0029] Vector: An array of length X (x ≥ 1), usually the output of an AI (artificial intelligence) model.

[0030] Approximate search: Find the N (x ≥ 1) vectors closest to the input vector. It is commonly used in scenarios such as product recommendation and image search.

[0031] Index construction: To quickly query vector data, it is necessary to construct an index for the vector in advance as the basis for subsequent approximate queries.

[0032] Tair: A Key / Value structured data storage system.

[0033] Approximate search algorithm library: A library that can implement approximate search algorithms. For example, a database that implements approximate search algorithms such as HNSW and PQ.

[0034] Rocksdb: is an embeddable, persistent Key-Value store.

[0035] With the development of Internet technology, most databases are almost used to store structured data, such as employee information, financial information, and other data that can be clearly represented by multiple columns; for unstructured data, such as pictures, videos, etc., usually only the corresponding path can be stored in the database or the information can be binarized and stored in the database as key-value pairs; but because the database cannot recognize unstructured data, its retrieval ability is also limited. In the field of AI, unstructured data is usually feature extracted and vectorized, and the output is an X-dimensional vector; the index built for the vector will be used for approximate queries, such as product recommendations, image search, etc. In other words, most databases can only provide storage and retrieval of structured data, and cannot provide approximate retrieval of vectors. At the same time, the general vector approximate retrieval library is only suitable for offline index construction of big data, and cannot be applied to online real-time index construction. Therefore, the index construction takes a long time and cannot meet the requirements of real-time online users.

[0036] Based on this, this specification provides a solution based on Milvus (a high-performance vector search engine) to solve the above problems. However, Milvu's algorithm library uses Faiss (an approximate retrieval algorithm library), the storage layer uses Rocksdb, and the essence is to integrate multiple open source components together, and it is not real-time online. At the same time, Milvus is an open source and more mainstream vector database, and its functions are as shown in Figure 1 As shown in the "Offline Services" section, Figure 1 This is a processing diagram based on the Milvus solution provided in an embodiment of this specification; wherein Milvus is implemented based on a computing layer, a storage layer, and an algorithm library. The offline service can be understood as the construction of an index for big data in an offline manner, the storage layer is used to store big data, the algorithm library is used to construct an index for big data, and the computing layer is used to support the computing resources required for index construction.

[0037] Specifically, users can insert data into Milvus, which constructs a similarity index for the data inserted by the user, so that subsequent users can perform approximate queries (retrieval) on the inserted data through the similarity index; the index construction process can be: obtain data → train data (build index based on data) → index (complete the constructed index).

[0038] However, Milvus can only be used for offline training and index building. If real-time user queries are required, an independent system (third party) is required to provide approximate query services, such as Figure 1 As shown in the dotted box below, in order to enable users' online approximate queries, Milvus imports the constructed approximate index into a third-party online service in an offline manner, and implements users' online approximate queries through the online service.

[0039] At the same time, Milvus cannot achieve plug-and-check. The process of building an approximate index requires obtaining specified data from the storage layer (such as database tables, etc.), and then sending them to the search engine (algorithm library) in batches to build the index. For large amounts of data, the index construction process usually takes 2-5 hours due to different algorithm libraries. At this time, the user side is blocked and cannot perform operations such as queries.

[0040] Based on this, an index construction method based on a KV database is provided in an embodiment of this specification, which supports real-time storage, real-time index construction, and fast approximate retrieval of vectors for AI scenarios.

[0041] Specifically, in this specification, an index construction method is provided. This specification also relates to an index construction device, a computing device, a computer-readable storage medium, and a computer program, which are described in detail one by one in the following embodiments.

[0042] See also Figure 2 , Figure 2 A flowchart of an index construction method provided according to an embodiment of this specification is shown, which specifically includes the following steps:

[0043] Step 202: Monitor the data writing operation of the user, and determine the writing data corresponding to the data writing operation.

[0044] In practical applications, the index construction method provided in this specification can be applied to a database to quickly construct an index for data written in real time in the database to meet the user's online real-time query needs. The database, the data written in real time, and the index can all be set according to the actual application scenario. This specification does not make specific limitations on this. For example, the database can be a KV database, specifically Tair; the data written in real time can be a vector, a KV key-value pair (Key-Value), etc. The constructed index includes but is not limited to an index for implementing approximate retrieval, a primary key index, a unique index, etc.

[0045] The user can be understood as a user who writes data to the database, and the data write operation can be understood as the operation of the user writing data to the database in real time. In actual applications, the data write operation can be implemented through native commands in the database. For example, when the database is Tair, the commands to implement user insertion can be lpush, rpush, etc.; wherein, lpush is a Tair native command used to save array type data, the first parameter is key, and the subsequent parameters are the elements under key; for example, lpush key1 1.0.2.8.3.3, wherein key1 is the first parameter of lpush, and 1.0.2.8.3.3 is the second parameter of lpush. Writing data can be understood as the data written by the user to the database, and the type of the written data can be set according to the actual application scenario, such as vector type data, KV key-value pair type data.

[0046] Specifically, the database can monitor the user's data writing operation in real time. When the user performs the data writing operation, the database can obtain the writing data corresponding to the data writing operation.

[0047] In actual applications, the data written by users to the database in real time will be stored in the database's storage layer (disk); at the same time, when the user performs a data writing operation, the database will generate an event corresponding to the operation. The event contains the command to implement the data writing operation and the key in the data written by the user in real time.

[0048] Based on this, obtaining the write data corresponding to the data write operation can be understood as obtaining the write data corresponding to the data write operation from the storage layer of the database, or obtaining the key in the write data corresponding to the data write operation from the event corresponding to the data write operation.

[0049] Furthermore, before the database monitors the user's data write operation, it is also necessary to create an index building thread to facilitate the subsequent rapid construction of the index based on the thread to meet the user's online query needs. The specific implementation method is as follows.

[0050] Before monitoring the data writing operation of the user and determining the writing data corresponding to the data writing operation, the method further includes:

[0051] Create an index building thread based on the index creation conditions.

[0052] Among them, the index creation condition can be understood as the condition for creating an index building thread. In actual applications, the index creation condition is the condition carried in the index creation request received by the database. In the case where the database is Tair, the index building thread can be the index building method provided in this specification, and a command for creating an index building thread in Tair is provided, which is tai.watch; wherein tai.watch has 3 parameters, the first parameter is the name of the index to be created, the second parameter is used to tell Tair to listen to the List data structure (such as a key-value pair) of the matching feature, and the third parameter tells Tair what the vector dimension is. For example, the tai.watch command is tai.watch table key*3. Among them, table is the first parameter, key* is the second parameter, and 3 is the third parameter. Among them, the above three parameters are the index creation conditions.

[0053] Specifically, after the database obtains the index creation conditions, it creates an index building thread based on the index creation conditions, so as to achieve rapid index building based on the index building thread in the future to meet the user's online query needs.

[0054] In specific implementation, the index construction method provided in this specification can determine the corresponding index construction object (i.e., the algorithm that can implement index construction) for the multiplication after completing the creation of the index construction thread; thereby enabling the subsequent index construction thread to call the corresponding index construction object to build the index. The specific implementation method is as follows.

[0055] After creating the index building thread based on the index creation condition, the method further includes:

[0056] receiving an object association instruction, wherein the object association instruction carries an object identifier of a target index building object;

[0057] The target index building object is determined based on the object identifier, and a corresponding relationship between the target index building object and the index building thread is established.

[0058] Among them, the object association instruction can be understood as an instruction that can instruct the database to establish an association relationship between the index building thread and the index building object, the target index building object can be understood as an object that needs to be associated with the index building thread, and the object identifier can be understood as an identifier that uniquely represents an index building object, such as the number, serial number, name, etc. of the index building object.

[0059] Using the above example, the index building object may be an index building algorithm, and the object identifier may be understood as the algorithm identifier of the index building algorithm.

[0060] After Tair completes the creation of the index building thread based on the tai.watch command, it can also receive an instruction to determine the corresponding index building algorithm for the index building thread. The instruction carries the identifier of the algorithm that establishes an object relationship with the thread; Tair determines the index building algorithm corresponding to the algorithm identifier from the multiple types of index building algorithms configured by itself based on the algorithm identifier, and establishes a corresponding relationship between the index building algorithm and the index building thread, and subsequently quickly and accurately determines the index building algorithm corresponding to the index building thread based on the corresponding relationship.

[0061] Furthermore, the database monitors the user's data writing operation, which can be achieved through data monitoring detection in the database, so as to accurately and quickly perceive the user's data writing operation, which is convenient for the subsequent rapid implementation of real-time index construction. The specific method is as follows.

[0062] The monitoring of the data writing operation of the user and determining the writing data corresponding to the data writing operation includes:

[0063] Monitor the user's data writing operation according to the data monitoring object;

[0064] When the data writing operation is monitored, the writing data corresponding to the data writing operation is obtained.

[0065] The data monitoring object can be understood as a mechanism in the database that can monitor user operations, such as a monitoring thread in the database, a script in the database that can implement a monitoring function, and the like.

[0066] For example, taking the application of the index construction method to index construction according to Tair as an example, the data monitoring object obtains the written data corresponding to the data write operation when monitoring the user's data write operation. Among them, the index can be understood as an index that can realize approximate retrieval, and the data monitoring mechanism can be a monitoring thread in Tair. Specifically, Tair is configured with a monitoring thread that can monitor the user's data write operation. The monitoring process of the thread can be that when the user writes data to Tair through a data write operation, Tair can generate an event for the operation; the monitoring mechanism in Tair can perceive the event generated by Tair in real time, and judge whether the event is an event corresponding to the data write operation performed by the user according to the event identifier, the command contained in the event and other factors. If so, the event corresponding to the data write operation performed by the perceived user is determined, so as to obtain the write data corresponding to the data write operation performed by the user based on the event. Among them, in the case where the event contains write data, the write data can be directly obtained from the event; in addition, when the listening thread monitors the user's data write operation, the write data corresponding to the data write operation can also be obtained from the storage layer of Tair.

[0067] In the embodiment of this specification, the user's data writing operation is monitored through the data monitoring object configured in the database, and when the data writing operation is monitored, the writing data corresponding to the data writing operation is obtained, so as to facilitate the subsequent rapid construction of the index based on the writing data.

[0068] Step 204: Obtain a keyword in the write data, and if the keyword satisfies an index creation condition, send the keyword to the index building thread, wherein the index creation condition is a condition corresponding to the index building thread.

[0069] Among them, the keyword can be understood as the data that can be indexed in the user's written data. For example, when the written data is a KV key-value pair, the keyword can be key. The index construction thread can be understood as a thread in the database used to perform real-time index construction, and can also be understood as a task in the database used to perform real-time index construction. Based on the index construction thread, fast index construction can be achieved, thereby realizing the user's need for real-time online query. The index creation condition can be understood as the second parameter (key*) in the tai.watch command, and can also be set according to the actual application scenario. This manual does not make specific restrictions on this.

[0070] Continuing with the above example, after Tair completes the creation of the index building thread based on receiving the tai.watch command, it monitors the events corresponding to the data write operation through the listening thread and obtains the key contained in the event. When the key satisfies the second parameter key* carried in the tai.watch command, the key in the event is sent to the index building thread to facilitate the subsequent index building based on the key.

[0071] Step 206: Based on the keyword, build an index of the write data through the index building thread.

[0072] Specifically, after sending the keyword to the index building thread, the index building thread uses the keyword to build an index for write data. This process can be implemented based on the index building object corresponding to the index building thread, as follows.

[0073] The step of constructing the index of the written data based on the keyword by the index construction thread includes steps 1 and 2:

[0074] Step 1: Determine the index building object corresponding to the index building thread.

[0075] The index building object may be understood as an algorithm configured in the database and capable of index building, such as the NSW algorithm, the HNSW algorithm, the PQ algorithm, and the like.

[0076] Specifically, after the database sends the keyword to the index building thread, the database can determine the index building object corresponding to the index building thread from the multiple types of index building objects configured by itself based on the correspondence between the index building thread and the index concept object, and subsequently complete the index construction based on the index building object.

[0077] In addition, after determining the index building object corresponding to the index building thread, the database predetermines the correspondence between the index building thread and the index building object, and the database also provides an index building object for the user to choose to index the data written into the database, as follows.

[0078] The determining the index building object corresponding to the index building thread includes:

[0079] Generate an object selection request based on the object identifier of the index construction object, and send the object selection request to the user;

[0080] Receiving an object determination instruction sent by the user, wherein the object determination instruction carries an object identifier of an index building object selected by the user;

[0081] An index building object corresponding to the index building thread is determined based on the object identifier.

[0082] The object identifier can be understood as the name, number, etc. of the index building object, and the object selection request can be understood as the database's request to the user to select the index building object required by the user from one or more index building objects configured by the user. The object determination instruction can be understood as the request sent by the user to the database after selecting the required index building object.

[0083] Specifically, after the user writes data to the database, the database can send an object selection request to the user requesting the user to select an index building object. After the user determines the required index building object based on the received object selection request, the user sends an object determination instruction carrying the object identifier of the selected index building object to the database; after receiving the object determination instruction, the database determines the index building object corresponding to the object identifier from the multiple types of index building objects configured by itself based on the object identifier carried in the instruction, establishes a corresponding relationship between the object and the index building thread, and builds an index for the data written by the user together with the subsequent index building objects corresponding to the index building thread.

[0084] Step 2: When the index building object is an index incremental building object, based on the keyword, the index of the written data is built through the index incremental building object.

[0085] In the case where the index building object is an index building algorithm, the index incremental building object can be understood as an algorithm that can support incremental building, such as the NSW algorithm and the HNSW algorithm.

[0086] Specifically, after sending the keyword to the index building thread, the database determines that the object is an index incremental building object based on the type of the index building object corresponding to the thread, and then builds an index for writing data based on the keyword based on the index incremental building object.

[0087] Further, the step of constructing the index of the written data based on the keyword by constructing the index incrementally includes:

[0088] Based on the keyword, the historical index is updated through the index incremental construction object to obtain an updated index corresponding to the written data, wherein the historical index is an index constructed by the index incremental construction object based on the historical keyword.

[0089] Among them, the historical index can be understood as the index constructed by the index incremental construction object for the historical written data. The historical written data can be understood as the data written by the user history to the database, for example, the KV key-value pairs written by the user history to the database; the corresponding historical keyword can be understood as the key in the KV key-value pair written historically; the updated index can be understood as the historical index after being updated based on the keyword.

[0090] Using the above example, the index incremental construction object is a construction algorithm that supports incremental construction. When Tair sends the key of the written data to the index construction thread and determines that the construction algorithm corresponding to the index construction thread supports incremental construction, based on the construction algorithm that supports incremental construction, the key in the KV key-value pair written in real time by the user is used to update the historical index built based on the key of the historically written KV key-value pair, thereby obtaining the index corresponding to the real-time written KV key-value pair.

[0091] Among them, updating the historical index through the key can be that after the index construction thread receives the key in the KV key-value pair written by the user in real time, it calls the construction algorithm corresponding to the thread that supports incremental construction to determine the historical index of the historical construction. The historical index can be an index of a tree structure, and the leaf nodes of the index of the tree structure are built based on the key in the KV key-value pair written by the user in history.

[0092] After determining the historical index of the tree structure, the construction algorithm that supports incremental construction uses the key as a leaf node and adds the leaf node to the historical index of the tree structure according to specific rules, thereby updating the historical index. The updated index makes it easier for subsequent users to perform online queries on the data written to Tair in real time based on the updated index, thereby improving query efficiency.

[0093] In the embodiments of this specification, only the historical index as a tree-structured index is taken as an example to further explain the index construction method. The structure of the historical index can be set according to the actual application scenario, and this specification does not make any specific restrictions on the structure of the historical index.

[0094] The index construction method provided in this specification updates the historical index by constructing an object through index increment based on keywords to obtain an updated index corresponding to the written data. This facilitates subsequent users to perform online queries on the written data in real time based on the updated index, thereby improving the query efficiency.

[0095] In addition, when the index building object is not an index incremental building object but an index non-incremental building object, the index non-incremental building object cannot support the incremental building of the index, so it is necessary to obtain the write data corresponding to the keyword and build a new index for the write data. The specific method is as follows.

[0096] After determining the index building object corresponding to the index building thread, the method further includes:

[0097] In the case where the index building object is an index non-incremental building object, obtaining write data corresponding to the keyword according to the index building thread;

[0098] Based on the written data, an index of the written data is constructed by using the index non-incremental construction object.

[0099] In the case where the index building object is an index building algorithm, the index non-incremental building object can be understood as a building algorithm that does not support incremental building, such as the PQ algorithm.

[0100] Specifically, when the database determines that the index building object is a non-incremental building object that does not support incremental building, the database cannot update the historical index based on the keyword. Based on this, the database obtains the write data corresponding to the keyword from the storage layer. For example, when the keyword is the key of a KV key-value pair, the write data corresponding to the keyword can be the KV key-value pair corresponding to the key, or the vector (value) corresponding to the key.

[0101] After obtaining the written data, the database builds a new index based on the written data through the index non-incremental construction object corresponding to the index construction thread, thereby obtaining the index corresponding to the written data, thereby facilitating users to perform online queries on the data written to the database based on the index.

[0102] Furthermore, since the index non-incremental construction object needs to build a new index for the written data, if the database creates an index for each data written by the user to the database, it may cause a waste of database computing resources and fail to achieve the goal of fast data query. Based on this, in the index construction method provided in this specification, the index construction thread will start construction after waiting for 1 second after receiving the keyword. If the user continues to add data to the database after 1 second, the construction is stopped and the waiting time is reset to wait for construction. When the waiting time reaches 1 second and the user does not add data to the database, the index construction thread obtains the KV key-value pair corresponding to the key from the storage layer, or the vector (value) corresponding to the key, and performs retrieval and construction based on the value and key. The specific method of obtaining the vector corresponding to the key from the storage layer is as follows.

[0103] The acquiring the write data corresponding to the keyword according to the index building thread includes:

[0104] S1: determining a target keyword according to the index building thread;

[0105] S2: recording the receiving time of the target keyword according to the index building thread, and determining a time interval between the receiving time and the current time;

[0106] S3: when the time interval satisfies a preset interval condition, determining the keyword corresponding to the time interval as the index building keyword according to the index building thread, and determining whether the index building thread receives the keyword again, if not, obtaining the write data corresponding to the index building keyword according to the index building thread;

[0107] If so, the keyword received again by the index building thread is determined as the target keyword, and step S2 is executed.

[0108] During the specific implementation process, after the keyword received by the index building thread again is determined as the target keyword and step S2 is executed, step S3 needs to be executed. Therefore, when the time interval meets the preset interval condition, the keyword corresponding to the time interval is determined as the index building keyword according to the index building thread, and it is continued to be determined whether the index building thread receives the keyword again.

[0109] The target keyword can be understood as a keyword used to instruct the index building thread to start timing, for example, the first key sent to the index building thread, or the first key received during the retiming process of the index building thread. In actual applications, when the index building thread receives the target keyword, the timing starts, and the index building thread uses the time when the target keyword is received as the starting time of the timing.

[0110] The time interval can be understood as a time range. For example, when the receiving time is 12:12:12.001 and the current time is 12:12:12.051, the time interval between the two can be 12:12:12.001-12:12:12.051; the preset interval condition can be set according to the actual application scenario. For example, the preset interval condition can be that the time interval is greater than 1s (1 second).

[0111] The index building keyword can be understood as a keyword that can be used to build an index.

[0112] Specifically, when the database determines the index building algorithm corresponding to the index building thread and it is an index non-incremental building object, in order to avoid wasting computing resources, after sending the keyword to the index building thread, when the index building thread determines that the received keyword is the target keyword, the database records the receiving time when the index building thread receives the target keyword, and starts timing from the receiving time, determining the time interval between the receiving time and the current time in real time.

[0113] The database determines that the time interval meets the preset interval condition when the time interval meets the preset interval condition. For example, the preset interval condition is that the time interval is greater than or equal to 1s; based on this, when the time interval is greater than or equal to 1s, it is determined that the time interval meets the preset interval condition, and the keyword corresponding to the time interval is determined as the index construction keyword.

[0114] After the database determines that the time interval meets the preset interval condition, it needs to determine again whether the user continues to write new data to the database. If so, it means that the user's data writing operation is still continuing and not all data has been written to the database. Therefore, the database uses the first keyword received by the index building thread after the time interval as the target keyword. For example, after determining that the user continues to write new data to the database, the database uses the keyword received by the index building thread after the time interval 12:12:12.001-12:12:13.001 as the target keyword and restarts the timing.

[0115] If not, the database obtains the write data corresponding to the index building keyword from the storage layer of the database through the index building thread, so that subsequent index non-incremental building objects can build an index of the write data through the write data.

[0116] Furthermore, after the database quickly builds an index for the written data, the user can use the index to perform fast real-time online queries, as shown below.

[0117] After building the index of the written data by the index building thread based on the keyword, the method further includes:

[0118] Receiving a data query request, wherein the data query request includes a target index identifier and a data query condition;

[0119] Determining a target index from the indexes according to the target index identifier;

[0120] Determine the write data corresponding to the data query condition according to the target index.

[0121] Among them, the data query request can be understood as a command for the user to query the data in the database, and the target index identifier can be understood as the identifier of the index that needs to be queried; the data query condition can be understood as the condition based on which the data query is performed, and the data query condition can be set according to the actual application scenario. In actual applications, when the database is Tair, the command for the user to query the data in the database can be tai.search; tai.search is a newly developed command provided by the index construction method in the embodiment of this specification, and its first parameter is the index name (i.e., the target index identifier), the second parameter indicates that the TOPN approximate vectors are expected to be obtained, and the following parameters are the requested vectors. The second parameter and the parameters after the second parameter can be data query conditions. For example, the tai.search command can be tai.search table 10 1 2 3; wherein table is the first parameter, the second parameter is 10, and the parameters after the second parameter are 1 2 3, wherein 1 2 3 are the vectors that the user needs to perform similar searches. After receiving the command, the database can retrieve 10 vectors similar to vector 123 from the index named table;

[0122] Using the above example, Tair quickly completes the real-time index construction of the written data through the index construction thread, and then receives the data query command tai.search table 10 1 2 3 sent by the user; based on the table carried in the command, it determines the index that the data query command needs to query, and based on 101 2 3 carried in the data query command, it searches the table index for the 10 vectors with the highest similarity to the parameters 1 2 3. This enables the user's real-time online query.

[0123] The index building method provided in the embodiments of the present specification monitors the user's data writing operation in real time, so that when the keyword in the written data corresponding to the data writing operation meets the index creation condition corresponding to the index building thread, the index of the written data is built according to the keyword according to the index building thread, thereby realizing online real-time index building. At the same time, the index building thread is used to quickly build an index for the data written by the user in real time, avoiding the problem of time-consuming index building for big data, thereby meeting the needs of subsequent users for real-time online query based on the index.

[0124] The following combination Figure 3 , taking the application of the index construction method provided in this specification in the scenario of vector approximate query as an example, the index construction method is further described. Figure 3A processing flow chart of an index construction method for an approximate query scenario of a vector provided by an embodiment of the present specification is shown. The index construction method is applied to Tair, and Tair includes a storage layer, a computing layer, and an algorithm library, wherein Tair receives various commands sent by users and operation and maintenance personnel based on an API interface. Including tai.watch, lpush, tai.search, etc.; the storage layer is used to store the KV key-value pairs written by the user to Tair, and the KV key-value pairs include a vector (value) and a key corresponding to the vector; the computing layer is a computing module in Tair, which is used to provide computing resource support for the operation of Tair; the algorithm library is used to implement indexing of KV key-value pairs written to Tair in Tair. Among them, the index is an index that can realize approximate retrieval.

[0125] Specifically, before a user inserts data into Tair, Tair can receive the command tai.watch table key*3 sent by the operation and maintenance personnel. The operation and maintenance personnel can use this command to tell Tair that it is necessary to build an index for the data inserted (added) by the user, and the indexed data comes from the KV key-value pairs whose keys meet the "keys*" feature. Tair creates an index building thread based on this command, and monitors the user's insertion operations in real time through its own configured listening thread.

[0126] After the index building thread is created, the user inserts the data KV key-value pair into Tair through List commands such as lpush / rpush. Figure 3 When lpush key1 1.0.2.8.3.3 to lpush key4 6.6.4.9.6.1 are executed and the KV key-value pair is stored in the Tair storage layer, the Tair listening thread obtains the key contained in the event corresponding to the insertion operation, where the key is the key in the KV key-value pair inserted into Tair by the user through the command.

[0127] If the key of the KV key-value pair meets the feature key*, the key is added to the key* queue, and the key is sent to the index building thread based on the queue, so that the index building thread obtains the key corresponding to the KV key-value pair. Figure 3 , where multi-threaded consumption can be understood as multiple index building threads, jointly building an index, the key required by each index building thread can be different and corresponding, and the index built by each index building thread based on the key is also different. In actual applications, the key required by each index building thread and the index to be built can be defined based on the tai.watch command.

[0128] After obtaining the key, Tair's index building thread quickly completes index building in the background, making it easier for subsequent users to perform real-time online queries. Specifically, it includes:

[0129] When creating an index building thread, Tair can determine corresponding building algorithms for different index building threads from an algorithm library, wherein the algorithm library is configured with different types of building algorithms.

[0130] After Tair's index building thread obtains the key, Tair determines whether the construction algorithm corresponding to the index building thread supports incremental construction.

[0131] If the construction algorithm itself supports incremental construction, such as NSW and HNSW, then when the index construction thread captures the user's data insertion operation, it will immediately build the index based on the key in the KV key-value pair inserted by the user.

[0132] If the construction algorithm itself does not support incremental construction, such as PQ, Tair will instruct the index construction thread to wait for 1 second after receiving the key, and start index construction after 1 second. If the user continues to add data to Tair after 1 second, Tair will instruct the index construction thread to stop index construction and restart the timing. The index construction thread will start index construction when the timing reaches 1 second and the user stops adding data to Tair.

[0133] During construction, the index construction thread obtains the vector corresponding to the key from Tair's storage layer, and performs retrieval and construction based on the key and the vector corresponding to the key.

[0134] In actual applications, when Tair is building a real-time index in the background, users can still execute other commands, such as the "tai.stat" command provided by Tair, to view the current index building status. When the building is completed, users can search the index.

[0135] In the embodiments provided in this specification, only the case where the data inserted by the user is a KV key-value pair is used as an example to illustrate this specification. The data inserted by the user can be any type of data, and this specification does not impose specific restrictions on this, for example, data of any List data structure.

[0136] The index building method provided in this manual is based on the native operation of the database (Tair), uses monitoring technology to monitor user operations, and builds indexes in the background, so that users can build indexes without feeling it, and realize that users can automatically complete the construction by operating the native List data structure.

[0137] At the same time, by utilizing the background construction framework technology, the problem of user operation blocking in the case of big data construction is solved. In addition, in the process of background index construction, background asynchronous multi-threaded index construction is adopted to support users to insert and build indexes in real time, and support real-time online approximate retrieval.

[0138] Corresponding to the above method embodiment, this specification also provides an index construction device embodiment, Figure 4 FIG. 1 shows a schematic diagram of the structure of an index building device provided in an embodiment of the present specification. Figure 4 As shown, the device comprises:

[0139] A monitoring module 402 is configured to monitor a user's data writing operation and determine the writing data corresponding to the data writing operation;

[0140] The sending module 404 is configured to obtain a keyword in the written data, and if the keyword satisfies an index creation condition, send the keyword to the index building thread, wherein the index creation condition is a condition corresponding to the index building thread;

[0141] The construction module 406 is configured to construct an index of the write data based on the keyword through the index construction thread.

[0142] Optionally, the index building method further includes a creation module configured to create an index building thread based on the index creation condition.

[0143] Optionally, the index building method further includes an association module configured to:

[0144] receiving an object association instruction, wherein the object association instruction carries an object identifier of a target index building object;

[0145] The target index building object is determined based on the object identifier, and a corresponding relationship between the target index building object and the index building thread is established.

[0146] Optionally, the monitoring module 402 is further configured to:

[0147] Monitor the user's data writing operation according to the data monitoring object;

[0148] When the data writing operation is monitored, the writing data corresponding to the data writing operation is obtained.

[0149] Optionally, the construction module 406 is further configured to:

[0150] Determine the index building object corresponding to the index building thread;

[0151] In the case where the index building object is an index incremental building object, the index of the written data is built through the index incremental building object based on the keyword.

[0152] Optionally, the construction module 406 is further configured to:

[0153] Generate an object selection request based on the object identifier of the index construction object, and send the object selection request to the user;

[0154] Receiving an object determination instruction sent by the user, wherein the object determination instruction carries an object identifier of an index building object selected by the user;

[0155] An index building object corresponding to the index building thread is determined based on the object identifier.

[0156] Optionally, the construction module 406 is further configured to:

[0157] Based on the keyword, the historical index is updated through the index incremental construction object to obtain an updated index corresponding to the written data, wherein the historical index is an index constructed by the index incremental construction object based on the historical keyword.

[0158] Optionally, the construction module 406 is further configured to:

[0159] In the case where the index building object is an index non-incremental building object, obtaining write data corresponding to the keyword according to the index building thread;

[0160] Based on the written data, an index of the written data is constructed by using the index non-incremental construction object.

[0161] Optionally, the construction module 406 is further configured to:

[0162] S1: determining a target keyword according to the index building thread;

[0163] S2: recording the receiving time of the target keyword according to the index building thread, and determining a time interval between the receiving time and the current time;

[0164] S3: when the time interval satisfies a preset interval condition, determining the keyword corresponding to the time interval as the index building keyword according to the index building thread, and determining whether the index building thread receives the keyword again, if not, obtaining the write data corresponding to the index building keyword according to the index building thread;

[0165] If so, the keyword received again by the index building thread is determined as the target keyword, and step S2 is executed.

[0166] Optionally, the index building method further includes a query module configured to:

[0167] Receiving a data query request, wherein the data query request includes a target index identifier and a data query condition;

[0168] Determining a target index from the indexes according to the target index identifier;

[0169] Determine the write data corresponding to the data query condition according to the target index.

[0170] The index building device provided in the present specification monitors the user's data writing operation in real time, so that when the keyword in the written data corresponding to the data writing operation meets the index creation condition corresponding to the index building thread, the index of the written data is built based on the keyword through the index building thread, thereby realizing online real-time index building and achieving the purpose of quickly building an index for the data written by the user in real time through the index building thread, avoiding the problem of time-consuming index building for big data, and meeting the needs of subsequent users for real-time online query based on the index.

[0171] The above is a schematic scheme of an index building device of this embodiment. It should be noted that the technical scheme of the index building device and the technical scheme of the above index building method belong to the same concept, and the details not described in detail in the technical scheme of the index building device can be found in the description of the technical scheme of the above index building method.

[0172] Figure 5 The structure block diagram of a computing device 500 provided according to an embodiment of the present specification is shown. The components of the computing device 500 include but are not limited to a memory 510 and a processor 520. The processor 520 is connected to the memory 510 via a bus 530, and the database 550 is used to store data.

[0173] The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface card (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a World Wide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0174] In one embodiment of the present specification, the above components of the computing device 500 and Figure 5 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Figure 5 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0175] The computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or PC. The computing device 500 may also be a mobile or stationary server.

[0176] The processor 520 is used to execute computer executable instructions, which, when executed by the processor 520, implement any step of the index building method.

[0177] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the above index construction method belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the above index construction method.

[0178] An embodiment of the present specification further provides a computer-readable storage medium storing computer-executable instructions, which can implement any step of the index building method when executed by a processor.

[0179] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above index construction method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above index construction method.

[0180] An embodiment of the present specification further provides a computer program, wherein when the computer program is executed in a computer, the computer is caused to execute any step of the index building method.

[0181] The above is a schematic scheme of a computer program of this embodiment. It should be noted that the schematic scheme of the computer program and the technical scheme of the above-mentioned index construction method belong to the same concept, and the details not described in detail in the schematic scheme of the computer program can be referred to the description of the technical scheme of the above-mentioned index construction method.

[0182] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0183] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0184] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that this specification is not limited by the order of the actions described, because according to this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this specification.

[0185] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0186] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. An index construction method, applied to a database, include: Monitoring a user's data writing operation, and determining the writing data corresponding to the data writing operation; Acquire a keyword in the written data, and if the keyword satisfies an index creation condition, send the keyword to an index building thread, wherein the index creation condition is a condition corresponding to the index building thread; An index building object is determined through the index building thread, and an index of the written data is built through the index building object based on the keyword, wherein the index building object includes an index incremental building object or an index non-incremental building object, and the index of the written data is built through the index building object based on the keyword, including building an index of the written data through the index incremental building object based on the keyword, and the index of the written data is built through the index incremental building object based on the keyword, including updating a historical index through the index incremental building object to obtain an updated index corresponding to the written data, and the historical index is an index built by the index incremental building object based on the historical keyword.

2. The index building method according to claim 1, wherein before monitoring the user's data writing operation and determining the write data corresponding to the data writing operation, include: Create an index building thread based on the index creation conditions.

3. The index building method according to claim 2, after creating the index building thread based on the index building condition, further include: receiving an object association instruction, wherein the object association instruction carries an object identifier of a target index building object; The target index building object is determined based on the object identifier, and a corresponding relationship between the target index building object and the index building thread is established.

4. The index building method according to claim 1, wherein the monitoring of the user's data writing operation determines the writing data corresponding to the data writing operation, include: Monitor the user's data writing operation according to the data monitoring object; When the data writing operation is monitored, the writing data corresponding to the data writing operation is obtained.

5. The index building method according to claim 1, wherein the index building object is determined by the index building thread, include: An index building object corresponding to the index building thread is determined.

6. The index building method according to claim 5, wherein the determining the index building object corresponding to the index building thread, include: Generate an object selection request based on the object identifier of the index construction object, and send the object selection request to the user; Receiving an object determination instruction sent by the user, wherein the object determination instruction carries an object identifier of an index building object selected by the user; An index building object corresponding to the index building thread is determined based on the object identifier.

7. The index building method according to claim 5, after determining the index building object corresponding to the index building thread, further comprising: include: In the case where the index building object is an index non-incremental building object, obtaining write data corresponding to the keyword according to the index building thread; Based on the written data, an index of the written data is constructed by using the index non-incremental construction object.

8. The index building method according to claim 7, wherein the writing data corresponding to the keyword is obtained according to the index building thread, include: S1: determining a target keyword according to the index building thread; S2: recording the receiving time of the target keyword according to the index building thread, and determining a time interval between the receiving time and the current time; S3: when the time interval satisfies a preset interval condition, determining the keyword corresponding to the time interval as the index building keyword according to the index building thread, and determining whether the index building thread receives the keyword again, if not, obtaining the write data corresponding to the index building keyword according to the index building thread; If so, the keyword received again by the index building thread is determined as the target keyword, and step S2 is executed.

9. The index building method according to claim 1, after determining the index building object through the index building thread and building the index of the written data through the index building object based on the keyword, further include: Receiving a data query request, wherein the data query request includes a target index identifier and a data query condition; Determining a target index from the indexes according to the target index identifier; Determine the write data corresponding to the data query condition according to the target index.

10. An index building device, applied to a database, include: A monitoring module, configured to monitor a user's data writing operation and determine the writing data corresponding to the data writing operation; A sending module is configured to obtain a keyword in the written data, and if the keyword satisfies an index creation condition, send the keyword to an index building thread, wherein the index creation condition is a condition corresponding to the index building thread; A construction module is configured to determine an index construction object through the index construction thread, and to construct an index of the written data through the index construction object based on the keyword, wherein the index construction object includes an index incremental construction object or an index non-incremental construction object, and the construction of the index of the written data through the index construction object based on the keyword includes constructing the index of the written data through the index incremental construction object based on the keyword, and the construction of the index of the written data through the index incremental construction object based on the keyword includes updating the historical index through the index incremental construction object to obtain the updated index corresponding to the written data, and the historical index is the index constructed by the index incremental construction object based on the historical keyword.

11. A computing device, include: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the index construction method described in any one of claims 1 to 9 are implemented.

12. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the index building method described in any one of claims 1 to 9.