Index processing method and device, data query method and device and electronic equipment

Abstract

The invention provides an index processing method, a data query method and device and electronic equipment, and relates to the technical field of computers, in particular to the technical field of data processing. The specific implementation scheme of the index processing method comprises the steps that newly-added data in a processing request for an index database is responded, the newly-added data is written into a target storage block in a plurality of storage blocks, the index database comprises original data and composition data, storage addresses of the original data are continuous, and the composition data comprises composition nodes generated according to the original data; adding the target storage block to an index database, and generating a newly added node according to the newly added data; according to the composition data, determining neighbor nodes for the newly-added nodes, and obtaining at least one newly-added neighbor node associated with the newly-added data; according to the at least one newly added neighbor node, adding node information for the newly added node in the composition data; and updating node information of newly added neighbor nodes in the composition data by using atomic operation to obtain a processed index database.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, in particular to the field of data processing technology, and specifically to an index processing method, a data query method, a device, an electronic device, a storage medium, and a program product. Background Art

[0002] ANN (Approximate Nearest Neighbors) is to find one or more data that are most similar to the target data from a data set based on the similarity of the data. However, before using ANN retrieval technology, it is necessary to build a database of the retrieval candidate data to form an index library, and the retrieval candidate data is a closed set, making it difficult to update the retrieval candidate data in the index library. Summary of the invention

[0003] The present disclosure provides an index processing method, a data query method, a device, an electronic device, a storage medium, and a program product.

[0004] According to one aspect of the present disclosure, there is provided an index processing method, comprising: in response to newly added data in a processing request for an index library, writing the newly added data into a target storage block among a plurality of storage blocks, wherein the index library comprises original data and composition data, the storage addresses of the original data are continuous, and the composition data comprises composition nodes generated according to the original data; adding the target storage block to the index library, and generating an newly added node according to the newly added data; determining a neighbor node for the newly added node according to the composition data, and obtaining at least one newly added neighbor node associated with the newly added data; adding node information for the newly added node in the composition data according to the at least one newly added neighbor node; and updating the node information of the newly added neighbor node in the composition data using atomic operations, and obtaining a processed index library.

[0005] According to another aspect of the present disclosure, a data query method is provided, comprising: in response to a data query request, determining an index library according to the data to be queried in the query request; determining target data matching the data to be queried according to the index library; wherein the index library is constructed using the above-mentioned index processing method.

[0006] According to another aspect of the present disclosure, an index processing device is provided, including: a writing module, for writing the newly added data in a processing request for an index library into a target storage block among a plurality of storage blocks in response to the newly added data in the processing request, wherein the index library includes original data and composition data, the storage address of the original data is continuous, and the composition data includes composition nodes generated according to the original data; a generating module, for adding the target storage block to the index library, and generating a newly added node according to the newly added data; a first determining module, for determining a neighbor node for the newly added node according to the composition data, and obtaining at least one newly added neighbor node associated with the newly added data; an adding module, for adding node information for the newly added node in the composition data according to the at least one newly added neighbor node; and an updating module, for updating the node information of the newly added neighbor node in the composition data using atomic operations, and obtaining a processed index library.

[0007] According to another aspect of the present disclosure, a data query device is provided, including: an index library determination module, used to respond to a data query request and determine the index library according to the data to be queried in the query request; a query module, used to query target data matching the data to be queried according to the index library; wherein the index library is constructed using the above-mentioned index processing method.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the present disclosure.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method of the present disclosure.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and the computer program implements the method of the present disclosure when executed by a processor.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1A schematic diagram schematically shows an application scenario in which the index processing method and device according to an embodiment of the present disclosure can be applied;

[0014] Figure 2 A schematic diagram of the structure of composition data according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 3 A flowchart of an index processing method according to an embodiment of the present disclosure is schematically shown;

[0016] Figure 4 The schematic diagram of the structure of the resource pool according to the embodiment of the present disclosure is shown;

[0017] Figure 5 A schematic diagram of determining neighbor nodes according to an embodiment of the present disclosure is schematically shown;

[0018] Fig. 6A A schematic diagram of determining optimization information according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 6B The result of determining optimization information according to an embodiment of the present disclosure is schematically shown; FIG.

[0020] Fig. 7A A schematic diagram of a principle diagram for determining optimization sub-information according to an embodiment of the present disclosure is shown;

[0021] Figure 7B Schematically shows a principle diagram for determining optimized sub-information according to another embodiment of the present disclosure;

[0022] Figure 7C Schematically shows a principle diagram of determining optimized sub-information according to yet another embodiment of the present disclosure;

[0023] Fig.7D A schematic diagram showing a principle diagram for determining optimized sub-information according to another embodiment of the present disclosure is shown;

[0024] Figure 8 A schematic diagram of determining optimization information according to an embodiment of the present disclosure is schematically shown;

[0025] Fig.9A A schematic diagram of a topological composition according to an embodiment of the present disclosure is schematically shown;

[0026] Fig. 9B A schematic diagram schematically shows current node information according to an embodiment of the present disclosure;

[0027] Fig. 9C A schematic diagram schematically shows node information during an update process according to an embodiment of the present disclosure.

[0028] Fig.9DA schematic diagram schematically shows updated node information according to an embodiment of the present disclosure;

[0029] Fig.10 A flowchart of a data query method according to an embodiment of the present disclosure is schematically shown;

[0030] Fig.11 A block diagram of an index processing device according to an embodiment of the present disclosure is schematically shown;

[0031] Fig.12 A block diagram schematically shows a data query device according to an embodiment of the present disclosure; and

[0032] Fig.13 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0033] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0034] Figure 1 A schematic diagram of an application scenario to which the index processing method and device according to an embodiment of the present disclosure can be applied is schematically shown.

[0035] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the index processing method and apparatus can be applied may include a terminal device, but the terminal device may implement the index processing method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0036] like Figure 1 As shown, the application scenario 100 according to this embodiment may include a terminal device 101, a server 102, an index library 103 and a network. The network is used to provide a medium for a communication link between the terminal device 101, the server 102 and the index library 103. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0037] The user can generate a processing request for the index library 103 through the interactive interface provided by the terminal device 101, and send the processing request to the server 102 through the network, so that the server 102 processes the original data and composition data in the index library 103 according to the new data in the processing request to obtain a processed index library.

[0038] Various communication client applications may be installed on the terminal device 101, such as intelligent assistant applications, knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software (only as examples). Users may input new data generation processing requests in the interactive interfaces of these client applications.

[0039] The terminal device 101 can be configured with various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0040] Server 102 can be a server that provides various services, such as a background management server that provides support for the content browsed by the user using the interactive interface of the terminal device 101 (only as an example). The background management server can add the newly added data in the received processing request to the index library, add the node information of the newly added node in the composition data, and update the node information of the newly added neighboring nodes. Server 102 can also be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). Server 102 can also be a server of a distributed system, or a server combined with blockchain.

[0041] It should be noted that the index processing method provided in the embodiment of the present disclosure can generally be executed by the server 102. Accordingly, the index processing device provided in the embodiment of the present disclosure can also be set in the server 102. The index processing method provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 102 and can communicate with the terminal device 101 and / or the server 102. Accordingly, the index processing device provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 102 and can communicate with the terminal device 101 and / or the server 102.

[0042] Alternatively, the index processing method provided in the embodiment of the present disclosure may also be generally executed by the terminal device 101. Accordingly, the index processing apparatus provided in the embodiment of the present disclosure may generally be arranged in the terminal device 101.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, servers and index libraries in the embodiment is only for illustration. Any number of terminal devices, networks, servers and index libraries may be provided according to implementation requirements.

[0044] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0045] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0046] In the related art, the ANN retrieval algorithms used include graph-based indexing algorithms such as NSW (Navigable SmallWorld), HNSW (Hierarchical Navigable Small World), and PQ (Product Quantization)-based retrieval algorithms such as OPQ (Optimal Product Quantization). These ANN retrieval technologies need to build a database to form an index database for the candidate data to be retrieved before use, and the candidate data to be retrieved is a closed set. The candidate data set remains unchanged during the ANN retrieval process, and it is difficult to update the candidate data in the candidate data set.

[0047] In one example, a method for updating candidate data for an ANN index includes: the ANN index service includes two slots, a main slot and a backup slot, the main slot refers to the slot that is currently undergoing ANN retrieval; the backup slot refers to an empty slot; when updating data, the ANN index that is currently undergoing retrieval is first copied to the backup slot as a copy, and the ANN index on the backup slot is incrementally updated; after the incremental update of the ANN index on the backup slot is completed, the backup slot is converted to the main slot to provide ANN retrieval services, the main slot releases the old ANN index and is converted to the backup slot, and the index update is completed. However, this update method requires copying the full index data when performing incremental data updates, which doubles the memory overhead and cannot support real-time incremental updates.

[0048] In another example, incremental updates are performed with locks. This method can add, delete, and modify data in the index for ANN retrieval. Since ANN retrieval requires reading the data in the index, the write operation during incremental update and the read operation during ANN retrieval will cause conflicts when accessing the index data together. To avoid read-write conflicts, the index needs to be locked during the incremental update process. However, this method not only needs to lock the corresponding data during incremental updates, but also needs to lock it during ANN retrieval, which will increase the performance overhead during retrieval.

[0049] In view of this, an embodiment of the present disclosure provides an index processing method, which updates the original data by writing the new data into the target storage block and adding the target storage block to the index library; at the same time, a new node is generated according to the new data, the node information of the new node is added to the composition data, and the node information of the new neighbor nodes affected by the new node is updated using atomic operations, thereby obtaining a processed index library. The method provided by the present disclosure can realize real-time update of the index library, thereby helping to improve the timeliness of retrieval. At the same time, the method provided by the present disclosure can avoid the problem of read-write conflicts without locking the data, which helps to improve query performance.

[0050] According to an embodiment of the present disclosure, the HNSW index library may include two types of data: vector data and composition data. The vector data stores the vector information corresponding to the original data, and the composition data stores the connection relationship between the original data in the form of an adjacency matrix. The original data is a composition node in the composition, and the composition topology in the HNSW index is jointly constituted according to the connection relationship between the composition nodes.

[0051] Figure 2 The structural diagram of composition data according to an embodiment of the present disclosure is schematically shown.

[0052] According to an embodiment of the present disclosure, the composition data includes N composition nodes, and each composition node includes at most 10 neighbor nodes. The composition data can be represented as follows: Figure 2 The N×10 adjacency matrix shown in FIG. 1 includes a node identification area 201, a neighbor node number area 202, and a neighbor node identification area 203. The node identification area 201 includes the node identifications of all the composition nodes contained in the composition data, such as 1, 2, ..., N. The neighbor node number area 202 includes the number of neighbor nodes of each composition node. For example, the number of neighbor nodes corresponding to composition node 1 is 1, indicating that composition node 1 includes 1 neighbor node. The neighbor node identification area 203 includes the node identifications of the neighbor nodes corresponding to each composition node. For example, the neighbor node corresponding to composition node 1 is composition node N.

[0053] Each row in the adjacency matrix corresponds to the node information of a graph node, and each node information includes the number of neighbor nodes and a group of node identifiers sorted by position number. For example, the first position of the node information is located in the neighbor node number area, identifying the number of neighbor nodes of the node, and starting from position 1, it is located in the neighbor node identifier area 203, and the node identifiers of the neighbor nodes of the node are recorded in sequence. For example, for a composition node No. N, the number of its neighbor nodes is 10, and the neighbor nodes are composition node No. 1, composition node No. 2, composition node No. 3, composition node No. 4, composition node No. 5, composition node No. 6, composition node No. 7, composition node No. 8, composition node No. 9 and composition node No. 10, respectively, and composition node No. 1 corresponds to position number 1, composition node No. 2 corresponds to position number 2, composition node No. 3 corresponds to position number 3, composition node No. 4 corresponds to position number 4, composition node No. 5 corresponds to position number 5, composition node No. 6 corresponds to position number 6, composition node No. 7 corresponds to position number 7, composition node No. 8 corresponds to position number 8, composition node No. 9 corresponds to position number 9, and composition node No. 10 corresponds to position number 10. It should be noted that in order to improve query performance, the above composition data can be stored in a continuous memory, that is, the storage address of the composition data is continuous.

[0054] Figure 3 The flowchart of the index processing method according to the embodiment of the present disclosure is schematically shown.

[0055] like Figure 3 As shown, the method 300 includes operations S310 to S350.

[0056] In operation S310 , in response to newly added data in a processing request for an index library, the newly added data is written to a target storage block among a plurality of storage blocks.

[0057] In operation S320, the target storage block is added to the index library, and a new node is generated according to the new data.

[0058] In operation S330, neighbor nodes for the newly added node are determined according to the composition data, and at least one newly added neighbor node associated with the newly added data is obtained.

[0059] In operation S340, node information for the newly added node is added to the mapping data according to the at least one newly added neighbor node.

[0060] In operation S350, the node information of the newly added neighbor nodes in the composition data is updated by using atomic operations to obtain a processed index library.

[0061] According to an embodiment of the present disclosure, the index library includes original data and composition data, the storage address of the original data is continuous, and the composition data includes composition nodes generated according to the original data. The original data may include vector data, and the vector data stores vector information of the original data.

[0062] According to an embodiment of the present disclosure, multiple storage blocks may be stored in a linked list or in an array.

[0063] According to an embodiment of the present disclosure, multiple storage blocks can be pre-initialized and generated in a resource pool. When new data is added, a target storage block can be obtained from the resource pool, the new data can be written into the target storage block and the target storage block can be added to the index library.

[0064] The HNSW algorithm will perform random access to vector data during ANN retrieval. In order to optimize query efficiency, a continuous memory is used for the storage of vector data. This method does not support dynamic expansion and deletion of data. Using a linked list to store vector data can easily add and delete vector candidate data, but it cannot efficiently support random access during ANN retrieval, which will increase the performance overhead during ANN retrieval. The disclosed embodiment provides a data structure that can support incremental reading and writing, hereinafter referred to as a resource pool.

[0065] Figure 4 The schematic diagram shows the structure of a resource pool according to an embodiment of the present disclosure.

[0066] like Figure 4 As shown, the resource pool 400 of this embodiment is a linked list structure as a whole. The resource pool 400 can initialize multiple storage blocks, such as storage block 0, storage block 1, ..., storage block N. Each storage block in the resource pool 400 can be a continuous memory, that is, the storage address is continuous.

[0067] In one embodiment, the original data in the index library can be stored in a linked list, and the original data in the index library can be stored in storage block 0, and storage block 0 is closed data; when new data is needed, the new data can be written to storage block 1, and then storage block 1 is added to the end of the linked list structure to realize the addition of the original data. Since each storage block has a corresponding storage block identifier, when searching, the storage block identifier can be obtained first, and then the data can be obtained from the corresponding storage block.

[0068] According to an embodiment of the present disclosure, the resource pool includes the following three modes: read-only mode, write-only mode and read-write mode, and different modes may correspond to different usage scenarios.

[0069] When the read-only mode is used for retrieval, no write operation is involved; the vector data read in the read-only mode is the vector data when the index library is created. The read-only mode does not involve the writing or deletion of new data, so only storage block 0 is used in the read-only mode.

[0070] The write-only mode is used to create an index library, and no read operations are involved; in this write-only mode, the original data is used to build the library, so only storage block 0 is used for incremental writing. If the memory of the initially opened storage block 0 is insufficient, the new memory space can be expanded to ensure that the original data is written into a continuous memory.

[0071] The read-write mode is used for data update. In this read-write mode, memory block 0 cannot perform memory adjustment operations, so it is necessary to add a new memory block after memory block 0, such as memory block 1, and write the newly added data into memory block 1.

[0072] According to an embodiment of the present disclosure, determining a neighbor node for a newly added node based on the composition data may include: using the newly added node as a search node, searching for a preset number of nearest neighbors from the composition data as neighbor nodes of the newly added node.

[0073] Figure 5 A schematic diagram of determining neighbor nodes according to an embodiment of the present disclosure is schematically shown.

[0074] like Figure 5 As shown, in this embodiment, the composition topology includes composition node 501, composition node 502, composition node 503, composition node 504, composition node 505, composition node 506, composition node 507, composition node 508, composition node 509, and composition node 510, wherein the unidirectional line in the composition data indicates that the composition node pointed by the line is a neighbor node of the composition node where the arrow starts, for example, composition node 503 is a neighbor node of composition node 502, while composition node 502 is not a neighbor node of composition node 503. The bidirectional line indicates that two connected composition nodes are neighbor nodes of each other, for example, composition node 502 is a neighbor node of composition node 507, and composition node 507 is a neighbor node of composition node 502.

[0075] In this embodiment,

[0076] Nth composition node 501 is a newly added node. After composition node 501 is inserted into the composition topology, according to the composition data, determining the neighbor nodes for the newly added node may include: taking the newly added node 501 as the starting search node, searching for build_num (for example, build_num is 5) composition nodes that are closer to the newly added node 501 from the composition data, and obtaining composition node 0 502, composition node 1 503, composition node 2 504, composition node 3 505, and composition node 4 506, then these 5 composition nodes are the newly added neighbor nodes of the newly added node 501. build_num is a parameter configured when creating an index library, and build_num indicates that the newly added node retrieves build_num most similar nearest neighbor nodes at a time.

[0077] According to an embodiment of the present disclosure, the node information may include the number of neighbor nodes and node identifiers of the neighbor nodes.

[0078] According to an embodiment of the present disclosure, an atomic operation represents an operation that will not be interrupted before being executed. For example, when updating composition data, only a simple type of variable is written each time.

[0079] According to an embodiment of the present disclosure, updating the node information of a newly added neighbor node in the composition data may include adding a node identifier of the newly added node in the node information, and may also include adding a node identifier of the newly added node in the node information while deleting node identifiers of some original nodes.

[0080] For example, after the node identifier of the newly added node is added to the node information of the newly added neighbor node, if the number of its neighbor nodes does not reach the shrink threshold, the node identifier of the newly added node can be directly added to the node information. The shrink threshold is pre-configured when the index library is created. The shrink threshold means that when the number of neighbor nodes of a node is greater than or equal to the threshold, it is necessary to optimize its neighbor nodes so that the number of its neighbor nodes is less than the threshold. The specific operation of optimizing its neighbor nodes can be to cancel the operation on the neighbor nodes.

[0081] For another example, after the node identifier of a newly added neighbor node is added to the node information of the newly added neighbor node, if the number of its neighbor nodes reaches the shrink threshold, it is necessary to add the node identifier of the newly added node to the node information and delete the node identifiers of some original nodes so that the number of neighbor nodes of the newly added neighbor node is less than the shrink threshold.

[0082] According to an embodiment of the present disclosure, the original data is updated by writing the newly added data into the target storage block and adding the target storage block to the index library; at the same time, a new node is generated according to the newly added data, the node information of the newly added node is added to the composition data, and the node information of the newly added neighboring nodes affected by the newly added node is updated using atomic operations, thereby obtaining a processed index library. The method provided by the present disclosure can realize real-time update of the index library, thereby helping to improve the timeliness of retrieval. At the same time, the method provided by the present disclosure can avoid the problem of read-write conflicts without locking the data, which helps to improve query performance.

[0083] 6 to 9, in combination with specific embodiments Figure 3 The method shown is further explained.

[0084] According to an embodiment of the present disclosure, the above method may also include: determining newly added neighbor nodes whose number of neighbor nodes is greater than or equal to a preset threshold, and obtaining nodes to be optimized; determining optimization information for the nodes to be optimized; wherein, using atomic operations to update the node information of newly added neighbor nodes in the composition data includes: using atomic operations to update the node information of the nodes to be optimized according to the optimization information.

[0085] The preset threshold may be the shrink threshold as described above, and the specific value of the preset threshold may be set according to actual needs. For example, when the number of neighbor nodes of the composition node a is greater than or equal to the preset threshold, the neighbor nodes of the composition node a need to be cancelled, and the composition node a is determined to be a node to be optimized.

[0086] The optimization information may include specific content that needs to be updated in the node information of the node to be optimized. For example, the neighbor nodes of the node to be optimized include composition node No. 1, composition node No. 2, composition node No. 3, and composition node No. 4, wherein after analysis, composition node No. 4 needs to be deleted, then the optimization information may include adding a new node and deleting composition node No. 4.

[0087] According to the embodiments of the present disclosure, by screening out nodes that need to perform node information update, obtaining nodes to be optimized, and then performing node information update on the nodes to be optimized, it helps to improve the update efficiency.

[0088] According to an embodiment of the present disclosure, determining optimization information for a node to be optimized includes: for each node to be optimized, when it is determined that the number of neighboring nodes to be optimized for the node to be optimized is equal to a preset threshold, based on a first preset pruning rule, the optimization information is determined according to the distance between the neighboring node to be optimized and the node to be optimized, and the distance between the neighboring nodes to be optimized, wherein the neighboring node to be optimized is a neighboring node of the node to be optimized.

[0089] The preset threshold may be the shrink threshold as described above, and when the number of neighboring nodes to be optimized of the node to be optimized is equal to the preset threshold, a cancellation operation needs to be performed. Then, the composition nodes to be cancelled are determined according to the first preset pruning rule.

[0090] The first preset pruning rule may include selecting a neighbor node to be optimized that is farther from the node to be optimized as a node to be deleted, or may include selecting one of two neighbor nodes to be optimized that are closer as a node to be deleted, or may include selecting a neighbor node to be optimized that is farther from the node to be optimized and closer to any neighbor node to be optimized as a node to be deleted. It should be noted that the purpose of performing the cancellation operation is to obtain neighbor nodes that are more evenly distributed.

[0091] Fig. 6A A schematic diagram of determining optimization information according to an embodiment of the present disclosure is schematically shown.

[0092] Figure 6B The result diagram of determining optimization information according to an embodiment of the present disclosure is schematically shown.

[0093] like Fig. 6A As shown, the composition topology of this embodiment includes composition node 601, composition node 602, composition node 603, composition node 604, composition node 605, composition node 606, composition node 607, composition node 608, composition node 609, and composition node 610. Among them, composition node 601 No. N is a newly added node. After adding composition node 601, the number of neighbor nodes of composition node 602 No. 0 reaches 5, including composition node 603 No. 1, composition node 604 No. 2, composition node 607 No. 5, composition node 609 No. 6, and composition node 601 No. N. Since the number of neighbor nodes of composition node 602 No. 0 is equal to the preset threshold, canceling the operation on composition node 602 No. 0 may include: determining the optimization information using the first preset pruning rule. For example, since the composition node No. 2 604 is far from the composition node No. 0 602, it can be determined that the composition node No. 2 604 is the node to be deleted. At the same time, since the composition node No. 1 603 and the composition node No. N 601 are close to each other, and the composition node No. 1 603 is farther from the composition node No. 0 602 than the composition node No. N 601 is from the composition node No. 0 602, it can be determined that the composition node No. 1 603 is the node to be deleted. At this time, the optimization information obtained may include adding the composition node No. N 601, deleting the composition node No. 1 603 and the composition node No. 2 604. The updated composition topology is as follows: Figure 6B As shown. Figure 6BIt can be seen that the connection between composition node No. 0 602 and composition node No. 1 603 and composition node No. 2 604 has been cancelled, and a connection pointing to composition node No. N 601 is added between composition node No. 0 602 and composition node No. N 601 .

[0094] According to an embodiment of the present disclosure, when it is determined that the number of neighbor nodes to be optimized is greater than a preset threshold, for each neighbor node to be optimized, based on a second preset pruning rule, according to the first distance between the neighbor node to be optimized and the node to be optimized, the second distance between the newly added node and the node to be optimized, and the third distance between the neighbor node to be optimized and the newly added node, the optimization sub-information for the neighbor node to be optimized is determined; and the optimization information is determined based on the optimization sub-information of each neighbor node to be optimized.

[0095] According to an embodiment of the present disclosure, when it is determined that the number of neighbor nodes to be optimized is greater than a preset threshold, it indicates that the neighbor nodes to be optimized have previously undergone a cancellation operation, and at this time, the second preset pruning rule can be used to determine the optimization information.

[0096] The second preset pruning rule may include determining not to delete the neighboring node to be optimized and not to add the node to be added when the first distance is greater than the second distance and the second distance is less than the third distance; determining not to delete the neighboring node to be optimized and to add the node to be added when the first distance is greater than the second distance and the second distance is greater than the third distance; determining to delete the neighboring node to be optimized and to add the node to be added when the first distance is less than the second distance and the second distance is less than the third distance; determining not to delete the neighboring node to be optimized and to add the node to be added when the first distance is less than the second distance and the second distance is greater than the third distance.

[0097] Fig. 7A The schematic diagram shows a principle diagram of determining optimized sub-information according to an embodiment of the present disclosure.

[0098] like Fig. 7AAs shown, in this embodiment, the composition node 710 is the current node to be optimized, the composition node 711 is one of the neighboring nodes to be optimized of the composition node 710, and the composition node 701 is a newly added node. Determining the optimization sub-information for the composition node 711 may include: determining a first distance between the composition node 711 and the composition node 710, and a second distance between the composition node 701 and the composition node 710. If the first distance is less than the second distance, indicating that the composition node 711 is closer to the composition node 710, the composition node 711 is preferentially placed in the neighboring nodes of the composition node 711. Then, the third distance between the composition node 701 and the composition node 711 is determined. Since the third distance is less than the first distance, the composition node 701 will not be added to the neighboring nodes of the composition node 710. At this time, it is equivalent to drawing a circle (a hypersphere in a high-dimensional space) with the composition node 711 as the center and the first distance as the radius. If the third distance is less than the radius, the composition node 701 will not be added to the neighboring nodes of the composition node 710. The optimization sub-information determined for the composition node 711 includes: the composition node 701 is not added to the neighboring nodes of the composition node 710, and the composition node 711 is not deleted.

[0099] Figure 7B The schematic diagram shows a principle diagram of determining optimized sub-information according to another embodiment of the present disclosure.

[0100] like Figure 7B As shown, in this embodiment, the composition node 710 is the current node to be optimized, the composition node 712 is another neighboring node to be optimized of the composition node 710, and the composition node 701 is a newly added node. Determining the optimization sub-information for the composition node 712 may include: determining a first distance between the composition node 712 and the composition node 710, and a second distance between the composition node 701 and the composition node 710. If the first distance is less than the second distance, indicating that the composition node 712 is closer to the composition node 710, the composition node 712 is preferentially placed in the neighboring nodes of the composition node 712. Then, the third distance between the composition node 701 and the composition node 712 is determined. Since the third distance is greater than the first distance, the composition node 701 is added to the neighboring nodes of the composition node 710. It is determined that this is equivalent to drawing a circle with the composition node 712 as the center and the first distance as the radius (the optimization sub-information for the composition node 712 in the high-dimensional space includes: the composition node 701 is added to the neighboring nodes of the composition node 710, and the composition node 712 is not deleted.

[0101] Figure 7C The schematic diagram shows a principle diagram of determining optimized sub-information according to another embodiment of the present disclosure.

[0102] like Figure 7CAs shown, in this embodiment, the composition node 710 is the current node to be optimized, the composition node 713 is another neighboring node to be optimized of the composition node 710, and the composition node 701 is a newly added node. Determining the optimization sub-information for the composition node 713 may include: determining the first distance between the composition node 713 and the composition node 710, and the second distance between the composition node 701 and the composition node 710. If the first distance is greater than the second distance, indicating that the composition node 701 is closer to the composition node 710, the composition node 701 is temporarily added to the neighboring nodes of the composition node 710. Then, the third distance between the composition node 701 and the composition node 713 is determined. Since the third distance is less than the first distance, the composition node 713 will be deleted from the neighboring nodes. At this time, it is equivalent to drawing a circle with the composition node 701 as the center and the second distance as the radius. If the third distance is less than the radius, the composition node 713 is deleted. It can be determined that the optimized sub-information for the composition node 713 includes: the composition node 701 is added to the neighboring nodes of the composition node 710, and the composition node 713 is deleted.

[0103] Fig.7D The schematic diagram shows a principle diagram of determining optimized sub-information according to another embodiment of the present disclosure.

[0104] like Fig.7D As shown, in this embodiment, the composition node 710 is the current node to be optimized, the composition node 714 is another neighboring node to be optimized of the composition node 710, and the composition node 701 is a newly added node. Determining the optimization sub-information for the composition node 714 may include: determining the first distance between the composition node 714 and the composition node 710, and the second distance between the composition node 701 and the composition node 710. If the first distance is greater than the second distance, indicating that the composition node 701 is closer to the composition node 710, the composition node 701 is temporarily added to the neighboring node of the composition node 710. Then, the third distance between the composition node 701 and the composition node 714 is determined. Since the third distance is greater than the first distance, the composition node 714 will be retained. It can be determined that the optimization sub-information of the composition node 714 includes: the composition node 701 is added to the neighboring node of the composition node 710, and the composition node 714 is not deleted.

[0105] According to the embodiments of the present disclosure, different numbers of neighbor nodes to be optimized use different pruning rules, which can make the distribution of neighbor nodes of the nodes to be optimized more reasonable, thereby helping to improve retrieval efficiency.

[0106] According to an embodiment of the present disclosure, the optimization sub-information includes adding a new node and deleting at least one of the neighboring nodes to be optimized; determining the optimization information according to the optimization sub-information of each neighboring node to be optimized includes: determining the node to be deleted according to the optimization sub-information of each neighboring node to be optimized; when it is determined that each optimization sub-information includes adding a new node, determining that the optimization information includes the node identifier of the new node and the node identifier of the node to be deleted; when it is determined that there is optimization sub-information that does not include adding a new node, determining that the optimization information includes the node identifier of the node to be deleted.

[0107] Figure 8 A schematic diagram of determining optimization information according to an embodiment of the present disclosure is schematically shown.

[0108] like Figure 8 As shown, the nodes to be optimized include neighbor nodes to be optimized 801, neighbor nodes to be optimized 802, neighbor nodes to be optimized 803, neighbor nodes to be optimized 804, and neighbor nodes to be optimized 805. Optimization sub-information 811 is determined for neighbor nodes to be optimized 801, optimization sub-information 811 is determined for neighbor nodes to be optimized 802, optimization sub-information 813 is determined for neighbor nodes to be optimized 803, optimization sub-information 814 is determined for neighbor nodes to be optimized, and optimization sub-information 815 is determined for neighbor nodes to be optimized 805, wherein each optimization sub-information includes whether to add a new node and whether to delete the current neighbor node to be optimized. Then, the node to be deleted 806 is determined according to the optimization sub-information 811, optimization sub-information 812, optimization sub-information 813, optimization sub-information 814, and optimization sub-information 815. Then perform operation S820 to determine whether the optimized sub-information 811, optimized sub-information 812, optimized sub-information 813, optimized sub-information 814, and optimized sub-information 815 all contain the addition of new nodes. If the judgment result is yes, the first optimization information is obtained, and the first optimization information may include the node identifier of the newly added node and the node identifier of the node to be deleted; if the judgment result is no, the second optimization information is obtained, and the second optimization information may include the node identifier of the node to be deleted.

[0109] For example, optimization sub-information 811 includes adding new nodes and deleting neighbor nodes 801 to be optimized; optimization sub-information 812 includes adding new nodes and not deleting neighbor nodes 802 to be optimized; optimization sub-information 813 includes adding new nodes and not deleting neighbor nodes 803 to be optimized; optimization sub-information 814 includes adding new nodes and not deleting neighbor nodes 804 to be optimized; optimization sub-information 815 includes adding new nodes and deleting neighbor nodes 805 to be optimized. At this time, the first optimization information is obtained, which includes adding new nodes, deleting neighbor nodes 801 to be optimized and neighbor nodes 805 to be optimized.

[0110] According to an embodiment of the present disclosure, each neighbor node to be optimized is analyzed separately using the second preset pruning rule. The node to be added is added to the node information of the node to be optimized only when the optimization sub-information of each neighbor node to be optimized contains the addition of the node to be added. The optimization information determined by this method can make the neighbor nodes of the node to be optimized more evenly distributed.

[0111] According to an embodiment of the present disclosure, wherein, based on the optimization information, updating the node information of the node to be optimized using atomic operations includes: for the node information of the node to be optimized, when it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes to be deleted is equal to 1, using atomic operations, replacing the node identifier of the node to be deleted with the node identifier of the newly added node, wherein the node identifier of the node to be deleted represents the node identifier that needs to be deleted from the node information of the node to be optimized.

[0112] For example, the node information of the node to be optimized includes node identifier 1, node identifier 2, node identifier 3, node identifier 4, and node identifier 5. The optimization information for the node to be optimized includes the node identifier of the newly added node, such as node identifier N, and the node identifier of the node to be deleted, such as node identifier 1. Then, updating the node information of the node to be optimized using atomic operations may include: replacing node identifier 1 in the node information with node identifier N, thereby completing the update of the node information for the node to be optimized.

[0113] According to an embodiment of the present disclosure, the node information includes a group of node identifiers sorted by position number, and each node identifier corresponds to a position number; the method also includes: when it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes to be deleted is greater than 1, using atomic operations to replace the node identifier of the target node in the node to be deleted with the node identifier of the newly added node, wherein the target node is the node to be deleted with the smallest position number in the node to be deleted; for other nodes in the node to be deleted, using atomic operations to replace the node identifiers of other nodes with the node identifiers of the remaining nodes, wherein the other nodes include the node to be deleted after removing the target node, and the node identifiers of the remaining nodes represent the node identifiers that do not need to be deleted in the node information of the node to be optimized. The position number represents the position of the node identifier in the node information.

[0114] According to an embodiment of the present disclosure, by utilizing atomic operations, the node identifier of the node to be deleted is replaced with the node identifier of the node to be added or the node identifier of the remaining nodes, ensuring that only one variable is written each time, avoiding read-write conflicts, and realizing real-time updating of the index library.

[0115] According to an embodiment of the present disclosure, using atomic operations to replace the node identifiers of other nodes with the node identifiers of the remaining nodes includes: for the node information of the node to be optimized, using a first pointer to take the position number of the target node as the starting point, searching in the direction of increasing position number, and determining that the node identifier of the first other node found is the current node identifier to be deleted; using a second pointer to take the end of the node information of the node to be optimized as the starting point, searching in the direction of decreasing position number, and determining that the node identifier of the first remaining node found is the current node identifier to be shifted; when it is determined that the position number of the current node identifier to be deleted is less than the position number of the current node identifier to be shifted, using atomic operations to replace the current node identifier to be deleted with the current node identifier to be shifted.

[0116] Fig.9A A schematic diagram of a composition topology according to an embodiment of the present disclosure is schematically shown.

[0117] Fig. 9B A schematic diagram schematically shows current node information according to an embodiment of the present disclosure.

[0118] Fig. 9C A schematic diagram schematically shows node information during an update process according to an embodiment of the present disclosure.

[0119] Fig.9D A schematic diagram schematically shows updated node information according to an embodiment of the present disclosure.

[0120] like Fig.9A and 9B As shown, the composition topology of this embodiment includes composition node No. 1, composition node No. 2, composition node No. 3, composition node No. 4, composition node No. 5, composition node No. 6, composition node No. 7, composition node No. 8, composition node No. 9, composition node No. 10, composition node No. 11, and composition node No. N, wherein composition node No. 11 is a newly added node. This embodiment updates the node information of composition node No. N, a neighbor node of composition node No. 11. Fig.9A As shown, before adding composition node No. 11, composition node No. N includes 10 neighbor nodes, including composition node No. 1, composition node No. 2, composition node No. 3, composition node No. 4, composition node No. 5, composition node No. 6, composition node No. 7, composition node No. 8, composition node No. 9, and composition node No. 10. The node information is as follows Fig. 9B After adding the composition node No. 11, the optimization information for the composition node No. N is determined by using the above method for determining the optimization information, including: adding the composition node No. 11, and deleting the composition nodes No. 3, 5, 7, and 10.

[0121] According to the optimization information for the Nth composition node, the node information update of the Nth composition node may include: step 1, determining the node to be deleted with the smallest position number as the target node, that is, the 3rd composition node, and then using the node identifier 11 of the 11th composition node to replace the node identifier 3 of the 3rd composition node, that is, writing the node identifier 11 to the position with the position number 3, and the result is as follows: Fig. 9C Step 2: Introduce a double pointer. Pointer i finds the first node to be deleted from left to right, and pointer j finds the first node not to be deleted from right to left. Fig. 9C As shown, at this time, pointer i points to node identifier 5, pointer j points to node identifier 9, and 9 is written at the position of pointer i. After writing, the node identifier pointed to by pointer j; in this process, 9 may have been written at the position of pointer i, and the position of pointer j has not been updated. At this time, 9 will be read twice during retrieval, but because the HNSW algorithm has the function of recording visited nodes, the second time 9 is read, it will be filtered; so this situation does not affect the retrieval. Loop through step 2 until the position number pointed to by pointer i is greater than or equal to the position number pointed to by pointer j, then end the update, and finally obtain the node information of the Nth composition node as shown in Fig.9D As shown, the updated node information of composition node No. N includes composition node No. 1, composition node No. 2, composition node No. 11, composition node No. 4, composition node No. 9, composition node No. 6, and composition node No. 8, a total of 7 neighbor nodes.

[0122] According to the embodiments of the present disclosure, a double-pointer reverse search method is used to search for nodes to be deleted and remaining nodes, which can effectively avoid repeated searches and improve search efficiency.

[0123] According to an embodiment of the present disclosure, the index library also includes a data identification table; the method may also include: in response to the data to be deleted in the processing request for the index library, modifying the data identification corresponding to the data to be deleted in the data identification table to a deletion identification, wherein the deletion identification is used to filter the data.

[0124] According to an embodiment of the present disclosure, in the data identification table, for example, identification 1 may be used to identify the data that needs to be returned, and identification 0 may be used to indicate filtering of the data.

[0125] According to an embodiment of the present disclosure, a bitmap storage structure may be used to identify the existence status of each data with a bit, so as to maximize memory saving.

[0126] According to the embodiments of the present disclosure, the data to be deleted is represented by a deletion mark, and the data does not need to be deleted, which can ensure the integrity of the composition data and does not affect the connectivity of the composition.

[0127] It should be noted that the addition, deletion and modification in the index can be abstracted into a new operation and a deletion operation, and the modification operation can be abstracted into a deletion and a new operation. The above has been described in detail about the new operation and the deletion operation, and the new operation for vector data and the new operation for composition data are described in detail in the new operation part. Therefore, the modification operation can be implemented according to the above-described new operation and deletion operation, and the modification operation is not repeated in the embodiment of the disclosure.

[0128] Fig.10 The flowchart of the data query method according to the embodiment of the present disclosure is schematically shown.

[0129] like Fig.10 As shown, the method in this embodiment includes operations S1010 to S1020.

[0130] In operation S1010 , in response to a data query request, an index library is determined according to the data to be queried in the query request.

[0131] In operation S1020, target data matching the to-be-queried data is determined according to the index library.

[0132] According to an embodiment of the present disclosure, the index library is constructed according to the above-mentioned index processing method.

[0133] Fig.11 The block diagram of the index processing device according to the embodiment of the present disclosure is schematically shown.

[0134] like Fig.11 As shown, the apparatus may include a writing module 1110 , a generating module 1120 , a first determining module 1130 , an adding module 1140 and an updating module 1150 .

[0135] The writing module 1110 is used to write the newly added data into a target storage block in the plurality of storage blocks in response to the newly added data in the processing request for the index library, wherein the index library includes original data and pattern data, the storage addresses of the original data are continuous, and the pattern data includes pattern nodes generated according to the original data. In one embodiment, the writing module 1110 can be used to perform the operation S310 described above, which will not be described in detail here.

[0136] The generation module 1120 is used to add the target storage block to the index library and generate a new node according to the new data. In one embodiment, the generation module 1120 can be used to perform the operation S320 described above, which will not be described in detail here.

[0137] The first determination module 1130 is used to determine neighbor nodes for the newly added node according to the composition data, and obtain at least one newly added neighbor node associated with the newly added data. In one embodiment, the first determination module 1130 can be used to perform the operation S330 described above, which will not be described in detail here.

[0138] The adding module 1140 is used to add node information for the newly added node in the mapping data according to the at least one newly added neighboring node. In one embodiment, the adding module 1140 can be used to perform the operation S340 described above, which will not be described in detail here.

[0139] The updating module 1150 is used to update the node information of the newly added neighboring nodes in the composition data by using atomic operations to obtain a processed index library. In one embodiment, the adding module 1150 can be used to perform the operation S350 described above, which will not be repeated here.

[0140] According to an embodiment of the present disclosure, the index processing device 1100 further includes a second determination module and a third determination module.

[0141] The second determination module is used to determine the newly added neighbor nodes whose number of neighbor nodes is greater than or equal to a preset threshold, and obtain the nodes to be optimized.

[0142] The third determination module is used to determine optimization information for the node to be optimized.

[0143] The updating module 1150 includes a first updating submodule.

[0144] The first updating submodule is used to update the node information of the node to be optimized by using atomic operations according to the optimization information.

[0145] According to an embodiment of the present disclosure, the first updating submodule includes a first replacing unit.

[0146] The first replacement unit is used to replace the node identifier of the node to be deleted with the node identifier of the newly added node by using atomic operations for the node information of the node to be optimized, when it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes to be deleted is equal to 1, wherein the node identifier of the node to be deleted represents the node identifier that needs to be deleted from the node information of the node to be optimized.

[0147] According to an embodiment of the present disclosure, the first updating submodule further includes a second replacement unit and a third replacement unit.

[0148] The second replacement unit is used to replace the node identifier of the target node in the nodes to be deleted with the node identifier of the newly added node by using atomic operations when it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes to be deleted is greater than 1, wherein the target node is the node to be deleted with the smallest position number in the nodes to be deleted.

[0149] The third replacement unit is used to replace the node identifiers of other nodes in the node to be deleted with the node identifiers of the remaining nodes using atomic operations, wherein the other nodes include the nodes to be deleted after removing the target node, and the node identifiers of the remaining nodes represent the node identifiers that do not need to be deleted in the node information of the node to be optimized.

[0150] According to an embodiment of the present disclosure, the third replacement unit includes a first determining subunit, a second determining subunit and a replacement subunit.

[0151] The first determination subunit is used to use the first pointer to search in the direction of increasing position number with the position number of the target node as the starting point for the node information of the node to be optimized, and determine the node identifier of the first other node found as the current node identifier to be deleted.

[0152] The second determination subunit is used to use the second pointer to start from the end of the node information of the node to be optimized, search in the direction of decreasing position number, and determine the node identifier of the first remaining node found as the current node identifier to be shifted.

[0153] The replacement subunit is used to replace the current to-be-deleted node identifier with the current to-be-shifted node identifier using an atomic operation when it is determined that the position number of the current to-be-deleted node identifier is smaller than the position number of the current to-be-shifted node identifier.

[0154] According to an embodiment of the present disclosure, the third determining module includes a first determining unit, a second determining unit and a third determining unit.

[0155] The first determination unit is used to determine the optimization information for each node to be optimized based on the first preset pruning rule, according to the distance between the neighbor node to be optimized and the node to be optimized, and the distance between the neighbor nodes to be optimized, when it is determined that the number of neighbor nodes to be optimized for the node to be optimized is equal to a preset threshold, wherein the neighbor node to be optimized is a neighbor node of the node to be optimized.

[0156] The second determination unit is used to determine, when it is determined that the number of neighbor nodes to be optimized is greater than a preset threshold, for each neighbor node to be optimized, based on a second preset pruning rule, according to the first distance between the neighbor node to be optimized and the node to be optimized, the second distance between the newly added node and the node to be optimized, and the third distance between the neighbor node to be optimized and the newly added node.

[0157] The third determining unit is used to determine the optimization information according to the optimization sub-information of each neighbor node to be optimized.

[0158] According to an embodiment of the present disclosure, the third determining unit includes a third determining subunit, a fourth determining subunit, and a fifth determining subunit.

[0159] The third determining subunit is used to determine the node to be deleted according to the optimization sub-information of each neighboring node to be optimized.

[0160] The fourth determining subunit is used to determine that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted when it is determined that each optimization sub-information includes adding a newly added node.

[0161] The fifth determining subunit is used to determine that the optimization information includes the node identifier of the node to be deleted when it is determined that there is optimization sub-information that does not include the addition of the newly added node.

[0162] According to an embodiment of the present disclosure, the index processing device 1100 further includes a modification module.

[0163] The modification module is used to modify the data identifier corresponding to the data to be deleted in the data identifier table into a deletion identifier in response to the data to be deleted in the processing request for the index library, wherein the deletion identifier is used to filter data.

[0164] Fig.12 A block diagram of a data query device according to an embodiment of the present disclosure is schematically shown.

[0165] like Fig.12 As shown, the device 1200 may include an index library determination module 1210 and a query module 1220 .

[0166] The index library determination module 1210 is used to respond to a data query request and determine an index library according to the data to be queried in the query request.

[0167] The query module 1220 is used to query the target data matching the data to be queried according to the index library; wherein the index library is constructed using the above-mentioned index processing method.

[0168] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0169] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method of the present disclosure.

[0170] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method of the present disclosure.

[0171] According to an embodiment of the present disclosure, a computer program product includes a computer program, wherein the computer program is stored in at least one of a readable storage medium and an electronic device, and the computer program implements the method of the present disclosure when executed by a processor.

[0172] Fig.13 A schematic block diagram of an example electronic device 1300 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0173] like Fig.13 As shown, the device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1302 or a computer program loaded from a storage unit 1308 into a random access memory (RAM) 1303. In the RAM 1303, various programs and data required for the operation of the device 1300 can also be stored. The computing unit 1301, the ROM 1302, and the RAM 1303 are connected to each other via a bus 1304. An input / output (I / O) interface 1305 is also connected to the bus 1304.

[0174] A number of components in the device 1300 are connected to an input / output (I / O) interface 1305, including: an input unit 1306, such as a keyboard, a mouse, etc.; an output unit 1307, such as various types of displays, speakers, etc.; a storage unit 1308, such as a disk, an optical disk, etc.; and a communication unit 1309, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1309 allows the device 1300 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0175] The computing unit 1301 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1301 performs the various methods and processes described above, such as the index processing method. For example, in some embodiments, the index processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1300 via the ROM 1302 and / or the communication unit 1309. When the computer program is loaded into the RAM 1303 and executed by the computing unit 1301, one or more steps of the index processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to perform the index processing method in any other appropriate manner (e.g., by means of firmware).

[0176] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0177] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0178] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0179] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0180] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0181] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0182] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0183] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An index processing method, comprising: In response to newly added data in a processing request for an index library, writing the newly added data into a target storage block among a plurality of storage blocks, wherein the index library includes original data and pattern data, the storage addresses of the original data are continuous, and the pattern data includes pattern nodes generated according to the original data; Adding the target storage block to the index library, and generating a new node according to the new data; Determine, according to the composition data, a neighbor node for the newly added node, and obtain at least one newly added neighbor node associated with the newly added data; According to at least one of the newly added neighbor nodes, adding node information for the newly added node to the composition data; and The node information of the newly added neighbor node in the composition data is updated by using atomic operations to obtain a processed index library.

2. The method according to claim 1, further comprising: Determine the newly added neighbor nodes whose number of neighbor nodes is greater than or equal to a preset threshold, and obtain the nodes to be optimized; Determining optimization information for the node to be optimized; Wherein, the updating of the node information of the newly added neighbor node in the composition data by using atomic operation includes: According to the optimization information, the node information of the node to be optimized is updated using the atomic operation.

3. The method according to claim 2, wherein: The updating the node information of the node to be optimized by using the atomic operation according to the optimization information includes: For the node information of the node to be optimized, when it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes of the node to be deleted is equal to 1, the node identifier of the node to be deleted is replaced with the node identifier of the newly added node by using the atomic operation, wherein the node identifier of the node to be deleted represents the node identifier that needs to be deleted from the node information of the node to be optimized.

4. The method according to claim 3, wherein: The node information includes a group of node identifiers sorted by position numbers, each node identifier corresponding to a position number; The method further comprises: When it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of the nodes to be deleted is greater than 1, the node identifier of the target node in the nodes to be deleted is replaced with the node identifier of the newly added node by using the atomic operation, wherein the target node is the node to be deleted with the smallest position number in the nodes to be deleted; For other nodes in the node to be deleted, the node identifiers of the other nodes are replaced with the node identifiers of the remaining nodes using the atomic operation, wherein the other nodes include the nodes to be deleted after removing the target node, and the node identifiers of the remaining nodes represent the node identifiers that do not need to be deleted in the node information of the node to be optimized.

5. The method according to claim 4, wherein: Replacing the node identifiers of the other nodes with the node identifiers of the remaining nodes by using the atomic operation includes: For the node information of the node to be optimized, using the first pointer, starting from the position number of the target node, searching in the direction of increasing position number, and determining the node identifier of the first other node found as the current node identifier to be deleted; Using the second pointer, starting from the end of the node information of the node to be optimized, searching in the direction of decreasing position number, and determining the node identifier of the first remaining node found as the current node identifier to be shifted; When it is determined that the position number of the current node identifier to be deleted is smaller than the position number of the current node identifier to be shifted, the current node identifier to be deleted is replaced with the current node identifier to be shifted by using the atomic operation.

6. The method according to claim 2, wherein: The determining of optimization information for the node to be optimized includes: For each node to be optimized, when it is determined that the number of neighboring nodes to be optimized for the node to be optimized is equal to a preset threshold, based on a first preset pruning rule, the optimization information is determined according to the distance between the neighboring node to be optimized and the node to be optimized, and the distance between the neighboring nodes to be optimized, wherein the neighboring node to be optimized is a neighboring node of the node to be optimized; In the case where it is determined that the number of the neighboring nodes to be optimized is greater than the preset threshold, for each of the neighboring nodes to be optimized, based on a second preset pruning rule, according to a first distance between the neighboring node to be optimized and the node to be optimized, a second distance between the newly added node and the node to be optimized, and a third distance between the neighboring node to be optimized and the newly added node, determining the optimization sub-information for the neighboring node to be optimized; and The optimization information is determined according to the optimization sub-information of each neighbor node to be optimized.

7. The method according to claim 6, wherein: The optimization sub-information includes at least one of adding a new node and deleting the neighboring node to be optimized; The determining the optimization information according to the optimization sub-information of each neighbor node to be optimized includes: Determine the node to be deleted according to the optimized sub-information of each neighbor node to be optimized; In the case where it is determined that each of the optimization sub-information includes adding a newly added node, determining that the optimization information includes a node identifier of the newly added node and a node identifier of the node to be deleted; In the case where it is determined that there is optimized sub-information that does not include adding a newly added node, it is determined that the optimized information includes a node identifier of the node to be deleted.

8. The method according to claim 1, wherein: The index library also includes a data identification table; The method further comprises: In response to the data to be deleted in the processing request for the index library, the data identifier corresponding to the data to be deleted in the data identifier table is modified to a deletion identifier, wherein the deletion identifier is used to filter data.

9. A data query method, comprising: In response to a data query request, determining an index library according to the data to be queried in the query request; Determine target data matching the data to be queried according to the index library; Wherein, the index library is constructed using the method described in any one of claims 1 to 8.

10. An index processing device, comprising: a writing module, configured to write the newly added data in a processing request for an index library into a target storage block among the plurality of storage blocks in response to the newly added data in the processing request, wherein the index library includes original data and pattern data, the storage addresses of the original data are continuous, and the pattern data includes pattern nodes generated according to the original data; A generation module, used to add the target storage block to the index library and generate a new node according to the new data; A first determination module, configured to determine a neighbor node for the newly added node according to the composition data, and obtain at least one newly added neighbor node associated with the newly added data; an adding module, configured to add node information for the newly added node in the composition data according to at least one of the newly added neighbor nodes; and An updating module is used to update the node information of the newly added neighbor nodes in the composition data by using atomic operations to obtain a processed index library.

11. The apparatus according to claim 10, further comprising: The second determination module is used to determine the newly added neighbor nodes whose number of neighbor nodes is greater than or equal to a preset threshold, and obtain the nodes to be optimized; A third determination module is used to determine optimization information for the node to be optimized; The updating module includes: a first updating submodule, which is used to update the node information of the node to be optimized by using the atomic operation according to the optimization information.

12. The device according to claim 11, wherein: The first updating submodule comprises: The first replacement unit is used to replace the node identifier of the node to be deleted with the node identifier of the newly added node by using the atomic operation for the node information of the node to be optimized, when it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes of the node to be deleted is equal to 1, wherein the node identifier of the node to be deleted represents the node identifier that needs to be deleted from the node information of the node to be optimized.

13. The device according to claim 12, wherein: The node information includes a group of node identifiers sorted by position numbers, each node identifier corresponding to a position number; The first updating submodule further includes: A second replacement unit is used to replace the node identifier of the target node in the nodes to be deleted with the node identifier of the newly added node by using the atomic operation when it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the nodes to be deleted, and the number of nodes of the nodes to be deleted is greater than 1, wherein the target node is the node to be deleted with the smallest position number in the nodes to be deleted; The third replacement unit is used to replace the node identifiers of other nodes in the node to be deleted with the node identifiers of the remaining nodes by using the atomic operation, wherein the other nodes include the nodes to be deleted after removing the target node, and the node identifiers of the remaining nodes represent the node identifiers that do not need to be deleted in the node information of the node to be optimized.

14. The device according to claim 13, wherein: The third replacement unit comprises: A first determination subunit is used for searching the node information of the node to be optimized by using a first pointer and taking the position number of the target node as the starting point in the direction of increasing position number, and determining the node identifier of the first other node found as the current node identifier to be deleted; A second determination subunit is used to use a second pointer to start from the end of the node information of the node to be optimized, search in a direction of decreasing position number, and determine the node identifier of the first remaining node found as the current node identifier to be shifted; The replacement subunit is used to replace the current to-be-deleted node identifier with the current to-be-shifted node identifier using the atomic operation when it is determined that the position number of the current to-be-deleted node identifier is less than the position number of the current to-be-shifted node identifier.

15. The device according to claim 11, wherein The third determination module comprises: A first determining unit is configured to determine, for each node to be optimized, the optimization information based on a first preset pruning rule according to a distance between the neighboring node to be optimized and the node to be optimized, and a distance between the neighboring nodes to be optimized, when it is determined that the number of neighboring nodes to be optimized for the node to be optimized is equal to a preset threshold, wherein the neighboring node to be optimized is a neighboring node of the node to be optimized; a second determining unit, configured to determine, for each of the neighboring nodes to be optimized, optimization sub-information for the neighboring nodes to be optimized based on a second preset pruning rule, according to a first distance between the neighboring node to be optimized and the node to be optimized, a second distance between the newly added node and the node to be optimized, and a third distance between the neighboring node to be optimized and the newly added node, when it is determined that the number of the neighboring nodes to be optimized is greater than the preset threshold; and The third determining unit is used to determine the optimization information according to the optimization sub-information of each neighbor node to be optimized.

16. The device according to claim 15, wherein: The optimization sub-information includes at least one of adding a new node and deleting the neighboring node to be optimized; The third determining unit includes: A third determining subunit, configured to determine a node to be deleted according to the optimization sub-information of each neighboring node to be optimized; A fourth determining subunit, configured to, when it is determined that each of the optimization sub-information includes adding a newly added node, determine that the optimization information includes a node identifier of the newly added node and a node identifier of the node to be deleted; The fifth determining subunit is used to determine that the optimization information includes the node identifier of the node to be deleted when it is determined that there is optimization sub-information that does not include adding a newly added node.

17. The device according to claim 10, wherein: The index library also includes a data identification table; The device also includes: a modification module. The modification module is used to modify the data identifier corresponding to the data to be deleted in the data identification table into a deletion identifier in response to the data to be deleted in the processing request for the index library, wherein the deletion identifier is used to filter data.

18. A data query device, comprising: An index library determination module, used to respond to a data query request and determine an index library according to the data to be queried in the query request; A query module, used to query target data matching the data to be queried according to the index library; Wherein, the index library is constructed using the method described in any one of claims 1 to 8.

19. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-9.

21. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images