Index processing method, data query method, device and electronic equipment

Abstract

This disclosure provides an indexing processing method, a data query method, an apparatus, and an electronic device, relating to the field of computer technology, and particularly to the field of data processing technology. The specific implementation of the indexing processing method is as follows: In response to a processing request for new data in an index database, the new data is written to a target storage block in multiple storage blocks. The index database includes original data and graph data, with the original data stored at contiguous addresses, and the graph data including graph nodes generated based on the original data. The target storage block is added to the index database, and a new node is generated based on the new data. Based on the graph data, neighboring nodes for the new node are determined, resulting in at least one new neighboring node associated with the new data. Based on at least one new neighboring node, node information for the new node is added to the graph data. Finally, atomic operations are used to update the node information of the new neighboring nodes in the graph data, resulting in the processed index database.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the field of data processing technology. Specifically, it relates to an indexing method, a data query method, an apparatus, an electronic device, a storage medium, and a program product. Background Technology

[0002] ANN (Approximate Nearest Neighbors) retrieval refers to finding one or more data points in a dataset that are most similar to target data based on their similarity. However, ANN retrieval technology requires building an index of candidate data before use, and since the candidate data is a closed set, it is difficult to update the index. Summary of the Invention

[0003] This disclosure provides an indexing method, a data query method, an apparatus, an electronic device, a storage medium, and a program product.

[0004] According to one aspect of this disclosure, an index processing method is provided, comprising: responding to newly added data in a processing request for an index library, writing the newly added data into a target storage block among multiple storage blocks, wherein the index library includes original data and graph data, the original data is stored at contiguous addresses, and the graph data includes graph nodes generated based on the original data; adding the target storage block to the index library and generating new nodes based on the newly added data; determining neighboring nodes for the new nodes based on the graph data to obtain at least one new neighboring node associated with the newly added data; adding node information for the new nodes to the graph data based on the at least one new neighboring node; and updating the node information of the new neighboring nodes in the graph data using atomic operations to obtain a processed index library.

[0005] According to another aspect of this disclosure, a data query method is provided, comprising: in response to a data query request, determining an index library based on the data to be queried in the query request; and determining target data matching the data to be queried based on the index library; wherein the index library is constructed using the above-described index processing method.

[0006] According to another aspect of this disclosure, an index processing apparatus is provided, comprising: a writing module, configured to write new data into a target storage block among multiple storage blocks in response to a processing request for an index library, wherein the index library includes original data and graph data, the original data is stored at contiguous addresses, and the graph data includes graph nodes generated based on the original data; a generating module, configured to add the target storage block to the index library and generate new nodes based on the new data; a first determining module, configured to determine neighboring nodes of the new nodes based on the graph data, thereby obtaining at least one new neighboring node associated with the new data; an adding module, configured to add node information for the new nodes to the graph data based on the at least one new neighboring node; and an updating module, configured to update the node information of the new neighboring nodes in the graph data using atomic operations, thereby obtaining a processed index library.

[0007] According to another aspect of this disclosure, a data query apparatus is provided, comprising: an index database determination module, configured to determine an index database based on the data to be queried in the query request in response to a data query request; and a query module, configured to query target data matching the data to be queried based on the index database; wherein the index database is constructed using the index processing method described above.

[0008] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of this disclosure.

[0009] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the methods of this disclosure.

[0010] According to another aspect of this disclosure, a computer program product is provided, including a computer program stored on at least one of a readable storage medium and an electronic device, wherein the computer program implements the method of this disclosure when executed by a processor.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1This illustration shows an application scenario in which the indexing processing method and apparatus can be applied according to embodiments of the present disclosure;

[0014] Figure 2 A schematic diagram illustrating the structure of the pattern data according to an embodiment of the present disclosure is shown.

[0015] Figure 3 A flowchart illustrating an indexing process according to an embodiment of the present disclosure is shown schematically.

[0016] Figure 4 A schematic diagram illustrating the structure of a resource pool according to an embodiment of the present disclosure is shown.

[0017] Figure 5 A schematic diagram illustrating the determination of neighboring nodes according to an embodiment of the present disclosure is shown.

[0018] Figure 6A A schematic diagram illustrating the determination of optimization information according to an embodiment of the present disclosure is shown.

[0019] Figure 6B A schematic diagram illustrating the results of determining optimization information according to embodiments of the present disclosure; Figure;

[0020] Figure 7A A schematic diagram illustrating the principle of determining optimization sub-information according to an embodiment of the present disclosure is shown.

[0021] Figure 7B A schematic diagram illustrating the determination of optimization sub-information according to another embodiment of the present disclosure is shown.

[0022] Figure 7C A schematic diagram illustrating the determination of optimization sub-information according to yet another embodiment of the present disclosure is shown.

[0023] Figure 7D A schematic diagram illustrating the determination of optimization sub-information according to yet another embodiment of the present disclosure is shown.

[0024] Figure 8 A schematic diagram illustrating the determination of optimization information according to an embodiment of the present disclosure is shown.

[0025] Figure 9A A schematic diagram of the topology according to an embodiment of the present disclosure is shown.

[0026] Figure 9B This schematic diagram illustrates current node information according to an embodiment of the present disclosure;

[0027] Figure 9C A schematic diagram illustrating node information during the update process according to an embodiment of this disclosure is shown.

[0028] Figure 9DA schematic diagram illustrating updated node information according to an embodiment of this disclosure is shown.

[0029] Figure 10 A flowchart illustrating a data query method according to an embodiment of the present disclosure is shown schematically;

[0030] Figure 11 A block diagram of an index processing apparatus according to an embodiment of the present disclosure is shown schematically;

[0031] Figure 12 A block diagram of a data query apparatus according to an embodiment of the present disclosure is schematically shown; and

[0032] Figure 13 A schematic block diagram of an example electronic device that can be used to implement embodiments of the present disclosure is shown. Detailed Implementation

[0033] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0034] Figure 1 The illustration shows a schematic diagram of an application scenario in which the indexing processing method and apparatus can be applied according to embodiments of the present disclosure.

[0035] It is important to note that Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the indexing processing method and apparatus can be applied may include a terminal device, but the terminal device can implement the indexing processing method and apparatus provided by the embodiments of this disclosure without interacting with the server.

[0036] like Figure 1 As shown, application scenario 100 according to this embodiment may include terminal device 101, server 102, index 103, and network. The network is used as a medium to provide a communication link between terminal device 101, server 102, and index 103. The network may include various connection types, such as wired and / or wireless communication links, etc.

[0037] Users can generate processing requests for index database 103 through the interactive interface provided by terminal device 101, and send the processing requests to server 102 via the network. Server 102 then processes the original data and mapping data in index database 103 based on the new data in the processing request to obtain the processed index database.

[0038] The terminal device 101 can be equipped with various communication client applications, such as smart assistant applications, knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social media platform software (for example only). Users can input new data into the interactive interface of these client applications to generate processing requests.

[0039] Terminal device 101 can be configured with various electronic devices that have a display screen and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0040] Server 102 can be a server providing various services, such as a backend management server supporting the content browsed by users through the interactive interface of terminal device 101 (for example only). The backend management server can add new data from received processing requests to the index, add node information of new nodes to the graph data, and update node information of new neighbor nodes. Server 102 can also be a cloud server, also known as a cloud computing server or cloud host, a host product in the cloud computing service system, to solve the shortcomings of traditional physical hosts and VPS services ("Virtual Private Server", or simply "VPS") in terms of high management difficulty and weak business scalability. Server 102 can also be a server for a distributed system, or a server combined with blockchain.

[0041] It should be noted that the indexing method provided in this embodiment can generally be executed by server 102. Correspondingly, the indexing device provided in this embodiment can also be located in server 102. The indexing method provided in this embodiment can also be executed by a server or server cluster that is different from server 102 and capable of communicating with terminal device 101 and / or server 102. Correspondingly, the indexing device provided in this embodiment can also be located in a server or server cluster that is different from server 102 and capable of communicating with terminal device 101 and / or server 102.

[0042] Alternatively, the indexing method provided in this embodiment can generally be executed by the terminal device 101. Accordingly, the indexing apparatus provided in this embodiment can generally be located in the terminal device 101.

[0043] It should be understood that Figure 1 The number of terminal devices, networks, servers, and indexes shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, servers, and indexes can be included.

[0044] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of users' personal information comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and there is no violation of public order and good morals.

[0045] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0046] In related technologies, the ANN retrieval algorithms used include graph-based indexing algorithms such as NSW (Navigable Small World) and HNSW (Hierarchical Navigable Small World), and PQ (Optimal Product Quantization)-based retrieval algorithms such as OPQ (Optimal Product Quantization). These ANN retrieval techniques require building an index of candidate data before use. However, this candidate data is a closed set, and the candidate data set remains unchanged during the ANN retrieval process, making it difficult to update the candidate data within the set.

[0047] In one example, the method for updating candidate data for an ANN index includes: the ANN index service contains two slots, a primary slot and a backup slot. The primary slot is the slot currently performing an ANN search; the backup slot is an empty slot. During data updates, the ANN index currently being searched is first copied as a replica to the backup slot, and the ANN index on the backup slot is incrementally updated. After the incremental update of the ANN index on the backup slot is complete, the backup slot becomes the primary slot to provide ANN search services, the primary slot releases the old ANN index and becomes a backup slot, completing the index update. However, this update method requires copying the entire index data during incremental data updates, which doubles the memory overhead and cannot support real-time incremental updates.

[0048] In another example, incremental updates are performed using a locking mechanism. This method allows for adding, deleting, and modifying data in the index used for ANN retrieval. Since ANN retrieval requires reading data from the index, conflicts can arise when write operations during incremental updates and read operations during ANN retrieval share access to the index data. To avoid these conflicts, the index needs to be locked during the incremental update process. However, this method requires locking not only during incremental updates but also during ANN retrieval, which increases the performance overhead of the retrieval process.

[0049] In view of this, embodiments of this disclosure provide an index processing method. This method involves writing new data into a target storage block and adding the target storage block to the index database to update the original data. Simultaneously, new nodes are generated based on the new data, and the node information of these new nodes is added to the graph data. Atomic operations are then used to update the node information of the new neighboring nodes affected by the new nodes, thereby obtaining the processed index database. The method provided by this disclosure enables real-time updates to the index database, thus improving retrieval timeliness. Furthermore, the method avoids read / write conflicts without requiring data locking, which helps improve query performance.

[0050] According to embodiments of this disclosure, the HNSW index library can contain two types of data: vector data and graph data. The vector data stores the vector information corresponding to the original data, and the graph data stores the connection relationships between the original data in the form of an adjacency matrix. The original data is a graph node in the graph, and the connection relationships between the graph nodes together constitute the graph topology in the HNSW index.

[0051] Figure 2 A schematic diagram illustrating the structure of the pattern data according to an embodiment of the present disclosure is shown.

[0052] According to embodiments of this disclosure, the graph data includes N graph nodes, and each graph node contains at most 10 neighboring nodes. The graph data can be represented as follows: Figure 2 The N×10 adjacency matrix shown includes a node identifier area 201, a neighbor node count area 202, and a neighbor node identifier area 203. The node identifier area 201 includes the node identifiers of all graph nodes included in the graph data, such as 1, 2, ..., N. The neighbor node count area 202 includes the number of neighbor nodes for each graph node; for example, graph node 1 has 1 neighbor node, indicating that graph node 1 has 1 neighbor node. The neighbor node identifier area 203 includes the node identifiers of the neighbor nodes corresponding to each graph node; for example, graph node 1 has N neighbor nodes.

[0053] Each row in the adjacency matrix corresponds to the node information of a graph node. Each node information includes the number of neighboring nodes and a set of node identifiers sorted by position number. For example, the first position of the node information is located in the neighboring node count area, which indicates the number of neighboring nodes of the node. Starting from position 1, the neighboring node identifier area 203 records the node identifiers of the neighboring nodes of the node in sequence. For example, for graph node N, its neighboring nodes are 10: nodes 1, 2, 3, 4, 5, 6, 7, 8, 9, and 10. Graph node 1 corresponds to position number 1, node 2 to position number 2, node 3 to position number 3, node 4 to position number 4, node 5 to position number 5, node 6 to position number 6, node 7 to position number 7, node 8 to position number 8, node 9 to position number 9, and node 10 to position number 10. It should be noted that, to improve query performance, the graph data can be stored in a contiguous block of memory, meaning the storage addresses of the graph data are continuous.

[0054] Figure 3 A flowchart illustrating an indexing process according to an embodiment of the present disclosure is shown.

[0055] like Figure 3 As shown, the method 300 includes operations S310 to S350.

[0056] In operation S310, in response to the new data in the processing request for the index, the new data is written to the target storage block in multiple storage blocks.

[0057] In the S320 operation, the target storage block is added to the index library, and a new node is generated based on the new data.

[0058] In operation S330, based on the graph data, the neighbor nodes for the newly added node are determined, and at least one new neighbor node associated with the newly added data is obtained.

[0059] In operation S340, based on at least one newly added neighbor node, node information for the newly added node is added to the graph data.

[0060] In operation S350, atomic operations are used to update the node information of newly added neighbor nodes in the graph data, resulting in the processed index library.

[0061] According to embodiments of this disclosure, the index library includes raw data and graph data. The raw data is stored at contiguous addresses, and the graph data includes graph nodes generated based on the raw data. The raw data may include vector data, which stores vector information of the raw data.

[0062] According to embodiments of this disclosure, multiple storage blocks can be stored using a linked list or an array.

[0063] According to embodiments of this disclosure, multiple storage blocks may be pre-initialized and generated in a resource pool. When new data is added, the target storage block can be obtained from the resource pool, the new data is written to the target storage block, and the target storage block is added to the index library.

[0064] The HNSW algorithm performs random access to vector data during ANN retrieval. To optimize query efficiency, it uses a contiguous block of memory for storing vector data. However, this method does not support dynamic expansion or deletion of data. Using linked lists to store vector data allows for convenient addition and deletion of candidate vector data, but it cannot efficiently support random access during ANN retrieval, increasing performance overhead. This disclosure provides a data structure that supports incremental read / write operations, hereinafter referred to as a resource pool.

[0065] Figure 4 A schematic diagram illustrating the structure of a resource pool according to an embodiment of the present disclosure is shown.

[0066] like Figure 4 As shown, the resource pool 400 in this embodiment has a linked list structure. The resource pool 400 can initialize multiple storage blocks, such as storage block 0, storage block 1, ..., storage block N. Each storage block in the resource pool 400 can be a contiguous segment of memory, i.e., with contiguous storage addresses.

[0067] In one embodiment, the original data in the index can be stored using a linked list. The original data can be stored in storage block 0, which is a closed data block. When new data needs to be added, it can be written to storage block 1, and then storage block 1 can be added to the end of the linked list structure to add the original data. Since each storage block has a corresponding storage block identifier, during retrieval, the storage block identifier can be obtained first, and then the data can be retrieved from the corresponding storage block.

[0068] According to embodiments of this disclosure, the resource pool includes the following three modes: read-only mode, write-only mode, and read-write mode, with different modes corresponding to different usage scenarios.

[0069] When using read-only mode for retrieval, no write operations are involved. Read-only mode reads the vector data when the index was created. This read-only mode does not involve writing or deleting new data, so only storage block 0 is used in read-only mode.

[0070] Write-only mode is used when creating an index database, and no read operations are involved. In this mode, the original data is used for database creation, so only storage block 0 is used for incremental writes. If the memory allocated in the initial storage block 0 is insufficient, new memory space can be added to ensure that the original data is written to a contiguous block of memory.

[0071] Read-write mode is used when updating data. In this mode, memory adjustment operations cannot be performed on storage block 0. Therefore, a new storage block, such as storage block 1, needs to be appended after storage block 0, and the new data needs to be written to storage block 1.

[0072] According to embodiments of this disclosure, determining neighboring nodes for a newly added node based on graph data may include: using the newly added node as a search node, searching for a preset number of nearest neighbors from the graph data as neighboring nodes of the newly added node.

[0073] Figure 5 A schematic diagram illustrating the determination of neighboring nodes according to an embodiment of the present disclosure is shown.

[0074] like Figure 5 As shown, in this embodiment, the graph topology includes graph nodes 501, 502, 503, 504, 505, 506, 507, 508, 509, and 510. In this graph data, a unidirectional line indicates that the graph node the line points to is a neighbor of the graph node from which the arrow originates. For example, graph node 503 is a neighbor of graph node 502, but graph node 502 is not a neighbor of graph node 503. A bidirectional line indicates that two connected graph nodes are neighbors of each other. For example, graph node 502 is a neighbor of graph node 507, and graph node 507 is a neighbor of graph node 502.

[0075] In this embodiment, with

[0076] Node N, 501, is a newly added node. After node 501 is inserted into the graph topology, the neighboring nodes for the newly added node are determined based on the graph data. This can include: using node 501 as the starting search node, searching the graph data for the nearest `build_num` (e.g., `build_num` is 5) graph nodes that are closest to node 501, resulting in graph nodes 0 (502), 1 (503), 2 (504), 3 (505), and 4 (506). These five graph nodes are the new neighboring nodes of node 501. `build_num` is a parameter configured when creating the index, representing the number of most similar nearest neighbors retrieved for each new node.

[0077] According to embodiments of this disclosure, node information may include the number of neighboring nodes and the node identifiers of the neighboring nodes.

[0078] According to embodiments of this disclosure, atomic operations represent operations that will not be interrupted until they are completed. For example, when updating graph data, only one simple type variable is written at a time.

[0079] According to embodiments of this disclosure, updating the node information of newly added neighboring nodes in the graph data may include adding the node identifier of the newly added node to the node information, or may include adding the node identifier of the newly added node to the node information while deleting some of the node identifiers of the original nodes.

[0080] For example, if the number of neighboring nodes does not reach the shrink threshold after adding the node identifier of the newly added neighboring node to its node information, the node identifier of the newly added node can simply be added to the node information. The shrink threshold is pre-configured when the index is created. The shrink threshold indicates that when the number of neighboring nodes of a node is greater than or equal to the threshold, its neighboring nodes need to be optimized to reduce the number of neighboring nodes below the threshold. Specifically, optimizing neighboring nodes can be achieved by canceling neighboring nodes.

[0081] For example, if the number of neighboring nodes reaches the shrink threshold after adding the node identifier of the newly added neighboring node to its node information, then the node identifier of the newly added node needs to be added to the node information and some node identifiers of the original nodes need to be deleted so that the number of neighboring nodes of the newly added neighboring node is less than the shrink threshold.

[0082] According to embodiments of this disclosure, the original data is updated by writing new data to a target storage block and adding the target storage block to the index database. Simultaneously, new nodes are generated based on the new data, and the node information of these new nodes is added to the graph data. Atomic operations are then used to update the node information of the new neighboring nodes affected by the new nodes, resulting in a processed index database. The method provided by this disclosure enables real-time updates to the index database, thereby improving retrieval timeliness. Furthermore, the method avoids read / write conflicts without requiring data locking, thus improving query performance.

[0083] Referring to Figures 6-9 below, and in conjunction with specific embodiments, Figure 3 The method shown will be further explained.

[0084] According to embodiments of this disclosure, the above method may further include: determining newly added neighbor nodes whose number of neighbor nodes is greater than or equal to a preset threshold to obtain nodes to be optimized; determining optimization information for nodes to be optimized; wherein updating the node information of newly added neighbor nodes in the graph data using atomic operations includes: updating the node information of nodes to be optimized using atomic operations based on the optimization information.

[0085] The preset threshold can be the shrink threshold as described above, and the specific value of the preset threshold can be set according to actual needs. For example, if the number of neighboring nodes of graph node a is greater than or equal to the preset threshold, it is necessary to cancel the neighboring nodes of graph node a, and then graph node a is determined to be a node to be optimized.

[0086] The optimization information can include the specific content that needs to be updated in the node information of the node to be optimized. For example, if the neighboring nodes of the node to be optimized include graph node 1, graph node 2, graph node 3, and graph node 4, and analysis shows that graph node 4 needs to be deleted, then the optimization information can include adding a new node and deleting graph node 4.

[0087] According to embodiments of this disclosure, by filtering out nodes that need to have their node information updated, nodes to be optimized are obtained, and then node information is updated on the nodes to be optimized, which helps to improve update efficiency.

[0088] According to embodiments of this disclosure, determining optimization information for a node to be optimized includes: for each node to be optimized, when the number of neighboring nodes to be optimized for the node to be optimized is equal to a preset threshold, optimization information is determined based on a first preset pruning rule, according to the distance between the neighboring nodes to be optimized and the distance between the neighboring nodes to be optimized, wherein the neighboring nodes to be optimized are the neighboring nodes of the node to be optimized.

[0089] The preset threshold can be the shrink threshold as described above. When the number of neighboring nodes of a node to be optimized equals the preset threshold, a cancellation operation needs to be performed. Then, the graph nodes that need to be cancelled are determined according to the first preset pruning rule.

[0090] The first preset pruning rule can include selecting the neighbor node that is farther away from the node to be optimized as the node to be deleted; it can also include selecting one of two neighbor nodes that are closer to the node to be optimized as the node to be deleted; or it can include selecting the neighbor node that is farther away from the node to be optimized but closer to any other neighbor node to be optimized as the node to be deleted. It should be noted that the purpose of the cancellation operation is to obtain a more evenly distributed distribution of neighbor nodes.

[0091] Figure 6A A schematic diagram illustrating the determination of optimization information according to an embodiment of the present disclosure is shown.

[0092] Figure 6B The illustration shows a schematic diagram of the results of determining optimization information according to an embodiment of the present disclosure.

[0093] like Figure 6A As shown, the graph topology of this embodiment includes graph nodes 601, 602, 603, 604, 605, 606, 607, 608, 609, and 610. Graph node N 601 is a newly added node. After adding graph node 601, the number of neighboring nodes of graph node 0 602 reaches five, including graph node 1 603, graph node 2 604, graph node 5 607, graph node 609, and graph node N 601. Since the number of neighboring nodes of graph node 0 602 equals a preset threshold, canceling graph node 0 602 can include: determining optimization information using a first preset pruning rule. For example, since graph node 2 (604) is far from graph node 0 (602), it can be determined that graph node 2 (604) is a node to be deleted. Meanwhile, since graph node 1 (603) and graph node N (601) are close to each other, and graph node 1 (603) is farther from graph node 0 (602) than graph node N (601) is from graph node 0 (602), it can be determined that graph node 1 (603) is a node to be deleted. The optimization information obtained at this time can include adding graph node N (601) and deleting graph node 1 (603) and graph node 2 (604). The updated graph topology is as follows: Figure 6B As shown. According to Figure 6BIt can be seen that the connection between graph node 0 (602) and graph nodes 1 (603) and 2 (604) has been cancelled, and a connection pointing to graph node N (601) has been added between graph node 0 (602) and graph node N (601).

[0094] According to embodiments of this disclosure, when it is determined that the number of neighbor nodes to be optimized is greater than a preset threshold, for each neighbor node to be optimized, based on a second preset pruning rule, optimization sub-information for the neighbor node to be optimized is determined according to the first distance between the neighbor node to be optimized and the node to be optimized, the second distance between the newly added node and the node to be optimized, and the third distance between the neighbor node to be optimized and the newly added node; and optimization information is determined based on the optimization sub-information of each neighbor node to be optimized.

[0095] According to embodiments of this disclosure, if the number of neighbor nodes to be optimized is greater than a preset threshold, it indicates that the neighbor node to be optimized has been cancelled once before. In this case, the optimization information can be determined using the second preset pruning rule.

[0096] The second preset pruning rules may include: if the first distance is greater than the second distance and the second distance is less than the third distance, then the neighbor node to be optimized will not be deleted, and the node to be added will not be added; if the first distance is greater than the second distance and the second distance is greater than the third distance, then the neighbor node to be optimized will not be deleted, and the node to be added will be added; if the first distance is less than the second distance and the second distance is less than the third distance, then the neighbor node to be optimized will be deleted, and the node to be added will be added; if the first distance is less than the second distance and the second distance is greater than the third distance, then the neighbor node to be optimized will not be deleted, and the node to be added will be added.

[0097] Figure 7A A schematic diagram illustrating the principle of determining optimization sub-information according to an embodiment of the present disclosure is shown.

[0098] like Figure 7AAs shown, in this embodiment, graph node 710 is the current node to be optimized, graph node 711 is one of the neighboring nodes of graph node 710 to be optimized, and graph node 701 is a newly added node. Determining the optimization sub-information for graph node 711 may include: determining a first distance between graph node 711 and graph node 710, and a second distance between graph node 701 and graph node 710. If the first distance is less than the second distance, it indicates that graph node 711 is closer to graph node 710, and graph node 711 is preferentially placed into the neighboring nodes of graph node 711. Next, the third distance between graph node 701 and graph node 711 is determined. Since the third distance is less than the first distance, graph node 701 will not be added to the neighboring nodes of graph node 710. This is equivalent to drawing a circle with graph node 711 as the center and the first distance as the radius (a hypersphere in high-dimensional space). If the third distance is less than this radius, graph node 701 will not be added to the neighboring nodes of graph node 710. Therefore, the optimization sub-information for graph node 711 includes: graph node 701 will not be added to the neighboring nodes of graph node 710, and graph node 711 will not be deleted.

[0099] Figure 7B A schematic diagram illustrating the determination of optimization sub-information according to another embodiment of the present disclosure is shown.

[0100] like Figure 7B As shown, in this embodiment, graph node 710 is the current node to be optimized, graph node 712 is another neighbor node to be optimized of graph node 710, and graph node 701 is a newly added node. Determining the optimization sub-information for graph node 712 may include: determining a first distance between graph node 712 and graph node 710, and a second distance between graph node 701 and graph node 710. If the first distance is less than the second distance, it indicates that graph node 712 is closer to graph node 710, and graph node 712 is preferentially placed into the neighbor nodes of graph node 712. Then, the third distance between the graph node 701 and the graph node 712 is determined. Since the third distance is greater than the first distance, the graph node 701 is added to the neighboring nodes of the graph node 710. Thus, it is determined that at this time, it is equivalent to drawing a circle with the graph node 712 as the center and the first distance as the radius. (In the high-dimensional space, the optimization sub-information for the graph node 712 includes: the graph node 701 is added to the neighboring nodes of the graph node 710, and the graph node 712 is not deleted.)

[0101] Figure 7C A schematic diagram illustrating the determination of optimization sub-information according to yet another embodiment of the present disclosure is shown.

[0102] like Figure 7CAs shown, in this embodiment, graph node 710 is the current node to be optimized, graph node 713 is another neighbor node to be optimized of graph node 710, and graph node 701 is a newly added node. Determining the optimization sub-information for graph node 713 may include: determining the first distance between graph node 713 and graph node 710, and the second distance between graph node 701 and graph node 710. If the first distance is greater than the second distance, it indicates that graph node 701 is closer to graph node 710, and graph node 701 is temporarily added to the neighbor nodes of graph node 710. Then, the third distance between graph node 701 and graph node 713 is determined. Since the third distance is less than the first distance, graph node 713 will be deleted from the neighbor nodes. At this time, it is equivalent to drawing a circle with graph node 701 as the center and the second distance as the radius. If the third distance is less than the radius, graph node 713 is deleted. Therefore, it can be determined that the optimized sub-information for graph node 713 includes: graph node 701 is added to the neighboring nodes of graph node 710, and graph node 713 is deleted.

[0103] Figure 7D A schematic diagram illustrating the determination of optimization sub-information according to yet another embodiment of the present disclosure is shown.

[0104] like Figure 7D As shown, in this embodiment, graph node 710 is the current node to be optimized, graph node 714 is another neighbor node to be optimized of graph node 710, and graph node 701 is a newly added node. Determining the optimization sub-information for graph node 714 may include: determining a first distance between graph node 714 and graph node 710, and a second distance between graph node 701 and graph node 710. If the first distance is greater than the second distance, it indicates that graph node 701 is closer to graph node 710, and graph node 701 is temporarily added to the neighbor nodes of graph node 710. Then, a third distance between graph node 701 and graph node 714 is determined. Since the third distance is greater than the first distance, graph node 714 will be retained. Thus, it can be determined that the optimization sub-information for graph node 714 includes: graph node 701 is added to the neighbor nodes of graph node 710, and graph node 714 is not deleted.

[0105] According to embodiments of this disclosure, different pruning rules are applied to different numbers of neighboring nodes to be optimized, which can make the distribution of neighboring nodes of the node to be optimized more reasonable, thereby helping to improve retrieval efficiency.

[0106] According to embodiments of this disclosure, the optimization sub-information includes at least one of adding new nodes and deleting neighbor nodes to be optimized; determining optimization information based on the optimization sub-information of each neighbor node to be optimized includes: determining a node to be deleted based on the optimization sub-information of each neighbor node to be optimized; when it is determined that each optimization sub-information includes adding new nodes, determining that the optimization information includes the node identifier of the new node and the node identifier of the node to be deleted; when it is determined that there is optimization sub-information that does not include adding new nodes, determining that the optimization information includes the node identifier of the node to be deleted.

[0107] Figure 8 A schematic diagram illustrating the determination of optimization information according to an embodiment of the present disclosure is shown.

[0108] like Figure 8 As shown, the nodes to be optimized include neighbor nodes 801, 802, 803, 804, and 805. Optimization sub-information 811 is determined for neighbor node 801, 802, 803, 804, and 805 respectively. Each optimization sub-information includes whether to add a new node and whether to delete the current neighbor node to be optimized. Then, based on optimization sub-information 811, 812, 813, 814, and 815, node 806 to be deleted is determined. Then, operation S820 is executed to determine whether each of the optimization sub-information 811, optimization sub-information 812, optimization sub-information 813, optimization sub-information 814, and optimization sub-information 815 contains the addition of a new node. If the determination result is yes, the first optimization information is obtained, which may include the node identifier of the newly added node and the node identifier of the node to be deleted. If the determination result is no, the second optimization information is obtained, which may include the node identifier of the node to be deleted.

[0109] For example, optimization sub-information 811 includes adding a new node and deleting a neighbor node to be optimized 801; optimization sub-information 812 includes adding a new node and not deleting a neighbor node to be optimized 802; optimization sub-information 813 includes adding a new node and not deleting a neighbor node to be optimized 803; optimization sub-information 814 includes adding a new node and not deleting a neighbor node to be optimized 804; optimization sub-information 815 includes adding a new node and deleting a neighbor node to be optimized 805. At this point, the first optimization information is obtained, which includes adding a new node, deleting a neighbor node to be optimized 801, and a neighbor node to be optimized 805.

[0110] According to the embodiments of this disclosure, each neighbor node to be optimized is analyzed using a second preset pruning rule. Only when the optimization sub-information of each neighbor node to be optimized includes the addition of a node to be added, is the node to be added added to the node information of the node to be optimized. The optimization information determined by this method can make the distribution of neighbor nodes of the node to be optimized more uniform.

[0111] According to an embodiment of this disclosure, updating the node information of the node to be optimized using atomic operations based on the optimization information includes: for the node information of the node to be optimized, if it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes to be deleted is equal to 1, using atomic operations, replacing the node identifier of the node to be deleted with the node identifier of the newly added node, wherein the node identifier of the node to be deleted represents the node identifier that needs to be deleted from the node information of the node to be optimized.

[0112] For example, the node information of the node to be optimized includes node identifier 1, node identifier 2, node identifier 3, node identifier 4, and node identifier 5. The optimization information for the node to be optimized includes the node identifier of the newly added node, such as node identifier N, and the node identifier of the node to be deleted, such as node identifier 1. Then, updating the node information of the node to be optimized using atomic operations can include replacing node identifier 1 in the node information with node identifier N, which completes the update of the node information of the node to be optimized.

[0113] According to embodiments of this disclosure, the node information includes a set of node identifiers sorted by position number, with each node identifier corresponding to a position number. The method further includes: when it is determined that the optimization information includes node identifiers of newly added nodes and node identifiers of nodes to be deleted, and the number of nodes to be deleted is greater than one, using atomic operations to replace the node identifier of the target node among the nodes to be deleted with the node identifier of the newly added node, wherein the target node is the node to be deleted with the smallest position number among the nodes to be deleted; for other nodes among the nodes to be deleted, using atomic operations to replace the node identifiers of the other nodes with the node identifiers of the remaining nodes, wherein the other nodes include the nodes to be deleted after removing the target node, and the node identifiers of the remaining nodes represent node identifiers in the node information of the nodes to be optimized that do not need to be deleted. The position number represents the position of the node identifier in the node information.

[0114] According to embodiments of this disclosure, by utilizing atomic operations to replace the node identifier of the node to be deleted with the node identifier of the node to be added or the node identifier of the remaining nodes, it is ensured that only one variable is written each time, which can avoid read-write conflict problems and realize real-time updates of the index database.

[0115] According to embodiments of this disclosure, replacing the node identifier of another node with the node identifier of the remaining node using atomic operations includes: for the node information of the node to be optimized, using a first pointer to search in the direction of increasing position number starting from the position number of the target node, determining the node identifier of the first other node found as the current node identifier to be deleted; using a second pointer to search in the direction of decreasing position number starting from the end of the node information of the node to be optimized, determining the node identifier of the first remaining node found as the current node identifier to be shifted; and if it is determined that the position number of the current node identifier to be deleted is less than the position number of the current node identifier to be shifted, replacing the current node identifier to be deleted with the current node identifier to be shifted using atomic operations.

[0116] Figure 9A A schematic diagram of the topological configuration according to an embodiment of the present disclosure is shown.

[0117] Figure 9B A schematic diagram illustrating current node information according to an embodiment of this disclosure is shown.

[0118] Figure 9C A schematic diagram illustrating node information during the update process according to an embodiment of this disclosure is shown.

[0119] Figure 9D A schematic diagram illustrating updated node information according to an embodiment of this disclosure is shown.

[0120] like Figure 9A and 9B As shown, the graph topology in this embodiment includes graph nodes 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, and N, where graph node 11 is a newly added node. This embodiment updates the node information of graph node N, a neighboring node of graph node 11. Figure 9A As shown, before the addition of the 11th graph node, the Nth graph node had 10 neighboring nodes, namely the 1st, 2nd, 3rd, 4th, 5th, 6th, 7th, 8th, 9th, and 10th graph nodes. The node information is as follows: Figure 9B As shown. After adding the 11th composition node, the optimization information for the Nth composition node is determined using the above method, including: adding the 11th composition node and deleting the 3rd, 5th, 7th, and 10th composition nodes.

[0121] Based on the optimization information for graph node N, updating the node information of graph node N can include: Step 1, determining the node to be deleted with the smallest position number as the target node, i.e., graph node 3, and then replacing the node identifier 3 of graph node 3 with the node identifier 11 of graph node 11, that is, writing the node identifier 11 into the position with position number 3, as shown below. Figure 9C As shown in the diagram. Step 2: Introduce two pointers. Pointer i moves from left to right to find the first node that needs to be deleted, and pointer j moves from right to left to find the first node that does not need to be deleted; as shown in the diagram. Figure 9C As shown, at this point, pointer i points to node identifier 5, and pointer j points to node identifier 9. 9 is written to the position of pointer i. After writing, pointer j points to the node identifier. During this process, it's possible that 9 has already been written to the position of pointer i, but the position of pointer j hasn't been updated. In this case, 9 will be read twice during retrieval. However, because the HNSW algorithm has the function of recording visited nodes, the second read of 9 will be filtered out; therefore, this situation does not affect the retrieval. Step 2 is executed repeatedly until the position number pointed to by pointer i is greater than or equal to the position number pointed to by pointer j, at which point the update ends. The final node information of the Nth graph node is as follows: Figure 9D As shown, the updated node information of graph node N includes graph node 1, graph node 2, graph node 11, graph node 4, graph node 9, graph node 6, and graph node 8, for a total of 7 neighboring nodes.

[0122] According to embodiments of this disclosure, using a two-pointer reverse search method to find the node to be deleted and the remaining nodes can effectively avoid duplicate searches and improve search efficiency.

[0123] According to embodiments of this disclosure, the index database further includes a data identifier table; the method may further include: in response to data to be deleted in a processing request for the index database, modifying the data identifier corresponding to the data to be deleted in the data identifier table to a deletion identifier, wherein the deletion identifier is used to filter data.

[0124] According to embodiments of this disclosure, the data identification table may use identifier 1 to identify that the data needs to be returned, and identifier 0 to indicate that the data is filtered.

[0125] According to embodiments of this disclosure, a bitmap storage structure can be used to identify the storage status of each piece of data using bits, thereby maximizing memory savings.

[0126] According to embodiments of this disclosure, deletion identifiers are used to represent data that needs to be deleted, eliminating the need to delete the data and ensuring the integrity of the graph data without affecting the connectivity of the graph.

[0127] It should be noted that the addition, deletion, and modification operations in the index can be abstracted into addition and deletion operations, and modification operations can be abstracted into a single deletion and addition operation. The addition and deletion operations have been described in detail above, with the addition operation section specifically detailing addition operations for vector data and addition operations for graph data. Therefore, modification operations can be implemented based on the addition and deletion operations described above, and this embodiment will not elaborate on modification operations.

[0128] Figure 10 A flowchart illustrating a data query method according to an embodiment of the present disclosure is shown schematically.

[0129] like Figure 10 As shown, the method in this embodiment includes operations S1010 to S1020.

[0130] In operation S1010, in response to a data query request, the index database is determined based on the data to be queried in the query request.

[0131] In operation S1020, target data matching the data to be queried is determined based on the index library.

[0132] According to embodiments of this disclosure, the index library is constructed based on the index processing method described above.

[0133] Figure 11 A block diagram of an index processing apparatus according to an embodiment of the present disclosure is shown schematically.

[0134] like Figure 11 As shown, the device may include a writing module 1110, a generating module 1120, a first determining module 1130, an adding module 1140, and an updating module 1150.

[0135] The write module 1110 is configured to write new data into a target storage block among multiple storage blocks in response to a processing request for the index. The index includes original data and graph data, with the original data stored at contiguous addresses, and the graph data including graph nodes generated based on the original data. In one embodiment, the write module 1110 may be used to perform the operation S310 described above, which will not be repeated here.

[0136] The generation module 1120 is used to add the target storage block to the index library and generate new nodes based on the new data. In one embodiment, the generation module 1120 can be used to perform the operation S320 described above, which will not be repeated here.

[0137] The first determining module 1130 is used to determine the neighboring nodes of the newly added node based on the graph data, thereby obtaining at least one newly added neighboring node associated with the newly added data. In one embodiment, the first determining module 1130 can be used to perform the operation S330 described above, which will not be repeated here.

[0138] The addition module 1140 is used to add node information for the newly added neighbor node to the graph data. In one embodiment, the addition module 1140 can be used to perform the operation S340 described above, which will not be repeated here.

[0139] The update module 1150 is used to update the node information of newly added neighbor nodes in the graph data using atomic operations, thereby obtaining the processed index library. In one embodiment, the addition module 1150 can be used to perform the operation S350 described above, which will not be repeated here.

[0140] According to embodiments of this disclosure, the index processing apparatus 1100 further includes a second determining module and a third determining module.

[0141] The second determination module is used to determine new neighbor nodes whose number of neighbor nodes is greater than or equal to a preset threshold, thereby obtaining the nodes to be optimized.

[0142] The third determination module is used to determine the optimization information for the node to be optimized.

[0143] The update module 1150 includes a first update submodule.

[0144] The first update submodule is used to update the node information of the node to be optimized using atomic operations based on the optimization information.

[0145] According to embodiments of this disclosure, the first update submodule includes a first replacement unit.

[0146] The first replacement unit is used to replace the node identifier of the node to be deleted with the node identifier of the newly added node using atomic operations, provided that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes to be deleted is equal to 1. The node identifier of the node to be deleted represents the node identifier that needs to be deleted from the node information of the node to be optimized.

[0147] According to embodiments of this disclosure, the first update submodule further includes a second replacement unit and a third replacement unit.

[0148] The second replacement unit is used to replace the node identifier of the target node in the nodes to be deleted with the node identifier of the newly added node when the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes to be deleted is greater than 1, using atomic operations. The target node is the node to be deleted with the smallest position number among the nodes to be deleted.

[0149] The third replacement unit is used to replace the node identifiers of other nodes in the node to be deleted with the node identifiers of the remaining nodes using atomic operations. The other nodes include the node to be deleted after removing the target node, and the node identifiers of the remaining nodes represent the node identifiers in the node information of the node to be optimized that do not need to be deleted.

[0150] According to embodiments of this disclosure, the third replacement unit includes a first determining subunit, a second determining subunit, and a replacement subunit.

[0151] The first determining subunit is used to determine the node identifier of the first other node found as the node identifier to be deleted, based on the node information of the node to be optimized, by using the first pointer to search in the direction of increasing position number of the target node.

[0152] The second determining subunit is used to use the second pointer to search in the direction of decreasing position number, starting from the end of the node information of the node to be optimized, and determine the node identifier of the first remaining node found as the current node identifier to be shifted.

[0153] The replacement sub-unit is used to replace the current node identifier to be deleted with the current node identifier to be shifted using atomic operations when it is determined that the position number of the current node identifier to be deleted is less than the position number of the current node identifier to be shifted.

[0154] According to embodiments of this disclosure, the third determining module includes a first determining unit, a second determining unit, and a third determining unit.

[0155] The first determining unit is used to determine optimization information for each node to be optimized, based on a first preset pruning rule, according to the distance between the node to be optimized and the distance between the neighboring nodes to be optimized, and the distance between the neighboring nodes to be optimized, when the number of neighboring nodes to be optimized for the node to be optimized is equal to a preset threshold. Here, the neighboring nodes to be optimized are the neighboring nodes of the node to be optimized.

[0156] The second determining unit is used to determine optimization sub-information for each neighbor node to be optimized, based on the second preset pruning rules and according to the first distance between the neighbor node to be optimized and the node to be optimized, the second distance between the newly added node and the node to be optimized, and the third distance between the neighbor node to be optimized and the newly added node, when the number of neighbor nodes to be optimized is greater than a preset threshold.

[0157] The third determining unit is used to determine the optimization information based on the optimization sub-information of each neighbor node to be optimized.

[0158] According to embodiments of this disclosure, the third determining unit includes a third determining subunit, a fourth determining subunit, and a fifth determining subunit.

[0159] The third determining sub-unit is used to determine the node to be deleted based on the optimization sub-information of each neighbor node to be optimized.

[0160] The fourth determination sub-unit is used to determine, when each optimization sub-information includes the addition of new nodes, the optimization information includes the node identifier of the new node and the node identifier of the node to be deleted.

[0161] The fifth determining sub-unit is used to determine the node identifier of the node to be deleted in the optimization information, provided that there is optimization sub-information that does not include the addition of new nodes.

[0162] According to embodiments of this disclosure, the index processing apparatus 1100 further includes a modification module.

[0163] The modification module is used to respond to the data to be deleted in the processing request for the index database by modifying the data identifier in the data identifier table corresponding to the data to be deleted to the deletion identifier. The deletion identifier is used to filter the data.

[0164] Figure 12 A block diagram of a data query apparatus according to an embodiment of the present disclosure is shown schematically.

[0165] like Figure 12 As shown, the device 1200 may include an index database determination module 1210 and a query module 1220.

[0166] The index database determination module 1210 is used to determine the index database based on the data to be queried in the query request in response to the data query request.

[0167] The query module 1220 is used to query target data that matches the data to be queried based on the index; wherein the index is constructed using the index processing method described above.

[0168] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0169] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method of the present disclosure.

[0170] According to embodiments of the present disclosure, a non-transitory computer-readable storage medium stores computer instructions, wherein the computer instructions are used to cause a computer to perform the methods of the present disclosure.

[0171] According to embodiments of the present disclosure, a computer program product includes a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the methods of the present disclosure when executed by a processor.

[0172] Figure 13 A schematic block diagram of an example electronic device 1300 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0173] like Figure 13 As shown, device 1300 includes a computing unit 1301, which can perform various appropriate actions and processes according to a computer program stored in read-only memory (ROM) 1302 or a computer program loaded from storage unit 1308 into random access memory (RAM) 1303. The RAM 1303 may also store various programs and data required for the operation of device 1300. The computing unit 1301, ROM 1302, and RAM 1303 are interconnected via bus 1304. Input / output (I / O) interface 1305 is also connected to bus 1304.

[0174] Multiple components in device 1300 are connected to input / output (I / O) interface 1305, including: input unit 1306, such as a keyboard, mouse, etc.; output unit 1307, such as various types of displays, speakers, etc.; storage unit 1308, such as a disk, optical disk, etc.; and communication unit 1309, such as a network card, modem, wireless transceiver, etc. Communication unit 1309 allows device 1300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0175] The computing unit 1301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1301 performs the various methods and processes described above, such as the indexing processing method. For example, in some embodiments, the indexing processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 1308. In some embodiments, part or all of the computer program may be loaded and / or installed on device 1300 via ROM 1302 and / or communication unit 1309. When the computer program is loaded into RAM 1303 and executed by the computing unit 1301, one or more steps of the indexing processing method described above may be performed. Alternatively, in other embodiments, the computing unit 1301 may be configured to perform the indexing processing method by any other suitable means (e.g., by means of firmware).

[0176] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0177] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0178] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0179] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0180] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0181] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, distributed system servers, or servers incorporating blockchain technology.

[0182] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0183] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. An index processing method, comprising: In response to new data in a processing request for an index repository, the new data is written to a target storage block in a plurality of storage blocks, wherein the index repository includes raw data and graph data, the storage addresses of the raw data are contiguous, and the graph data includes graph nodes generated based on the raw data; The target storage block is added to the index library, and a new node is generated based on the new data; Based on the graph data, determine the neighboring nodes for the newly added node, and obtain at least one newly added neighboring node associated with the newly added data; Based on at least one of the newly added neighbor nodes, add node information for the newly added node to the graph data; Identify new neighbor nodes whose number of neighbor nodes is greater than or equal to a preset threshold, and obtain the nodes to be optimized. Determine the optimization information for the node to be optimized; Regarding the node information of the node to be optimized, if it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes of the node to be deleted is equal to 1, an atomic operation is used to replace the node identifier of the node to be deleted with the node identifier of the newly added node, wherein the node identifier of the node to be deleted represents the node identifier that needs to be deleted from the node information of the node to be optimized.

2. The method of claim 1, wherein, The node information includes a set of node identifiers sorted by location number, with each node identifier corresponding to a location number; The method further includes: If the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes of the node to be deleted is greater than 1, the atomic operation is used to replace the node identifier of the target node in the node to be deleted with the node identifier of the newly added node, wherein the target node is the node to be deleted with the smallest position number in the node to be deleted. For the other nodes in the node to be deleted, the atomic operation is used to replace the node identifier of the other nodes with the node identifier of the remaining nodes. The other nodes include the node to be deleted after removing the target node, and the node identifier of the remaining nodes represents the node identifiers in the node information of the node to be optimized that do not need to be deleted.

3. The method of claim 2, wherein, Replacing the node identifiers of the other nodes with the node identifiers of the remaining nodes using the atomic operations includes: For the node information of the node to be optimized, the first pointer is used to search in the direction of increasing position number, starting from the position number of the target node, and the node identifier of the first other node found is determined as the current node to be deleted; Using the second pointer, starting from the end of the node information of the node to be optimized, search in the direction of decreasing position number, and determine the node identifier of the first remaining node found as the current node identifier to be shifted; If it is determined that the position number of the current node to be deleted is less than the position number of the current node to be shifted, the atomic operation is used to replace the current node to be deleted with the current node to be shifted.

4. The method of claim 1, wherein, The determination of optimization information for the node to be optimized includes: For each node to be optimized, if the number of neighboring nodes to be optimized for the node to be optimized is equal to a preset threshold, the optimization information is determined based on the first preset pruning rule, according to the distance between the neighboring nodes to be optimized and the distance between the neighboring nodes to be optimized, wherein the neighboring nodes to be optimized are the neighboring nodes of the node to be optimized. If the number of neighbor nodes to be optimized is greater than the preset threshold, for each neighbor node to be optimized, based on the second preset pruning rule, and according to the first distance between the neighbor node to be optimized and the node to be optimized, the second distance between the newly added node and the node to be optimized, and the third distance between the neighbor node to be optimized and the newly added node, optimization sub-information for the neighbor node to be optimized is determined; and The optimization information is determined based on the optimization sub-information of each of the neighboring nodes to be optimized.

5. The method according to claim 4, wherein, The optimization sub-information includes at least one of adding new nodes and deleting the neighbor nodes to be optimized; The step of determining the optimization information based on the optimization sub-information of each neighbor node to be optimized includes: The node to be deleted is determined based on the optimization sub-information of each of the neighbor nodes to be optimized; If it is determined that each of the optimization sub-information includes the addition of a new node, then the optimization information includes the node identifier of the new node and the node identifier of the node to be deleted. If it is determined that there is optimization sub-information that does not include the addition of new nodes, then the optimization information includes the node identifier of the node to be deleted.

6. The method according to claim 1, wherein, The index database also includes a data identification table; The method further includes: In response to the data to be deleted in the processing request for the index, the data identifier corresponding to the data to be deleted in the data identifier table is modified to a deletion identifier, wherein the deletion identifier is used to filter data.

7. A data query method, comprising: In response to a data query request, the index database is determined based on the data to be queried in the query request; Based on the index, determine the target data that matches the data to be queried; The index library is constructed using the method described in any one of claims 1 to 6.

8. An index processing apparatus, comprising: A writing module is used to write new data into a target storage block among multiple storage blocks in response to a processing request for the index library. The index library includes original data and graph data. The storage addresses of the original data are contiguous, and the graph data includes graph nodes generated based on the original data. A generation module is used to add the target storage block to the index library and generate new nodes based on the new data; The first determining module is used to determine the neighboring nodes of the newly added node based on the graph data, and obtain at least one newly added neighboring node associated with the newly added data. An addition module is used to add node information for the newly added neighbor node to the graph data based on at least one of the newly added neighbor nodes; The second determination module is used to determine newly added neighbor nodes whose number of neighbor nodes is greater than or equal to a preset threshold, and to obtain the nodes to be optimized. The third determining module is used to determine the optimization information for the node to be optimized; An update module, comprising a first update submodule, the first update submodule comprising a first replacement unit, configured to, for the node information of the node to be optimized, when determining that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes of the node to be deleted is equal to 1, use atomic operations to replace the node identifier of the node to be deleted with the node identifier of the newly added node, wherein the node identifier of the node to be deleted represents the node identifier that needs to be deleted from the node information of the node to be optimized.

9. The apparatus according to claim 8, wherein, The node information includes a set of node identifiers sorted by location number, with each node identifier corresponding to a location number; The first update submodule also includes: The second replacement unit is used to replace the node identifier of the target node in the nodes to be deleted with the node identifier of the newly added node when it is determined that the optimization information includes the node identifier of the newly added node and the node identifier of the node to be deleted, and the number of nodes of the node to be deleted is greater than 1, using the atomic operation, wherein the target node is the node to be deleted with the smallest position number among the nodes to be deleted. The third replacement unit is used to replace the node identifier of the other nodes in the node to be deleted with the node identifier of the remaining node using the atomic operation. The other nodes include the node to be deleted after removing the target node, and the node identifier of the remaining node represents the node identifier that does not need to be deleted in the node information of the node to be optimized.

10. The apparatus according to claim 9, wherein, The third replacement unit includes: The first determining subunit is used to, based on the node information of the node to be optimized, use a first pointer to search in the direction of increasing position number, starting from the position number of the target node, and determine the node identifier of the first other node found as the identifier of the current node to be deleted. The second determining subunit is used to use the second pointer to search in the direction of decreasing position number, starting from the end of the node information of the node to be optimized, and determine the node identifier of the first remaining node found as the current node identifier to be shifted. The replacement subunit is used to replace the current node identifier to be deleted with the current node identifier to be shifted using the atomic operation when it is determined that the position number of the current node identifier to be deleted is less than the position number of the current node identifier to be shifted.

11. The apparatus according to claim 8, wherein, The third determining module includes: The first determining unit is configured to, for each node to be optimized, determine the optimization information based on a first preset pruning rule, according to the distance between the node to be optimized and the distance between the neighboring nodes to be optimized, and the distance between the neighboring nodes to be optimized, when the number of neighboring nodes to be optimized for the node to be optimized is equal to a preset threshold, wherein the neighboring nodes to be optimized are the neighboring nodes of the node to be optimized. The second determining unit is configured to, when determining that the number of neighbor nodes to be optimized is greater than the preset threshold, determine, for each neighbor node to be optimized, optimization sub-information based on a second preset pruning rule and according to a first distance between the neighbor node to be optimized and the node to be optimized, a second distance between the newly added node and the node to be optimized, and a third distance between the neighbor node to be optimized and the newly added node; and The third determining unit is used to determine the optimization information based on the optimization sub-information of each of the neighbor nodes to be optimized.

12. The apparatus according to claim 11, wherein, The optimization sub-information includes at least one of adding new nodes and deleting the neighbor nodes to be optimized; The third determining unit includes: The third determining subunit is used to determine the node to be deleted based on the optimization sub-information of each of the neighbor nodes to be optimized; The fourth determining subunit is used to determine, when it is determined that each of the optimization sub-information includes the addition of a new node, that the optimization information includes the node identifier of the new node and the node identifier of the node to be deleted; The fifth determining subunit is used to determine, when it is determined that there is optimization sub-information that does not include the addition of new nodes, the optimization information includes the node identifier of the node to be deleted.

13. The apparatus according to claim 8, wherein, The index database also includes a data identification table; The device further includes: a modification module; The modification module is used to respond to the data to be deleted in the processing request for the index database by modifying the data identifier corresponding to the data to be deleted in the data identifier table to a deletion identifier, wherein the deletion identifier is used to filter data.

14. A data query device, comprising: The index database determination module is used to determine the index database based on the data to be queried in the query request in response to the data query request. The query module is used to query target data that matches the data to be queried based on the index. The index library is constructed using the method described in any one of claims 1 to 6.

15. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

16. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

17. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method according to any one of claims 1-7 when executed by a processor.

Images