Method, device, system and cluster for batch data cleaning of graph database

Abstract

The invention discloses a batch data cleaning method, device and system for a graph database and a cluster, the graph database is deployed in a target cluster in a distributed manner, the graph database is partitioned based on an external unique identifier of a vertex, and the method comprises the following steps: obtaining cluster information corresponding to the target cluster; obtaining a plurality of pieces of to-be-cleaned graph data containing an external unique identifier; based on the cluster information and the external unique identifier, calculating to obtain a target address corresponding to each to-be-cleaned graph data, the target address being a node address corresponding to the corresponding to-be-cleaned graph data in a target cluster; and based on the target address, sending the corresponding to-be-cleaned graph data to a corresponding node in the target cluster, and cleaning the received to-be-cleaned graph data in batches by the node. Compared with the prior art, the method does not need an intermediate node / coordination node for calculation and forwarding, and can effectively reduce the network overhead.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and particularly to a method, device, system and cluster for batch cleaning data in a graph database. Background Art

[0002] A graph database is a database based on a graph model, mainly used for the storage and processing of complex network data. The data in a graph database consists of vertices and edges, where vertices represent entities and edges represent the relationships between vertices.

[0003] Currently, graph databases are often distributed and deployed in a cluster; When the cluster has a central node, a deletion request is sent to the central node, which then distributes it to the storage node where the graph data to be cleaned is located, and the storage node completes the corresponding deletion operation; When the cluster includes several coordinating nodes (excluding the central node), at this time, the deletion request will be sent to any one of the coordinating nodes, and the coordinating node will forward it to the node where the graph data to be cleaned is located. Summary of the Invention

[0004] Aiming at the disadvantages in the prior art that deletion operations all require the central node or the coordinating node to perform calculation and forwarding, resulting in large network overhead and pressure concentrating on a single node in the batch deletion scenario, which affects the performance of the cluster, the present invention provides a method, device, system and cluster for batch cleaning data in a graph database.

[0005] To solve the above technical problems, the present invention is solved by the following technical solutions: First invention, the present invention provides a method for batch cleaning data in a graph database, where the graph database is distributed and deployed in a target cluster, and the graph database is partitioned based on the external unique identifier of the vertex, including the following steps: Obtain the cluster information corresponding to the target cluster, where the cluster information includes the number of nodes, as well as the address information and partition information corresponding to each node; Obtain several pieces of graph data to be cleaned that contain the external unique identifier; Based on the cluster information and the external unique identifier, calculate the target address corresponding to each piece of graph data to be cleaned, where the target address is the node address corresponding to the graph data to be cleaned in the target cluster; Based on the target address, send the corresponding graph data to be cleaned to the corresponding node in the target cluster, and the node batch cleans the received graph data to be cleaned.

[0006] This application calculates the node address where the graph data to be cleaned is located based on cluster information and an external unique identifier, and directly sends the graph data to be cleaned to the corresponding node. Compared with the prior art, it reduces the information interaction between clusters, thereby further improving performance and reducing network overhead.

[0007] As an implementable manner: Receive at least one piece of deletion data information input by the user. The deletion data information includes mapping data and filtering conditions. Among them, the mapping data is used to indicate the data source corresponding to the target deletion data, the location information of the target deletion data in the data source, and the graph data type corresponding to the target deletion data in the graph database; Determine the graph data to be cleaned corresponding to the deletion data information. Specifically: Determine the graph data type of the target deletion data based on the corresponding mapping data. The graph data type includes point type and edge type; When the graph data type of the target deletion data is the edge type, based on the corresponding mapping data and filtering conditions, query and obtain the corresponding target edge from the corresponding data source, use the two vertices connected by the target edge as the positioning vertices, and generate the graph data to be cleaned corresponding to the target edge based on the external unique identifiers of the two positioning vertices; When the graph data type of the target deletion data is the point type, based on the corresponding mapping data and filtering conditions, query and obtain the corresponding target point from the corresponding data source, generate the graph data to be cleaned corresponding to the target point based on the external unique identifier corresponding to the target point, and also use the edge connecting the corresponding neighbor vertex to the target point as the target edge, and generate the graph data to be cleaned corresponding to the target edge based on the external unique identifier of the corresponding neighbor vertex.

[0008] In this application, by defining the mapping data, the data that needs to be deleted in batches is specified, and the operation is more intuitive and convenient. It does not require the user to have a programming foundation, and there is no need to generate query statements or call API interfaces. The user only needs to configure the mapping data, which has a low usage difficulty, can improve the popularity rate, and meet the needs of more users.

[0009] Through the design of the mapping data, this application can support directly reading data from different data sources for batch deletion, avoiding the overhead of the overall process caused by the extra generation of intermediate files.

[0010] As an implementable manner: Based on the cluster information and the corresponding external unique identifier, determine the partition where the corresponding graph data to be cleaned is located, and use the node address corresponding to the partition as the target address corresponding to the graph data to be cleaned; Generate a batch deletion task containing partition information based on the graph data to be cleaned in the same partition, and directly send the batch deletion task to the corresponding node in the target cluster based on the corresponding target address.

[0011] As an implementable manner: Judge whether the quantity of the graph data to be cleaned to be sent corresponding to each partition exceeds a preset quantity threshold; If so, generate a batch deletion task with a pessimistic locking strategy; If not, generate a batch deletion task with an optimistic locking strategy; When the locking strategy is pessimistic, the corresponding node preferentially executes the batch deletion task and locks the corresponding partition when executing the batch deletion task; When the locking strategy is optimistic, the corresponding node only locks the corresponding graph data to be cleaned when executing the batch deletion task.

[0012] Through the design of the cache mechanism and the lock mechanism local to the node, the present invention can reduce data contention and conflicts and accelerate data processing.

[0013] In a second aspect, the present invention provides an apparatus for batch cleaning data in a graph database. The graph database is distributedly deployed in a target cluster, and the graph database is partitioned based on the external unique identifier of a vertex. The system includes: A cluster information acquisition module, configured to acquire the cluster information corresponding to the target cluster, where the cluster information includes the number of nodes, and the address information and partition information corresponding to each node; A cleaning information acquisition module, configured to acquire a plurality of pieces of graph data to be cleaned including external unique identifiers; An address calculation module, configured to calculate, based on the cluster information and the external unique identifier, the target address corresponding to each piece of graph data to be cleaned, where the target address is the node address corresponding to the corresponding graph data to be cleaned in the target cluster; A sending module, configured to send the corresponding graph data to be cleaned to the corresponding node in the target cluster based on the target address, and the node batch cleans the received graph data to be cleaned.

[0014] In a third aspect, the present invention provides a method for batch cleaning data in a graph database. The graph database is distributedly deployed in a target cluster, and the graph database is partitioned based on the external unique identifier of a vertex. The target cluster includes a plurality of storage nodes, and each storage node performs the following steps: Receive a cleaning task transmitted externally, where the cleaning task includes one or more pieces of graph data to be cleaned, and store the graph data to be cleaned in a cache; When all the cleaning tasks of the corresponding storage node are received, the graph data to be cleaned in the cache is batch-locked and cleaned based on partitions.

[0015] As an implementable manner, before all the cleaning tasks of the corresponding storage node are received, it further includes the step of over-limit cleaning, and the specific steps are as follows: After new graph data to be cleaned is added to the cache, or, based on a preset judgment period, it is judged whether the cache reaches a preset storage threshold. If so, the graph data to be cleaned corresponding to one or more partitions in the cache is batch-locked and cleaned based on partitions according to a preset cleaning rule.

[0016] As an implementable manner: The cleaning task is a batch deletion task, and the batch deletion task includes partition information, a locking strategy, and one or more pieces of graph data to be cleaned corresponding to the partition information; When the locking strategy is the pessimistic mode, the corresponding storage node writes the graph data to be cleaned into the cache, and after identifying the corresponding partition based on the partition information, locks the partition based on the corresponding partition information, and batch-cleans each piece of graph data to be cleaned corresponding to the partition in the cache; When the locking strategy is the optimistic mode, the corresponding storage node writes the graph data to be cleaned into the cache, and based on the partition information identifies the corresponding partition. When it is judged that the cache reaches the preset storage threshold, the graph data to be cleaned in the cache is batch-locked and cleaned based on partitions.

[0017] This application reduces data contention and conflicts through the cache mechanism and lock mechanism local to the node, and accelerates data processing.

[0018] In a fourth aspect, the present invention provides a cluster, in which a target graph database is distributedly deployed, and the target graph database is partitioned based on the external unique identifier of vertices. The cluster includes several storage nodes, and each storage node performs batch cleaning according to the method executed by the storage node described in any one of the above.

[0019] In a fifth aspect, the present invention provides a system for batch cleaning data of a graph database. The graph database is distributedly deployed in a target cluster, and the graph database is partitioned based on the external unique identifier of vertices, and includes: A cleaning system and a cluster; The cleaning system is the device provided above; The cluster is the cluster provided above.

[0020] Since the present invention adopts the above technical solutions, it has remarkable technical effects: The present invention calculates the target address corresponding to the graph data to be cleaned outside the cluster, and directly sends the graph data to be cleaned to the corresponding node based on the target address. Compared with the prior art, there is no need for an intermediate node / coordination node to perform calculation and forwarding, avoiding the pressure concentration on a single intermediate node / coordination node in the scenario of large-scale data batch cleaning, and effectively reducing network overhead. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0022] Figure 1 is a schematic flowchart of a method for batch cleaning data in a graph database provided by an exemplary embodiment; Figure 2 is a schematic diagram of module connections of a device for batch cleaning data in a graph database provided by an exemplary embodiment; Figure 3 is a schematic flowchart of a method for batch cleaning data in a graph database provided by another exemplary embodiment. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0023] The following further elaborates on the present invention in conjunction with embodiments. The following embodiments are explanations of the present invention, and the present invention is not limited to the following embodiments.

[0024] It should be noted that: in other embodiments, the steps of the corresponding method are not necessarily executed in the order shown and described in this specification. In some other embodiments, the steps included in the method may be more or less than those described in this specification. In addition, a single step described in this specification may be decomposed into multiple steps for description in other embodiments; and multiple steps described in this specification may also be combined into a single step for description in other embodiments.

[0025] When batch deleting data in a graph database distributed and deployed in a cluster, it is often the case that a central node or a certain coordination node calculates and forwards the corresponding data based on a deletion request, resulting in low batch cleaning efficiency; In view of the above problems, this specification provides a solution for a graph database distributed and deployed in a target cluster, where the graph database is partitioned based on the external unique identifier of vertices. The solution calculates the node addresses corresponding to each graph data to be cleaned outside the cluster, and directly sends the corresponding graph data to be cleaned to the corresponding nodes based on the node addresses, reducing the network bandwidth overhead caused by multiple data forwarding between clusters, enabling the nodes to focus on the execution of local tasks, and effectively improving the batch cleaning efficiency.

[0026] Referring to Figure 1 , Figure 1 is a schematic flowchart of a method for batch cleaning data of a graph database provided by an exemplary embodiment. As Figure 1 shown, the method includes the following steps: S100. Obtain the cluster information corresponding to the target cluster; In this embodiment, the cluster information includes the number of nodes, as well as the address information and partition information corresponding to each node, where the partition information is the primary and standby replica partition information responsible for the node; The cluster information may further include the partitioning algorithm used by the target cluster, such as the hash algorithm; The cluster information is the information required to determine the partitioning location and node address of relevant graph data in the cluster based on the external unique identifier. Those skilled in the art can set the required cluster information according to the actual situation.

[0027] During the process of maintaining and managing the graph database, there is often a need for large-scale batch cleaning of the data in the graph database. That is, the method for batch cleaning data provided in this embodiment is used by the administrator of the graph database, and the administrator of the graph database can obtain and provide the above cluster information.

[0028] S200. Obtain a number of graph data to be cleaned that includes the external unique identifier; When the graph data to be cleaned is vertex data, the external unique identifier is the external unique identifier of the corresponding vertex; When the graph data to be cleaned is edge data, the external unique identifier is the external unique identifier of one of the vertices it connects. That is, when deleting an edge, two graph data to be cleaned will be generated based on the two vertices connected by the edge.

[0029] S300. Based on the cluster information and the external unique identifier, calculate the target addresses corresponding to each graph data to be cleaned; The target address is the node address corresponding to the graph data to be cleaned in the target cluster; Based on the obtained unique external identifier of the point-edge and the partition information, the node address where the graph data to be cleaned is located can be directly determined through logical calculation, that is, based on the cluster information and the corresponding unique external identifier, the partition where the corresponding graph data to be cleaned is located is determined, and the node address corresponding to the partition is used as the target address corresponding to the graph data to be cleaned.

[0030] As an implementable manner, a partition algorithm corresponding to the target cluster (such as a hash algorithm) is pre-configured. Based on the cluster information and the external unique identifier, according to the partition algorithm, the partition where the graph data to be cleaned is located and the node address corresponding to the partition are calculated; The partition algorithm of the cluster will not be easily changed. The partition algorithm only needs to be configured once, and this algorithm can be called in subsequent tasks of batch cleaning data.

[0031] S400. Based on the target address, the corresponding graph data to be cleaned is sent to the corresponding node in the target cluster, and the node batch-cleans the received graph data to be cleaned; Note that when all the graph data to be cleaned corresponding to a certain node has been sent, a signal indicating the completion of transmission will be sent to the corresponding node. This is a conventional existing technology, so it will not be described in detail in this specification.

[0032] In this embodiment, the target address corresponding to the graph data to be cleaned is calculated outside the cluster, and based on the target address, the graph data to be cleaned is directly sent to the corresponding node. Compared with the prior art, there is no need for an intermediate node / coordinating node to perform calculations and forwarding, avoiding the pressure being concentrated on a single intermediate node / coordinating node in the scenario of large-scale data batch cleaning, and omitting the forwarding step of the central node / coordinating node, effectively reducing network overhead.

[0033] Currently, there are three solutions for batch deleting data from a graph database, which are as follows: A solution of using a query language to execute a DELETE statement; this solution has high flexibility, but very long query statements need to be written in the batch deletion scenario; In this solution, the point / edge information to be deleted needs to be passed to the central node in the form of concatenating Cypher statements; For example, when 100 points need to be deleted, 100 Delete Cypher statements need to be processed, or 1 Cypher statement is used to batch delete points that meet the corresponding conditions through the "in" method; In the demand scenario of batch deleting data, even if the management personnel provide the original data file to be deleted, it still needs to be processed into Cypher statements for the second time before deletion, increasing the process overhead; and in order to completely clean all data, Cypher statements need to be continuously concatenated and the task is completed by means of loop execution; And when generating Cypher statements, it will be limited by the length threshold of the Cypher statements themselves. Too long Cypher statements will cause excessive syntax parsing and memory occupation, while too short statements will result in too many generated Cypher statements. Wherein: The DELETE statement is a command used to implement the deletion function; The Cypher statement is a declarative query language used to query and operate on graph databases; The DeleteCypher statement is used to query and delete nodes, relationships, or paths from the graph.

[0034] The solution of batch deleting data by calling the API interface is relatively intuitive in operation but has a high operation threshold, requiring users to have certain programming capabilities. Moreover, the API interface usually only provides the function of deleting a single node or edge. For the situation that requires batch deletion, it needs to be implemented by looping to call the API interface.

[0035] The solution of batch deleting data using ETL tools supports various data formats and data sources, but it requires additional work to export, delete, and then import the data. In addition, when importing the data, the problem of index reconstruction needs to be considered, otherwise the query performance may decline.

[0036] In view of the defects existing in the existing solutions, this specification provides an implementable way to obtain the to-be-cleaned graph data, which is specifically as follows: S210. Obtain at least one piece of delete data information input by the user; The delete data information includes mapping data and filtering conditions. Among them, the mapping data is used to indicate the data source corresponding to the target delete data, the location information of the target delete data in the data source (used to indicate the table and column corresponding to the target delete data), and the graph data type corresponding to the target delete data in the graph database (point type or edge type); The filtering condition is used to indicate the range corresponding to the target delete data. Those skilled in the art can set the filtering condition according to actual needs, and the filtering condition can be empty.

[0037] In this embodiment, through the design of the mapping data, the data source and the graph data are associated, which can intuitively display the data information to be deleted and support multiple data sources and data formats.

[0038] S220. Determine the to-be-cleaned graph data corresponding to the delete data information, specifically: S221. Determine the graph data type of the target delete data based on the corresponding mapping data; The graph data type includes point type and edge type; S222. When the graph data type of the target deletion data is an edge type, based on the corresponding mapping data and filtering conditions, query and obtain the corresponding target edge from the corresponding data source, use the two vertices connected by the target edge as positioning vertices, and generate the graph data to be cleaned corresponding to the target edge based on the external unique identifiers of the two positioning vertices; That is, when the target deletion data is of the edge type, only the corresponding target edge needs to be deleted; At this time, for each target edge to be deleted, generate the graph data to be cleaned corresponding to the two positioning vertices connected thereto one by one, so as to delete the target edge under the two positioning vertices in the graph database; S223. When the graph data type of the target deletion data is a vertex type, based on the corresponding mapping data and filtering conditions, query and obtain the corresponding target vertex from the corresponding data source, and generate the graph data to be cleaned corresponding to the target vertex based on the external unique identifier corresponding to the target vertex. Also, use the edge connecting the corresponding neighbor vertex to the target vertex as the target edge, and generate the graph data to be cleaned corresponding to the target edge based on the external unique identifier of the corresponding neighbor vertex; That is, when the target deletion data is of the vertex type, not only the corresponding target vertex needs to be deleted, but also all the edges connecting to the target vertex need to be deleted; In this embodiment, generate the graph data to be cleaned corresponding to the target vertex one by one to clean the target vertex and the edges from the target vertex to its neighbor vertices, and generate the graph data to be cleaned corresponding to the neighbor vertices of the target vertex one by one to delete the edges connecting to the target vertex under each neighbor vertex; The target vertex and the edges from the target vertex to its neighbor vertices are in the same partition. In actual deletion, only generate the graph data to be cleaned corresponding to the target vertex, and the corresponding node will delete the references to the corresponding target vertex and the edges from the target vertex to its neighbor vertices based on this graph data to be cleaned. When flushing the disk, the edges from the target vertex to its neighbor vertices will be automatically cleared; The edges from the neighbor vertices to the target vertex will be in the partition corresponding to the neighbor vertices and will be deleted by the node where the neighbor vertices are located; The method proposed in this embodiment does not limit the order of the graph data to be cleaned, and only needs to ensure that all the graph data to be cleaned are cleaned;

[0039] In this embodiment, through the design of the mapping data, it is possible to support multiple data sources and data formats, and relevant management personnel only need to configure the cluster information and deletion data information to batch clean the corresponding data, enabling management personnel without programming skills to conveniently and intuitively perform batch deletion operations, effectively solving problems such as high operation difficulty and high usage threshold, improving the efficiency of data management, and enhancing the cleaning performance.

[0040] As an implementable manner, based on the graph data to be cleaned in the same partition, a batch deletion task including partition information is generated, and the batch deletion task is directly sent to the corresponding node in the target cluster based on the corresponding target address; Those skilled in the art can set the rules for task generation and sending according to actual needs. For example, the graph data to be cleaned in the same partition can be segmented to generate multiple batch deletion tasks for transmission. Also, when calculating the target address of the graph data to be cleaned, the tasks can be generated and sent based on the preset sending rules. This specification does not limit it in detail. It is only necessary to directly send the graph data to be cleaned to the corresponding node based on the target address.

[0041] As an implementable manner, the batch deletion task further includes a locking strategy, and the locking strategy can be manually selected or automatically selected. The way of automatically selecting the locking strategy is as follows: Judge whether the quantity of the graph data to be cleaned to be sent corresponding to each partition exceeds a preset quantity threshold. Those skilled in the art can set this quantity threshold according to actual needs. The quantity threshold described in this specification is greater than the corresponding storage threshold of the node cache; If so, generate a batch deletion task with a pessimistic locking strategy; If not, generate a batch deletion task with an optimistic locking strategy; When the locking strategy is in the pessimistic mode, the corresponding node preferentially executes the batch deletion task, and locks the corresponding partition when executing the batch deletion task. Those skilled in the art can set the priority according to the actual situation. For example, the currently corresponding deletion task of the node can be forcibly cancelled, the corresponding partition can be locked, and each batch deletion task corresponding to the partition can be executed; When the locking strategy is in the optimistic mode, when the corresponding node executes the batch deletion task, only the corresponding graph data to be cleaned is locked.

[0042] In this embodiment, the optimistic mode is the default mode. When a certain partition will receive a large amount of graph data to be cleaned, by setting the locking strategy in the pessimistic mode, the corresponding node can avoid other interferences and quickly consume the graph data to be cleaned in the corresponding partition.

[0043] Refer to Figure 2 , Figure 2 is a schematic diagram of module connections of a device for batch cleaning data in a graph database provided by an exemplary embodiment. As Figure 2 shown, it includes: A cluster information acquisition module 100, configured to acquire the cluster information corresponding to the target cluster, where the cluster information includes the number of nodes, and the address information and partition information corresponding to each node; The cleaning information acquisition module 200 is configured to acquire a plurality of to-be-cleaned graph data including an external unique identifier; The address calculation module 300 is configured to calculate a target address corresponding to each piece of to-be-cleaned graph data based on the cluster information and the external unique identifier, where the target address is the node address corresponding to the to-be-cleaned graph data in the target cluster; specifically, based on the cluster information and the corresponding external unique identifier, determine the partition where the corresponding to-be-cleaned graph data is located, and use the node address corresponding to the partition as the target address corresponding to the to-be-cleaned graph data; The sending module 400 is configured to send the corresponding to-be-cleaned graph data to the corresponding node in the target cluster based on the target address, and the node batch-cleans the received to-be-cleaned graph data.

[0044] As an implementable manner, the cleaning information acquisition module 200 includes: The receiving unit 210 is configured to receive at least one piece of deletion data information input by a user, where the deletion data information includes mapping data and filtering conditions, and the mapping data is used to indicate the data source corresponding to the target deletion data, the location information of the target deletion data in the data source, and the graph data type corresponding to the target deletion data in the graph database; The analysis unit 220 is configured to determine the to-be-cleaned graph data corresponding to the target deletion data information, and it includes: The graph type recognition unit is configured to determine the graph data type of the target deletion data based on the corresponding mapping data, and the graph data type includes a point type and an edge type; The edge processing unit is configured to, when the graph data type of the target deletion data is the edge type, query the corresponding target edge from the corresponding data source based on the corresponding mapping data and filtering conditions, use the two vertices connected by the target edge as the positioning vertices, and generate the to-be-cleaned graph data corresponding to the target edge based on the external unique identifiers of the two positioning vertices; The point processing unit, when the graph data type of the target deletion data is the point type, queries the corresponding target point from the corresponding data source based on the corresponding mapping data and filtering conditions, generates the to-be-cleaned graph data corresponding to the target point based on the external unique identifier corresponding to the target point, and also uses the edge connecting the corresponding neighbor vertex to the target point as the target edge, and generates the to-be-cleaned graph data corresponding to the target edge based on the external unique identifier of the corresponding neighbor vertex.

[0045] As an implementable manner, the sending module 400 includes: The task generation unit 410 is configured to generate a batch deletion task including partition information based on the to-be-cleaned graph data located in the same partition; A task sending unit 420 is configured to directly send the generated batch deletion tasks to corresponding nodes in a target cluster based on corresponding target addresses.

[0046] As an implementable manner, the task generation unit 410 is further configured to determine whether the quantity of the to-be-cleaned graph data to be sent corresponding to each partition exceeds a preset quantity threshold; if so, generate a batch deletion task with a pessimistic mode locking strategy; if not, generate a batch deletion task with an optimistic mode locking strategy.

[0047] For the apparatus embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and for the relevant parts, please refer to the partial description of the method embodiment.

[0048] An embodiment of the present application further provides a cluster, which includes a plurality of storage nodes. The cluster may include a central node or may not include a central node (each storage node serves as a coordination node). Refer to Figure 3 , and each storage node will execute Figure 3 the schematic diagram of the method for batch cleaning data of the graph database shown in As Figure 3 shown, when performing batch cleaning data, each storage node executes the following steps: S500. Receive a cleaning task transmitted externally, where the cleaning task includes one or more pieces of to-be-cleaned graph data, and store the to-be-cleaned graph data in a cache; In this embodiment, the cleaning task is the batch deletion task generated in the above embodiment. The batch deletion task includes partition information and one or more pieces of to-be-cleaned graph data corresponding to the partition information. At this time, the corresponding storage node writes the to-be-cleaned graph data into the cache and identifies the corresponding partition based on the partition information.

[0049] S600. When all the cleaning tasks of the corresponding storage node are received, batch lock and clean the to-be-cleaned graph data in the cache based on partitions.

[0050] In this embodiment, each storage node processes the cached to-be-cleaned graph data. Through the design of the lock mechanism and cache mechanism local to the node, each partition manages the lock state within its own partition, enabling the node to perform parallel cleaning on the partition without lock conflicts based on the lock state, reducing data contention and conflicts, and accelerating data processing.

[0051] The method proposed in this embodiment does not limit the order of the to-be-cleaned graph data, and only needs to ensure that all the to-be-cleaned graph data are cleaned.

[0052] In a distributed scenario, the cost of locking is huge, and it is necessary to coordinate the data state of the entire cluster; When deleting a point or an edge, it is necessary to apply a lock to the data block of this point or edge within the partition and synchronize with other nodes at the same time. In the prior art, every time a point or an edge is deleted, a process of completing lock application and release will be constructed, resulting in redundancy and inefficiency in the deletion process. In this embodiment, by establishing a cache and batch-locking the graph data to be cleaned in the cache according to partitions, the number of times of repeatedly sending messages in the process is greatly reduced; In this embodiment, the graph data to be cleaned belonging to the same partition is batch-cleaned. At this time, the basic overhead required for cleaning (such as partition information acquisition, partition status update, etc.) only needs to be called once. Compared with cleaning each piece of data separately based on partitions, the cleaning efficiency can be effectively improved.

[0053] As an implementable manner, before all cleaning tasks corresponding to the storage node are received, it further includes a step of over-limit cleaning. The specific steps are as follows: When new graph data to be cleaned is added to the cache, or, based on a preset judgment period, it is judged whether the cache reaches a preset storage threshold. If so, based on a preset cleaning rule, the graph data to be cleaned corresponding to one or more partitions in the cache is batch-locked and cleaned based on partitions; Considering that there may be a situation where there is a large amount of graph data to be cleaned cached by a certain node, and when the cache is too large, it will trigger the current limiting of the partition network. In this embodiment, through a preset storage threshold and a cleaning rule, when the graph data to be cleaned in the cache exceeds the storage threshold, the graph data to be cleaned corresponding to one or more partitions is batch-cleaned based on partitions according to the cleaning rule; Those skilled in the art can set the cleaning rule according to actual needs. For example, based on partitions, the graph data to be cleaned corresponding to each partition is batch-cleaned in the order from large to small according to the quantity until the graph data to be cleaned in the cache reaches a preset value; or, when the quantity of the graph data to be cleaned corresponding to a partition exceeds a preset value, the graph data to be cleaned corresponding to this partition is batch-locked and cleaned. This specification does not limit it in detail.

[0054] As an implementable manner, the batch deletion task further includes a locking strategy; When the locking strategy is the pessimistic mode, the corresponding storage node writes the corresponding graph data to be cleaned into the cache, and after marking the corresponding partition based on the partition information, locks the partition based on the corresponding partition information, and batch-cleans each graph data to be cleaned corresponding to the partition in the cache; During this process, if the storage node receives a new batch deletion task, it judges whether the new batch deletion task belongs to the locked partition. When it belongs to the locked partition, it writes it into the cache, otherwise it writes it into the temporary cache until all the graph data to be cleaned corresponding to the partition is cleaned, and then converts the data in the temporary cache into the cache; When the locking policy is in the optimistic mode, the corresponding storage node writes the corresponding graph data to be cleaned into the cache, and identifies the corresponding partition based on the partition information. When it is determined that the cache reaches the preset storage threshold, the graph data to be cleaned in the cache is batch-locked and cleaned based on the preset storage threshold partition.

[0055] Cleaning is a write operation that requires locking the data to prevent interference from other requests. In this embodiment, through the design of the cache mechanism and lock mechanism on the node local, the contention and conflict of data are reduced, and the data processing is accelerated.

[0056] An embodiment of the present application further provides a system for batch cleaning data in a graph database. The graph database is distributedly deployed in a target cluster, and the graph database is partitioned based on the external unique identifier of the vertex. The system includes; A cleaning system and a cluster; The cleaning system is any one of the devices described in the above embodiments; The cluster is any one of the clusters provided in the above embodiments.

[0057] Each embodiment in this specification is described in a progressive manner. The key point of each embodiment is to illustrate the differences from other embodiments. The same or similar parts among the embodiments can be referred to each other.

[0058] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a device, or a computer program product. Therefore, the present invention can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0059] The present invention is described with reference to the flowcharts and / or block diagrams of methods, terminal devices (systems), and computer program products according to the present invention. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, and the combination of flows and / or blocks in the flowcharts and / or block diagrams can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate for implementing in the process Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks the device with the functions specified therein.

[0060] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to operate in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means that implement the function specified in one or more of the flows Figure 1 one or more flows and / or blocks Figure 1 specified in the block or blocks.

[0061] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the function specified in one or more of the flows Figure 1 one or more flows and / or blocks Figure 1 specified in the block or blocks.

[0062] It should be noted that: As used herein, the phrase "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present invention. Thus, the appearances of the phrase "one embodiment" or "an embodiment" in various places throughout this specification are not necessarily all referring to the same embodiment.

[0063] Although the preferred embodiments of the present invention have been described, additional changes and modifications can be made by those skilled in the art once they learn of the basic inventive concept. Therefore, the appended claims are intended to be construed to include the preferred embodiments as well as all changes and modifications that fall within the scope of the present invention.

[0064] In addition, it should be noted that the specific embodiments described in this specification may have different names and the like. Any equivalent or simple changes made in accordance with the structure, features, and principles described in the inventive concept of the present invention are included in the scope of protection of the present invention. Those skilled in the art to which the present invention pertains may make various modifications, additions, or substitutions in a similar manner to the specific embodiments described, as long as they do not depart from the structure of the present invention or exceed the scope defined by the appended claims, and they shall fall within the scope of protection of the present invention.

Claims

1. A method for batch cleaning data in a graph database, where the graph database is distributedly deployed in a target cluster, and the graph database is partitioned based on the external unique identifier of vertices, characterized in that, It includes the following steps: Obtain the cluster information corresponding to the target cluster, where the cluster information includes the number of nodes, as well as the address information and partition information corresponding to each node; Obtain several pieces of to-be-cleaned graph data containing external unique identifiers; Based on the cluster information and the external unique identifier, calculate the target address corresponding to each piece of to-be-cleaned graph data, where the target address is the node address corresponding to the to-be-cleaned graph data in the target cluster; Based on the target address, send the corresponding to-be-cleaned graph data to the corresponding node in the target cluster, and the node batch-cleans the received to-be-cleaned graph data.

2. The method for batch-cleaning data in a graph database according to claim 1, wherein: Receive at least one piece of deletion data information input by the user, where the deletion data information includes mapping data and filtering conditions, and the mapping data is used to indicate the data source corresponding to the target deletion data, the location information of the target deletion data in the data source, and the graph data type corresponding to the target deletion data in the graph database; Determine the to-be-cleaned graph data corresponding to the deletion data information, specifically: Based on the corresponding mapping data, determine the graph data type of the target deletion data, where the graph data type includes vertex type and edge type; When the graph data type of the target deletion data is edge type, based on the corresponding mapping data and filtering conditions, query the corresponding target edge from the corresponding data source, use the two vertices connected by the target edge as the positioning vertices, and generate the to-be-cleaned graph data corresponding to the target edge based on the external unique identifiers of the two positioning vertices; When the graph data type of the target deletion data is vertex type, based on the corresponding mapping data and filtering conditions, query the corresponding target vertex from the corresponding data source, generate the to-be-cleaned graph data corresponding to the target vertex based on the external unique identifier corresponding to the target vertex, and also use the edges connecting the corresponding neighbor vertices to the target vertex as the target edges, and generate the to-be-cleaned graph data corresponding to the target edges based on the external unique identifiers of the corresponding neighbor vertices.

3. The method for batch-cleaning data in a graph database according to claim 1 or 2, wherein: Based on the cluster information and the corresponding external unique identifier, determine the partition where the corresponding to-be-cleaned graph data is located, and use the node address corresponding to the partition as the target address corresponding to the to-be-cleaned graph data; Based on the to-be-cleaned graph data located in the same partition, generate a batch deletion task containing partition information, and directly send the batch deletion task to the corresponding node in the target cluster based on the corresponding target address.

4. The method for batch-cleaning data in a graph database according to claim 3, wherein: Judge whether the quantity of the to-be-sent to-be-cleaned graph data corresponding to each partition exceeds a preset quantity threshold; If so, generate a batch deletion task with a pessimistic locking strategy; If not, generate a batch deletion task with an optimistic locking strategy; When the locking strategy is pessimistic, the corresponding node preferentially executes the batch deletion task and locks the corresponding partition when executing the batch deletion task; When the locking strategy is in the optimistic mode, when the corresponding node executes the batch deletion task, it only locks the corresponding graph data to be cleaned up.

5. An apparatus for batch cleaning data in a graph database, where the graph database is distributedly deployed in a target cluster, and the graph database is partitioned based on the external unique identifier of vertices, characterized in that Including: A cluster information acquisition module, configured to acquire the cluster information corresponding to the target cluster, where the cluster information includes the number of nodes, as well as the address information and partition information corresponding to each node; A cleanup information acquisition module, configured to acquire a plurality of pieces of graph data to be cleaned up including external unique identifiers; An address calculation module, configured to calculate the target address corresponding to each piece of graph data to be cleaned up based on the cluster information and the external unique identifier, where the target address is the node address corresponding to the graph data to be cleaned up in the target cluster; A sending module, configured to send the corresponding graph data to be cleaned up to the corresponding node in the target cluster based on the target address, and the node batch-cleans the received graph data to be cleaned up.

6. A method for batch cleaning data in a graph database, where the graph database is distributedly deployed in a target cluster, and the graph database is partitioned based on the external unique identifier of vertices, and the target cluster includes several storage nodes, characterized in that, Each storage node executes the following steps: Receive a cleanup task transmitted externally, where the cleanup task includes one or more pieces of graph data to be cleaned up, and store the graph data to be cleaned up in a cache; When all the cleanup tasks of the corresponding storage node are received, batch-lock and clean up the graph data to be cleaned up in the cache based on partitions.

7. A method for batch cleaning data in a graph database according to claim 6, characterized in that, Before all the cleanup tasks of the corresponding storage node are received, it further includes a step of over-limit cleanup, and the specific steps are: When new graph data to be cleaned up is added to the cache, or, based on a preset judgment period, it is judged whether the cache reaches a preset storage threshold. If so, based on a preset clearing rule, batch-lock and clean up the graph data to be cleaned up corresponding to one or more partitions in the cache based on partitions.

8. The method for batch-cleaning data in a graph database according to claim 6 or 7, wherein: The cleanup task is a batch deletion task, and the batch deletion task includes partition information, a locking strategy, and one or more pieces of graph data to be cleaned up corresponding to the partition information; When the locking strategy is in the pessimistic mode, the corresponding storage node writes the graph data to be cleaned up into the cache, and after identifying the corresponding partition based on the partition information, locks the partition based on the corresponding partition information, and batch-cleans each piece of graph data to be cleaned up corresponding to the partition in the cache; When the locking strategy is in the optimistic mode, the corresponding storage node writes the graph data to be cleaned up into the cache, and based on the partition identified by the partition information, when it is judged that the cache reaches a preset storage threshold, batch-lock and clean up the graph data to be cleaned up in the cache based on partitions.

9. A cluster, where the target graph database is distributedly deployed in the cluster, and the target graph database is partitioned based on the external unique identifier of vertices. The cluster includes several storage nodes, characterized in that, Each storage node performs batch cleanup according to the method described in any one of claims 6 to 8.

10. A system for batch cleaning data in a graph database, where the graph database is distributedly deployed in a target cluster, and the graph database is partitioned based on the external unique identifier of vertices, characterized in that Including; A cleanup system and a cluster; The cleanup system is the device described in claim 5; The cluster is the cluster described in claim 9.

Images