Method and device for processing graph data, electronic equipment and storage medium

Abstract

The present disclosure provides a graph data processing method and device, electronic equipment and storage medium, relates to the field of artificial intelligence, in particular to the field of deep learning and knowledge graph. The specific implementation scheme is: determining the task type of the to-be-processed graph data; transmitting the graph data to a first graph system matched with the task type; obtaining a processing result output by the first graph system, wherein the processing result is obtained by the first graph system performing a processing task corresponding to the task type on the graph data; and synchronizing the processing result to a second graph system matched with the processing result.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of artificial intelligence, in particular to a graph data processing method and device, an electronic device and a storage medium in the field of deep learning and knowledge graph. BACKGROUND

[0002] At present, in the graph-oriented technical architecture, several data processing systems are independent of each other, and a user needs to use corresponding systems to process data respectively in the process of data processing, and it is difficult to simultaneously consider multiple types of task processing on the same system. SUMMARY

[0003] The present disclosure provides a graph data processing method, device, electronic device and storage medium.

[0004] According to an aspect of the present disclosure, a graph data processing method is provided. The method comprises: determining a task type of graph data to be processed; transmitting the graph data to a first graph system matched with the task type; obtaining a processing result output by the first graph system, wherein the processing result is obtained by the first graph system performing a processing task corresponding to the task type on the graph data; and synchronizing the processing result to a second graph system matched with the processing result.

[0005] According to another aspect of the present disclosure, a graph data processing device is provided. The device comprises: a determination unit configured to determine a task type of graph data to be processed; a transmission unit configured to transmit the graph data to a first graph system matched with the task type; an obtaining unit configured to obtain a processing result output by the first graph system, wherein the processing result is obtained by the first graph system performing a processing task corresponding to the task type on the graph data; and a synchronization unit configured to synchronize the processing result to a second graph system matched with the processing result.

[0006] According to another aspect of the present disclosure, an electronic device is also provided. The electronic device can include at least one processor, and a memory connected with the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the graph data processing method of the embodiments of the present disclosure.

[0007] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium having computer instructions is also provided, wherein the computer instructions are used to enable a computer to perform the graph data processing method of the embodiments of the present disclosure.

[0008] According to another aspect of the present disclosure, a computer program product can include a computer program, and the computer program, when executed by a processor, implements the graph data processing method of the embodiments of the present disclosure.

[0009] It should be understood that the contents described in this part are not intended to identify key or important features of the embodiments of the present disclosure, nor to limit the scope of the present disclosure. Other features of the present disclosure will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0010] The accompanying drawings are used to better understand the present scheme and do not limit the present disclosure. Among them:

[0011] Figure 1 is a flow chart of a graph data processing method according to an embodiment of the present disclosure;

[0012] Figure 1 (a) is a flow chart of synchronizing processing results to a second graph system matching the processing results according to an embodiment of the present disclosure;

[0013] Figure 1 (b) is a flow chart of converting processing results into streaming data according to an embodiment of the present disclosure;

[0014] Figure 1 (c) is a flow chart of obtaining synchronization strategy data of processing results according to an embodiment of the present disclosure;

[0015] Figure 2 is a schematic diagram of a technical architecture of a graph data processing method according to an embodiment of the present disclosure;

[0016] Figure 3 is a flow chart of streaming synchronization according to an embodiment of the present disclosure;

[0017] Figure 4 is a schematic diagram of data partitioning according to an embodiment of the present disclosure;

[0018] Figure 5 is a schematic diagram of a graph data processing apparatus according to an embodiment of the present disclosure;

[0019] Figure 6 is a block diagram of an electronic device of a graph data processing method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0020] Exemplary embodiments of the present disclosure are described below with reference to the accompanying drawings, which include various details of the embodiments of the present disclosure to help understanding, and should be considered as merely exemplary. Therefore, those of ordinary skill in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Also, in order to be clear and concise, the description below omits the description of well-known functions and structures.

[0021] Figure 1is a flowchart of a method for processing graph data according to an embodiment of the present disclosure. As shown in Figure 1 The method can include the following steps:

[0022] In step S102, the task type of the graph data to be processed is determined.

[0023] In the technical solution provided by step S102 of the present disclosure, the graph data to be processed is obtained, and the task type of the graph data to be processed is determined, wherein the task type can include reading data or writing data, transaction data or non-transaction data, reading full-range data or reading small-range data.

[0024] In related technologies, in a graph-oriented technical architecture, including a graph data system, a graph computing system and a graph learning system, because there are great differences among the three in task type, when facing the three problems, three different systems are needed to solve them respectively, resulting in great resource cost, operation and maintenance cost and learning and using cost in the process of using the system. Embodiments of the present disclosure propose a unified architecture, determine the task type of the graph data to be processed, so as to achieve the purpose of solving multiple technical tasks in one system.

[0025] In step S104, the graph data is transmitted to a first graph system matched with the task type.

[0026] In the technical solution provided by step S104 of the present disclosure, the graph data is transmitted to a first graph system matched with the task type based on the data type corresponding to the graph data, wherein the first graph system can be a graph database system or a graph computing system.

[0027] For example, the graph database architecture can be set to complete the transactional read-write requirement, the graph computing system can be set to complete the non-transactional retrieval requirement involving a large amount of data, and the sampling stage of the graph learning system can sample data from the memory storage of the graph computing system, the sampling process can be regarded as a task type of the graph computing system, and can be submitted to the first graph system for completion.

[0028] In step S106, the processing result output by the first graph system is obtained, wherein the processing result is obtained by the first graph system performing a processing task corresponding to the task type on the graph data.

[0029] In the technical solution provided by step S106 of the present disclosure, the first graph system performs a processing task corresponding to the task type on the graph data, thereby obtaining the processing result output by the first graph system, wherein the processing task can be recorded and displayed by an execution log, and the processing result can be obtained by the first graph system performing a processing task corresponding to the task type on the graph data, for example, the data change content of the first graph system.

[0030] Step S108: Synchronize the processing results to the second graph system that matches the processing results.

[0031] In the technical solution provided by step S108 of this disclosure, the processing result is synchronized to the second graph system, wherein the second graph system can be a graph database system or a graph computing system.

[0032] Optionally, the graph data is transmitted to a first graph system that matches the task type of the graph data. The first graph system performs processing tasks on the graph data and obtains the processing results output by the first graph system. Through streaming data synchronization, the processing results can be synchronized to a second graph system that matches the processing results.

[0033] Optionally, the second graph system can actively retrieve the processing results of the first graph system from the storage of the first graph system, and re-execute the processing results in the second graph system, thereby achieving the purpose of synchronizing the processing results to the second graph system that matches the processing results.

[0034] In this embodiment of the disclosure, data synchronization is achieved automatically between the storage of the first graph system and the second graph system, thereby ensuring the real-time performance of the graph data.

[0035] Through steps S102 to S108, the task type of the graph data to be processed is determined; the graph data is transferred to a first graph system matching the task type; the processing result output by the first graph system is obtained, wherein the processing result is obtained by the first graph system performing the processing task corresponding to the task type on the graph data; and the processing result is synchronized to a second graph system matching the processing result. In other words, this disclosure, by comprehensively considering the impact of different task types on the system architecture design, enables different graph systems to transfer data to each other, thereby achieving automatic data synchronization, improving the technical effect of graph data processing efficiency, and solving the technical problem of low graph data processing efficiency.

[0036] The method described in this embodiment will now be described in further detail.

[0037] As an optional implementation, step S108, synchronizing the processing result to a second graph system that matches the processing result, includes: converting the processing result into streaming data; and synchronizing the streaming data to a second graph system that matches the streaming data.

[0038] Figure 1 (a) is a flowchart of an embodiment of the present disclosure of synchronizing processing results to a second graph system that matches the processing results, such as... Figure 1 As shown in (a), the method may include the following steps:

[0039] Step S1081: Convert the processing result into streaming data.

[0040] In the technical solution provided by step S1081 of this disclosure, the processing result can be converted into streaming data, wherein the streaming data can be real-time big data.

[0041] Step S1082: Synchronize the streaming data to a second graph system that matches the streaming data.

[0042] In the technical solution provided by step S1082 of this disclosure, the streaming data is synchronized to a second graph system that matches the streaming data.

[0043] Optionally, the processing results can be converted into streaming data. The second graph system can actively pull the processing results from the first graph system and reposition the processing results on the second graph system, thereby achieving the purpose of synchronizing the streaming data to the second graph system that matches the streaming data.

[0044] In this embodiment of the disclosure, synchronization is achieved through streaming data, enabling data processing based on real-time data streams. This allows for time spans from hundreds of milliseconds to several seconds, thus ensuring the timeliness of data storage.

[0045] As an optional implementation, converting the processing result into streaming data includes: acquiring synchronization strategy data of the processing result, wherein the synchronization strategy data is used to characterize the synchronization period when synchronizing the processing result and the amount of data of the processing result synchronized within the synchronization period; and determining the streaming data based on the synchronization strategy data.

[0046] Figure 1 (b) is a flowchart of converting processing results into streaming data according to an embodiment of the present disclosure, such as... Figure 1 As shown in (b), the method may include the following steps:

[0047] Step S1083: Obtain the synchronization strategy data of the processing results, wherein the synchronization strategy data is used to characterize the synchronization period when synchronizing the processing results and the amount of data of the processing results synchronized within the synchronization period.

[0048] In the technical solution provided by step S1083 of this disclosure, synchronization strategy data of the processing result is obtained. The synchronization strategy data can be a streaming synchronization strategy, which can be used to characterize the synchronization period when synchronizing the processing result and the amount of data of the processing result synchronized within the synchronization period.

[0049] Step S1084: Determine streaming data based on synchronization strategy data.

[0050] In the technical solution provided by step S1084 of this disclosure, synchronization strategy data of the processing result is obtained, and streaming data is determined based on the synchronization strategy data.

[0051] Optionally, data synchronization between the first and second graph systems can be achieved through streaming. Since different tasks have different requirements for data real-time performance, different streaming synchronization strategies can be set according to actual needs. For example, it can be time-based synchronization every minute / hour / day, or data change-based synchronization every 10 / 100 / 10000 data updates, or a combination of time-based and data-based synchronization.

[0052] This embodiment of the disclosure achieves the goal of meeting the real-time requirements of different task data by processing the processing results according to different synchronization strategies.

[0053] As an optional implementation, the synchronization strategy data for obtaining the processing results includes: determining the scene data from which the graph data originates; and determining the synchronization strategy data based on the scene data.

[0054] Figure 1 (c) is a flowchart of a synchronization strategy data for obtaining processing results according to an embodiment of the present disclosure, such as... Figure 1 As shown in (c), the method may include the following steps:

[0055] Step S1085: Determine the scene data from which the graph data originates.

[0056] In the technical solution provided by step S1085 of this disclosure, the scene data from which the graph data corresponding to the processing result is determined is determined, wherein the scene data can be used to represent different services.

[0057] Step S1086: Determine synchronization strategy data based on scenario data.

[0058] In the technical solution provided by step S1086 of this disclosure, the scene data from which the graph data corresponding to the processing result is determined is determined, and a synchronization strategy is determined based on the scene data. The scene data can be used to represent different services.

[0059] Optionally, when graph data is processed in batches for synchronization, different services have different requirements for the real-time performance of the graph data, and different streaming synchronization strategies can be supported. For example, when the real-time requirements are high, a synchronization strategy with a higher data synchronization frequency (e.g., every minute) can be selected; when the real-time requirements are low, a synchronization strategy with a lower data synchronization frequency (e.g., every hour, every day) can be selected.

[0060] In this embodiment of the disclosure, different synchronization strategies are used depending on the different scene data from which the graph data originates, so that the synchronization of streaming data ensures that the graph system can meet different levels of data real-time performance.

[0061] As an optional implementation, step S108, synchronizing the processing results to the second graph system that matches the processing results, includes: synchronizing the processing results to multiple target segments in the second graph system based on the task type, wherein the multiple target segments are independent of each other.

[0062] In this embodiment of the disclosure, the graph data is synchronized according to the target segments. The processing results are synchronized to multiple target segments in the second graph system based on the task type. The multiple target segments are independent of each other, and each target segment has a separate streaming synchronization channel.

[0063] In this embodiment of the disclosure, data synchronization is performed according to target shards, each target shard has a separate streaming synchronization channel, and the channels of different target shards are independent of each other, thereby realizing the ability to deploy synchronization channels according to the server architecture, which has a more flexible technical effect.

[0064] As an optional implementation, multiple original fragments of the second graph system are repartitioned to obtain multiple target fragments.

[0065] In this embodiment, different graph computing tasks have different requirements for graph data distribution and algorithm performance. Based on the task type, the processing results are synchronized to multiple target partitions in the second graph system. During the process of synchronizing the processing results to multiple target partitions in the second graph system, multiple original partitions of the second graph system are repartitioned to obtain multiple target partitions. The repartitioning can be performed according to the algorithm strategy corresponding to different graphs to meet the requirements of different graph data processing tasks.

[0066] For example, based on whether the edge attributes in the graph data are stored as source nodes or target nodes, multiple original fragments of the second graph system can be repartitioned according to the storage conditions to obtain multiple target fragments.

[0067] For example, based on the relationship information of the points, whether it is outgoing edge data or incoming edge data, multiple target partitions can be obtained, thereby achieving the purpose of repartitioning multiple original partitions of the second graph system.

[0068] Optionally, after repartitioning multiple original fragments of the second graph system, when data retrieval is performed again, the repartition calculation can be quickly performed using the address field of the relation index and the address field in the edge attribute to meet higher performance requirements. For example, the fragment positions of the source and target points of the edge can be known based on the relation index of the edge, and then the edge attribute data can be sent to the fragment where the source or target point is located according to the synchronization strategy data to achieve the purpose of fast data retrieval. At the same time, useless data such as address mapping relationships and attribute index data can be directly filtered out.

[0069] It should be noted that graph data differs significantly from typical tabular data. In a table, each data entry is independent and consists of different rows, while graph data is built upon relationships, i.e., the connection information between different nodes. Therefore, in a distributed scenario, the graph partitioning state has a performance impact on different computing tasks. This embodiment of the present disclosure repartitions multiple original partitions to obtain multiple target partitions, thereby achieving the technical effect of improving data processing efficiency.

[0070] As an optional implementation, in step S104, the amount of graph data is greater than a first data amount threshold. The step of transmitting the graph data to a first graph system that matches the task type includes transmitting the graph data to a graph computing system that matches the task type. The graph computing system is used to perform graph computing and / or graph retrieval on the graph data to obtain processing results. The first graph system includes the graph computing system.

[0071] In this embodiment of the disclosure, the amount of graph data involves a large amount of data, such as full graph data. Therefore, the amount of graph data is greater than the first data volume threshold. For example, offline full graph computation tasks and deep / complex graph retrieval tasks will have a large amount of data. The first data volume threshold can be the data volume threshold of the first graph system.

[0072] Optionally, the graph data is transmitted to a graph computing system that matches the task type. The graph computing system performs graph computing and / or graph retrieval on the graph data to obtain the processing results. The first graph system includes the graph computing system.

[0073] In this embodiment, the graph computing architecture execution engine can perform offline full graph computing tasks, such as pagerank, or it can perform deep or complex graph retrieval tasks, such as finding out which of a person's wife's friends work for a certain company and returning the insurance purchase information of these friends and their families.

[0074] Optionally, the data in the graph computing architecture execution engine can be stored in memory in the form of in-memory graph storage. The graph computing system is used to perform graph computing and / or graph retrieval on the graph data to obtain the processing results, thereby improving computing performance without considering data reliability.

[0075] As an optional implementation, step S108, synchronizing the processing result to a second graph system that matches the processing result, includes: synchronizing the processing result to a graph database system that matches the processing result, wherein the graph database system is used to store the processing result on a disk, and the second graph system includes the graph database system.

[0076] In this embodiment of the disclosure, the processing result can be synchronized to the disk storage in a graph database system that matches the processing result. The second graph system may include a graph database system, which can be memory storage or disk storage. During storage, the reliability of the data can be ensured by multiple backups, redundant data storage, etc.

[0077] For example, in a graph database system, the disk storage actively pulls the processing results from the in-memory graph storage in the graph computing system, and then relocates or re-executes the processing results on the disk storage, thereby synchronizing the processing results to the graph database system that matches the processing results.

[0078] As an optional implementation, in step S104, the amount of graph data is less than the second data amount threshold. The process of transmitting the graph data to a first graph system that matches the task type includes transmitting the graph data to a graph database system that matches the task type. The graph database system is used to retrieve and / or update the graph data to obtain processing results. The first graph system includes a graph database system.

[0079] In this embodiment of the disclosure, the graph data is transmitted to a graph database system that matches the task type. The graph database system retrieves and / or updates the graph data to obtain the processing result. The first graph system may include the graph database system.

[0080] Optionally, the graph database system can use disk storage, does not need to be specially designed for deep / complex graph retrieval tasks, and can be transaction-oriented for reading and writing. Transaction-oriented reading and writing only involves a small portion of the data, so the amount of graph data is less than the second data volume threshold.

[0081] Optionally, data in the graph data system can be stored on disk in the form of disk graph storage, without the need for special design for deep or complex graph retrieval tasks. In terms of design, it can be oriented only towards transactional read and write, thereby avoiding the design conflict problem that graph databases need to consider both transactional read and write tasks and deep / complex retrieval tasks at the same time.

[0082] As an optional implementation, step S108, synchronizing the processing result to a second graph system that matches the processing result, includes: synchronizing the processing result to a graph computing system that matches the processing result, wherein the graph computing system is used to store the processing result in memory, and the second graph system includes the graph computing system.

[0083] In this embodiment of the disclosure, the graph computing system is used to store the processing results in memory, which can be a memory graph storage. The processing results can be synchronized to the memory storage of a graph computing system that matches the processing results. The second graph system may include the graph computing system.

[0084] For example, in a graph computing system, the memory storage actively pulls the processing results from the disk storage of the graph database system, and then repositions or re-executes the processing results in the memory storage, thereby synchronizing the processing results to the graph computing system that matches the processing results.

[0085] As an optional implementation, the processing results from memory are determined as sampled data; a graph learning system is trained based on the sampled data.

[0086] In this embodiment, the task in a typical graph learning architecture includes two stages: "sampling to training". Training can refer to the model training process of a deep graph learning task. The training process of a graph model is the same as that of a deep learning model, except that the model being trained is different. In this embodiment, the processing results from memory can be determined as sampled data, and a graph learning system can be trained based on the sampled data.

[0087] Alternatively, data sampling can be performed on the graph data using a random walk method, for example, sampling can be performed on an in-memory graph storage.

[0088] Alternatively, random walks can be used to sample a portion of the data in a random manner, thereby reducing the amount of data required for training and enabling the actual training process to be completed quickly without affecting the results. For example, different random strategies or parameters can be selected.

[0089] Optionally, sampling can be submitted as a task of the graph computing architecture to the intermediate graph computing architecture execution engine for completion. Therefore, the graph learning architecture can also guarantee the real-time performance of data at different levels.

[0090] In this embodiment of the disclosure, by comprehensively considering the impact of different task types on the system architecture design, data can be transmitted between different graph systems, thereby achieving automatic data synchronization, which improves the efficiency of graph data processing and solves the technical problem of low graph data processing efficiency.

[0091] The above technical solutions of the present disclosure will be further illustrated below with reference to preferred embodiments.

[0092] In related technologies, graph-oriented technical systems, including graph database systems, graph computing systems, and graph learning systems, differ significantly in task types, such as read or write, transactional or non-transactional, and reads of the entire graph or small-scale data. Therefore, when addressing these three types of problems, they are solved by using three different types of systems, resulting in resource costs, operation and maintenance costs, and learning and usage costs during use.

[0093] In related technologies, the three systems are independent of each other, and data processing tasks can be completed by using the three systems separately. However, if data transfer between the three tasks is involved, data needs to be manually transferred between the systems through manual import and export. Therefore, this method leads to high resource costs, operation and maintenance costs, and costs of learning and using the systems. Furthermore, under deep and complex retrieval requirements, the graph database system needs to consume a large amount of cluster resources, which can cause other normal resource retrieval or update requests to be blocked and unable to be completed.

[0094] Based on this, some technical solutions integrate graph computing and some complex graph database retrieval functions into the technical framework of the graph computing system. However, although this method satisfies different types of retrieval needs at the data reading level, it cannot effectively support data changes, the real-time nature of the data cannot be met, and the data in the system is static data, which has the problem of not supporting data changes.

[0095] To address the aforementioned issues, this disclosure proposes a unified architecture to solve the three types of tasks within a single architecture. Taking into account the impact of different task types on the system architecture design, it achieves automatic synchronization of data between memory storage and disk storage through streaming data synchronization, ensuring a certain degree of real-time performance for the graph computing system architecture data. By splitting deep or complex graph database retrieval requirements into a unified scheduling and execution mechanism within the graph computing system architecture, better performance and improved resource utilization can be achieved.

[0096] In this embodiment of the disclosure, the graph database system fulfills transactional read and write requirements; the graph computing system fulfills non-transactional retrieval requirements involving large amounts of data; and the sampling phase of the graph learning system samples data from the memory storage of the graph computing system, with the sampling process being submitted to the graph computing system as a task to be completed.

[0097] Figure 2 This is a schematic diagram of the technical architecture of a graph data processing method according to an embodiment of the present disclosure, such as... Figure 2 As shown, the graph database system based on streaming synchronization in this embodiment may include a graph learning system, a graph computing system execution engine (OLAP execution engine), and a graph data system execution engine (OLTP execution engine). Streaming data can synchronously ensure that the OLAP architecture can meet different levels of data real-time requirements, while also eliminating the need to consider write (update) tasks and high data reliability.

[0098] In this embodiment, the graph computing system execution engine can perform offline full graph computing tasks, such as pagerank, or it can perform deep or complex graph retrieval tasks, such as finding out which of a person's wife's friends work for a certain company and returning the insurance purchase information of these friends and their families.

[0099] Optionally, the data in the execution engine of the graph computing system can be stored in memory in the form of in-memory graph storage, thereby improving computing performance without considering data reliability.

[0100] Optionally, the data in the graph data system execution engine can be stored on a disk in the form of disk graph storage. It does not require special design for depth or complex graph retrieval tasks. The design is oriented towards transactional read and write, and the reliability of the data can be ensured through multiple backups and redundant storage of data.

[0101] In this embodiment, a graph learning system is used to train the model.

[0102] Optionally, a task in a graph learning system typically includes two stages: "sampling to training". Training can refer to the model training process for a deep graph learning task. The training process for graph models is the same as that for deep learning models, only the models being trained are different.

[0103] Alternatively, sampling can be performed on graph data using random walks, for example, on a memory graph.

[0104] Alternatively, random walks can be used to sample a portion of the data in a random manner, thereby reducing the amount of data required for training and enabling the actual training process to be completed quickly without affecting the results. For example, different random strategies or parameters can be selected.

[0105] Optionally, sampling is submitted as a task of a graph computation system to an intermediate graph computation system execution engine for completion.

[0106] In this embodiment, the synchronization between the graph computing system execution engine and the graph data system execution engine can be accomplished through minibatch synchronization of streaming data. For example, the in-memory graph storage actively pulls the execution logs from the disk graph storage and replays (re-executes) the execution logs on the in-memory graph storage to reflect the data changes on the in-memory graph.

[0107] Figure 3 This is a flowchart of a streaming synchronization according to an embodiment of the present disclosure, such as... Figure 3 As shown, streaming synchronization may include the following steps:

[0108] Step S301: Determine whether the difference in the execution log exceeds the threshold.

[0109] The memory graph storage actively pulls the execution logs from the disk graph storage, and determines whether the difference between the execution logs in the memory graph storage and the execution logs in the disk graph storage exceeds a threshold. If it does, proceed to step S302; otherwise, proceed to step S306.

[0110] Step S302: Retrieve all data.

[0111] If the difference between the execution logs in the memory graph storage and the execution logs in the disk graph storage exceeds a threshold, then the memory graph storage will fetch a full snapshot of the data.

[0112] Step S303: Reload all data.

[0113] The in-memory graph storage pulls the full amount of data and reloads it into the graph computing system's execution engine.

[0114] Step S304: Update local log information.

[0115] Step S305 involves reloading all data into the graph computing system's execution engine and updating the local log information, followed by waiting for the synchronization cycle.

[0116] Waiting for the synchronization cycle, thereby achieving the goal of reflecting data changes on the memory graph.

[0117] Step S306: Retrieve the execution log from the disk graph storage.

[0118] Determine if the difference between the execution logs in the memory graph storage and the execution logs in the disk graph storage exceeds a threshold. If it does not exceed the threshold, the memory graph storage will actively retrieve the execution logs from the disk graph storage.

[0119] Step S307: Replay the log or pre-clean the data.

[0120] The memory graph storage actively pulls the execution logs from the disk graph storage and replays (re-executes) or pre-compacts the data on the memory graph storage.

[0121] In this embodiment, online and offline tasks are isolated on the system, and streaming synchronization supports data synchronization at different granularities. For example, if synchronization is performed once per hour, the changed data within each hour will be pre-organized and batch-updated to the in-memory graph storage.

[0122] Step S308: Batch write to the memory graph storage.

[0123] The memory graph storage actively pulls the execution log from the disk graph storage and replays (re-executes) the execution log on the memory graph storage to reflect data changes on the memory graph.

[0124] Optionally, considering that different businesses have different requirements for the real-time data in the graph computing system's execution engine, different streaming synchronization strategies can be supported. For example, synchronization can be performed by time, such as every minute / hour / day, or by data change, such as every 10 / 100 / 10000 data updates, or a combination of time-based and data-based synchronization. This is just an example and no specific restrictions are imposed.

[0125] In this embodiment, the update is completed in the graph data system, and the reliability of the data is also guaranteed by the disk graph storage through backup. The data in the storage graph storage is synchronized in batches. If the real-time requirement is high, the frequency of data synchronization can be higher (e.g., every minute). If the real-time requirement is low, the frequency of data synchronization can be lower (e.g., every hour or every day). If most of the data is lost, the data can be synchronized again from the disk graph storage.

[0126] As an optional implementation, the embodiments of this disclosure can be applied to the database of a distributed graph data system with sharded storage. That is, data synchronization is performed according to shards, each shard has a separate streaming synchronization channel, and the channels of different shards are independent of each other. Synchronization channels can be deployed according to the server system, which is more flexible.

[0127] As an optional implementation, this disclosure embodiment is designed for graph scenarios. Different OLAP graph computing tasks have different requirements for the distribution of graph data than OLTP. When replaying logs in the above process, the data can be repartitioned according to different graph algorithm strategies to meet the performance requirements of different graph computing algorithms and thus improve the efficiency of data retrieval.

[0128] Figure 4 This is a schematic diagram of a data partition according to an embodiment of the present disclosure, such as... Figure 4 As shown, data can be partitioned and stored according to corresponding attributes. For example, edge attributes can be partitioned and stored according to the source node (source point) or the target node (target point); the relationship information of the points can record whether the edge data is outgoing or incoming, etc.

[0129] It should be noted that related technologies typically use tabular data for storage, but there are significant differences between graph and tabular data. Each data entry in a table is independent and consists of different rows, while graph data itself relies on relationships, that is, it is a data model built through the connection information between different nodes. Therefore, in a distributed scenario, the graph partitioning state has a performance impact on different computing tasks.

[0130] In this embodiment, the graph can be stored in OLTP according to the following structure.

[0131] Optionally, during the synchronization process, repartitioning can be quickly calculated using the port address (vlan identitydocument, or vid) of the relation index and the address (identity document, or id) field in the edge attribute to meet higher performance requirements. For example, the sharding positions of the source node and target node of the edge can be determined based on the relation index of the edge, and then the edge attribute data can be sent to the shard where the source node or target node is located according to the pre-set strategy.

[0132] Optionally, data that is useless to OLAP tasks, such as address mapping relationships, attributes, and index data, can be directly filtered out.

[0133] Transactional read and write operations typically involve a small amount of data and have explicit requirements for data isolation. However, deep / complex retrieval tasks usually involve a large amount of data, such as the entire graph data, and have no explicit or very low requirements for data isolation. Therefore, there is a conflict between the two in the processing. In this embodiment of the disclosure, online and offline tasks are isolated on the system, which relieves the design conflict of graph databases having to consider both transactional read and write tasks and deep / complex retrieval tasks at the same time.

[0134] In this embodiment of the disclosure, by comprehensively considering the impact of different task types on the system architecture design, data can be transmitted between different graph systems, thereby achieving automatic data synchronization, which improves the efficiency of graph data processing and solves the technical problem of low graph data processing efficiency.

[0135] This disclosure also provides an embodiment for performing Figure 1 The illustrated embodiment is a graph data processing apparatus for a graph data processing method.

[0136] Figure 5 This is a schematic diagram of a graph data processing apparatus according to an embodiment of the present disclosure, such as... Figure 5 As shown, the data processing device 40 may include: a determining unit 51, a transmission unit 52, an acquisition unit 53, and a synchronization unit 54.

[0137] The determination unit 51 is used to determine the task type of the graph data to be processed.

[0138] The transmission unit 52 is used to transmit graph data to a first graph system that matches the task type.

[0139] The acquisition unit 53 is used to acquire the processing result output by the first graph system, wherein the processing result is obtained by the first graph system performing a processing task corresponding to the task type on the graph data.

[0140] Synchronization unit 54 is used to synchronize the processing results to a second graph system that matches the processing results.

[0141] Optionally, the synchronization unit 54 includes: a processing module for converting the processing result into streaming data; and synchronizing the streaming data to a second graph system that matches the streaming data.

[0142] Optionally, the processing module includes: an acquisition submodule, used to convert the processing results into streaming data, including: acquiring synchronization strategy data of the processing results, wherein the synchronization strategy data is used to characterize the synchronization period when synchronizing the processing results and the amount of data of the processing results synchronized within the synchronization period; and determining the streaming data based on the synchronization strategy data.

[0143] Optionally, the processing module includes: a determination submodule, used to determine the scene data from which the graph data originates; and to determine synchronization strategy data based on the scene data.

[0144] Optionally, the synchronization unit 54 includes: a first synchronization module, used to synchronize the processing results to multiple target segments in the second graph system based on the task type, wherein the multiple target segments are independent of each other.

[0145] Optionally, the device further includes a processing unit for repartitioning multiple original fragments of the second graph system to obtain multiple target fragments.

[0146] Optionally, the transmission unit 52 includes: a transmission module for transmitting graph data to a graph computing system that matches the task type, wherein the graph computing system is used to perform graph computing and / or graph retrieval on the graph data to obtain processing results, and the first graph system includes the graph computing system.

[0147] Optionally, the synchronization unit 54 includes: a second synchronization module, used to synchronize the processing results to a graph database system that matches the processing results, wherein the graph database system is used to store the processing results on a disk, and the second graph system includes a graph database system.

[0148] Optionally, the second synchronization module includes a transmission submodule for transmitting graph data to a graph database system that matches the task type, wherein the graph database system is used to retrieve and / or update the graph data to obtain processing results, and the first graph system includes the graph database system.

[0149] Optionally, the synchronization unit 54 includes: a third synchronization module for synchronizing the processing result to a graph computing system that matches the processing result, wherein the graph computing system is used to store the processing result in memory, and the second graph system includes the graph computing system.

[0150] Optionally, the device further includes: a determination unit for determining the processing results from memory as sampled data; and training a graph learning system based on the sampled data.

[0151] In the apparatus of this disclosed embodiment, a determining unit determines the task type of the graph data to be processed; a transmission unit transmits the graph data to a first graph system matching the task type; an acquiring unit acquires the processing result output by the first graph system, wherein the processing result is obtained by the first graph system performing a processing task corresponding to the task type on the graph data; and a synchronization unit synchronizes the processing result to a second graph system matching the processing result. In other words, this disclosed embodiment, by comprehensively considering the impact of different task types on the system architecture design, enables different graph systems to transmit data to each other, thereby achieving automatic data synchronization and improving the technical effect of high graph data processing efficiency, thus solving the technical problem of low graph data processing efficiency.

[0152] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0153] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0154] Embodiments of this disclosure provide an electronic device that may include: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the graph data processing method of the embodiments of this disclosure.

[0155] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0156] According to embodiments of this disclosure, this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause a computer to execute a graph data processing method according to embodiments of this disclosure.

[0157] Optionally, in this embodiment, the aforementioned non-transitory computer-readable storage medium may be configured to store a computer program for performing the following steps:

[0158] S1, Determine the task type of the graph data to be processed;

[0159] S2, transfer the graph data to the first graph system that matches the task type;

[0160] S3, obtain the processing result output by the first graph system, wherein the processing result is obtained by the first graph system performing the processing task corresponding to the task type on the graph data;

[0161] S4 synchronizes the processing results to the second graph system that matches the processing results.

[0162] Optionally, in this embodiment, the aforementioned non-transitory computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. More specific examples of readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0163] According to embodiments of this disclosure, this disclosure also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0164] S1, Determine the task type of the graph data to be processed;

[0165] S2, transfer the graph data to the first graph system that matches the task type;

[0166] S3, obtain the processing result output by the first graph system, wherein the processing result is obtained by the first graph system performing the processing task corresponding to the task type on the graph data;

[0167] S4 synchronizes the processing results to the second graph system that matches the processing results.

[0168] Figure 6 This is a block diagram of an electronic device for processing graph data according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0169] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded from storage unit 608 into random access memory (RAM) 603. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0170] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0171] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as method data processing methods. For example, in some embodiments, the method data processing method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform data processing methods by any other suitable means (e.g., by means of firmware).

[0172] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0173] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0174] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0175] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0176] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0177] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0178] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0179] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for processing graph data, comprising: Determine the task type of the graph data to be processed; The graph data is transmitted to a first graph system that matches the task type. Obtain the processing result output by the first graph system, wherein the processing result is obtained by the first graph system performing the processing task corresponding to the task type on the graph data; The processing result is synchronized to a second graph system that matches the processing result. The synchronization between the first graph system and the second graph system is accomplished based on a streaming synchronization strategy, which includes at least synchronization based on time and / or synchronization based on data change amount. Synchronizing the processing results to a second graph system that matches the processing results includes: synchronizing the processing results to multiple target fragments in the second graph system based on the task type, wherein the multiple target fragments are independent of each other; repartitioning multiple original fragments in the second graph system to obtain the multiple target fragments, wherein the processing results of different task types are repartitioned according to different algorithm strategies.

2. The method according to claim 1, wherein, Synchronizing the processing results to a second graph system that matches the processing results includes: The processing results are converted into streaming data; The streaming data is synchronized to the second graph system that matches the streaming data.

3. The method according to claim 2, wherein converting the processing result into streaming data comprises: Acquire synchronization strategy data of the processing result, wherein the synchronization strategy data is used to characterize the synchronization period when synchronizing the processing result and the amount of data of the processing result synchronized within the synchronization period; The streaming data is determined based on the synchronization strategy data.

4. The method according to claim 3, wherein obtaining the synchronization strategy data of the processing result includes: Determine the scene data from which the graph data originates; The synchronization strategy data is determined based on the scenario data.

5. The method according to claim 1, wherein the amount of graph data is greater than a first data amount threshold, wherein, Transmitting the graph data to a first graph system that matches the task type includes: The graph data is transmitted to a graph computing system that matches the task type, wherein the graph computing system is used to perform graph computing and / or graph retrieval on the graph data to obtain the processing result, and the first graph system includes the graph computing system.

6. The method according to claim 5, wherein synchronizing the processing result to a second graph system matching the processing result comprises: The processing result is synchronized to a graph database system that matches the processing result, wherein the graph database system is used to store the processing result on a disk, and the second graph system includes the graph database system.

7. The method according to claim 1, wherein the amount of graph data is less than a second data amount threshold, wherein, Transmitting the graph data to a first graph system that matches the task type includes: The graph data is transmitted to a graph database system that matches the task type, wherein the graph database system is used to retrieve and / or update the graph data to obtain the processing result, and the first graph system includes the graph database system.

8. The method according to claim 7, wherein synchronizing the processing result to a second graph system matching the processing result comprises: The processing result is synchronized to a graph computing system that matches the processing result, wherein the graph computing system is used to store the processing result in memory, and the second graph system includes the graph computing system.

9. The method according to claim 8, further comprising: The processing result from the memory is determined as sampled data; A graph learning system is trained based on the sampled data.

10. A graph data processing apparatus, comprising: The determination unit is used to determine the task type of the graph data to be processed. A transmission unit is used to transmit the graph data to a first graph system that matches the task type; The acquisition unit is used to acquire the processing result output by the first graph system, wherein the processing result is obtained by the first graph system performing a processing task corresponding to the task type on the graph data; A synchronization unit is used to synchronize the processing result to a second graph system that matches the processing result, wherein the synchronization between the first graph system and the second graph system is accomplished based on a streaming synchronization strategy, the streaming synchronization strategy including at least synchronization according to time and / or synchronization according to data change amount; The synchronization unit includes: a second synchronization module, used to synchronize the processing results to multiple target fragments in the second graph system based on the task type, wherein the multiple target fragments are independent of each other; and a processing module, used to repartition multiple original fragments of the second graph system to obtain the multiple target fragments, wherein the processing results of different task types are repartitioned according to different algorithm strategies.

11. The apparatus according to claim 10, wherein, The synchronization unit includes: A conversion module is used to convert the processing results into streaming data; The first synchronization module is used to synchronize the streaming data to the second graph system that matches the streaming data.

12. The apparatus according to claim 11, wherein, The conversion module includes: The acquisition submodule is used to acquire the synchronization strategy data of the processing result, wherein the synchronization strategy data is used to characterize the synchronization period when synchronizing the processing result and the amount of data of the processing result synchronized within the synchronization period; The determination submodule is used to determine the streaming data based on the synchronization strategy data.

13. The apparatus according to claim 12, wherein, The acquisition submodule obtains the synchronization strategy data of the processing result through the following steps: Determine the scene data from which the graph data originates; The synchronization strategy data is determined based on the scenario data.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-9.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-9.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-9.

Images