Graph analysis method and system for transactional data

Abstract

The application provides a transaction data graph analysis method and system, constructs a transaction node and a graph analysis node, defines a data model mapping, initializes graph data of the graph analysis node, sends transaction logs generated by the transaction node to the corresponding graph analysis node when the transaction node performs transaction submission, converts the transaction logs into a unified format by a log cleaner on the graph analysis node, restores the logs into updates on the graph data in sequence on the graph analysis node, and updates the graph data by using a concurrent control protocol, sends a graph analysis request to a server by a client, generates an execution plan by the server, and continues to forward the request to the involved graph analysis nodes, reads and calculates data from the graph data according to the execution plan by the graph analysis nodes to obtain each sub result of the analysis request, merges the sub results on each analysis node to generate a result set, and returns the result set to the client. The application can guarantee the consistency and timeliness of data, and does not interfere with original transaction execution.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a transactional data graph analysis method and system. BACKGROUND

[0002] Graph-structure data divides data into two categories of entities and relationships, stores entities as vertices in a graph, and stores relationships between entities as edges in the graph. Graph analytical processing refers to a type of data processing task that performs operations such as traversal, analysis, and query on graph-structure data. Some classic graph algorithms (such as the PageRank algorithm for web page ranking) can be efficiently processed on graph data for data analysis, matching of related data, and data mining in application scenarios.

[0003] However, the sources of data are often transactional data (referring to data generated by online transaction processing, such as e-commerce orders and stock transactions) on transaction nodes, and the data is mainly stored in a relational model. Directly performing graph analysis processing on transactional data is not only difficult to express using a relational model, but also inefficient. If the existing transactional data is directly switched to another data storage method, there will be compatibility problems and a large amount of manpower and material resources. Therefore, supporting graph processing and analysis on transactional data is not only a real demand but also a great challenge.

[0004] Existing graph analysis processing methods for transactional data include direct processing on transactional databases, using graph databases, and converting data models through offline extract-transform-load (ETL) technology. However, these technologies often lack performance, consistency, and data timeliness. For example, for direct analysis of transactional data using a new graph database, the transaction needs to be submitted to both the transactional database and the graph database, making it difficult to maintain consistency for two submissions. In the ETL method, data needs to be converted in batches and periodically, and due to the large amount of batch data, the conversion period is very long (such as hours to days).

[0005] Therefore, how to design a new graph analysis processing method for transactional data to improve data expression ability and data processing performance has become an important problem in the use and analysis of transactional data.

[0006] One opportunity for the graph analysis processing mode of new transactional data is that existing transactional databases often have logs and data backups at transaction submission for fault tolerance, the logs record the data involved in the transaction and its changes, and the transaction logs can obtain data change information and perform real-time synchronization of data, therefore, transforming the data backup into a graph structure, using the logs as the source of data changes, a new efficient transactional data graph analysis method and system can be established. SUMMARY

[0007] In view of the defects in the prior art, the application provides a transactional data graph analysis method and system.

[0008] According to the transactional data graph analysis method and system provided by the application, the scheme is as follows:

[0009] In a first aspect, a transactional data graph analysis method is provided, and the method comprises:

[0010] Step S1: constructing transaction nodes and graph analysis nodes according to user instructions;

[0011] Step S2: defining a data model mapping and initializing graph data of the graph analysis nodes;

[0012] Step S3: when the transaction nodes perform transaction submission, sending transaction logs generated by the transaction nodes to corresponding graph analysis nodes;

[0013] Step S4: the log cleaner on the graph analysis node converts the transaction logs into a unified format;

[0014] Step S5: on the graph analysis node, restoring the transaction logs into updates to graph data in order according to the data model mapping, and updating the graph data using a concurrency control protocol;

[0015] Step S6: sending a graph analysis request from a client to a server, the server generating an execution plan and continuing to forward the request to the involved graph analysis nodes;

[0016] Step S7: the graph analysis nodes reading and calculating data from the graph data according to the execution plan to obtain each sub-result of the analysis request;

[0017] Step S8: merging the sub-results on each analysis node to generate a result set, and returning the result set to the client.

[0018] Preferably, the graph analysis nodes and transaction nodes in step S1 are deployed in the same physical machine, or are deployed in different physical machines capable of communicating through a network to form a distributed system.

[0019] Preferably, the data model mapping in the step S2 comprises: mapping of database table name to a certain type of point or edge, and mapping of attribute in the database table to the attribute of the graph structure data.

[0020] Preferably, the transaction log in the step S3 comprises: database table name involved in the transaction, primary key and value of the modified data, modification type, and sequence information at the time of transaction submission, and the transaction log is transmitted from the transaction node to the graph analysis node through the network or the related mode in the shared memory.

[0021] Preferably, the log cleaner in the step S4 analyzes the structure of the transaction log, and unifies the transaction log into a unified log format containing the related necessary information including the database table name, the primary key and value, the update type, and the sequence information.

[0022] Preferably, the update of the graph data in the step S5 needs to clean the logs in sequence according to the sequence information in the logs, and update the graph data to ensure data consistency.

[0023] Preferably, the server in the step S6 is responsible for receiving the graph analysis request, and distributing the request to the related node according to the data involved in the request.

[0024] Preferably, the reading of the data in the step S7 needs to read a consistent data according to the concurrency control protocol.

[0025] In a second aspect, a graph analysis system of transactional data is provided, and the system comprises: a server and a client.

[0026] The server comprises a request scheduler and a plurality of data nodes.

[0027] The request scheduler is responsible for sending the request to the appropriate data node, and the data node is responsible for processing the transaction or graph analysis request, reading and writing data, and calculation.

[0028] The request scheduler comprises: a transaction request scheduler and a graph analysis request scheduler.

[0029] The data node comprises: a transaction node and a graph analysis node.

[0030] Preferably, the client and the server adopt a client / server (C / S) mode to form a local area network by using computers.

[0031] Compared with the prior art, the application has the following beneficial effects:

[0032] 1. Compared with the method of directly performing graph analysis on transactional data or deploying a new graph database and ETL, the application can better ensure data consistency, timeliness, and analysis processing performance.

[0033] 2、The application can simultaneously perform online transaction processing and graph analysis in an integrated system, reducing operation and maintenance costs.

[0034] 3、The application can reuse existing log and backup mechanisms to some extent, simplifying system design and development, and being compatible with different sources of transactional data without interfering with transaction processing.

[0035] Other benefits of the application will be described in the specific embodiments by introducing specific technical features and technical solutions, and those skilled in the art should be able to understand the benefits brought by the technical features and technical solutions. BRIEF DESCRIPTION OF DRAWINGS

[0036] Other features, objects and advantages of the application will become more apparent through reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0037] Figure 1 is a flowchart of the application;

[0038] Figure 2 is a system structure example diagram of the application. DETAILED DESCRIPTION

[0039] The application will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the application, but do not limit the application in any form. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the application. These are within the scope of protection of the application.

[0040] The application embodiment provides a graph analysis method for transactional data, as shown in Figure 1 The method uses transaction database logs and backups to establish a new type of graph structure copy for graph analysis, i.e., transactional data is stored as a graph structure, and the graph data is updated in real time using logs, and the graph analysis of transactional data is performed on the graph structure copy. The graph analysis method for transactional data includes the following steps:

[0041] Step S1: According to user-defined construction, the node (referred to as "transaction node") for transaction is established to store data in a heterogeneous attribute graph structure model, and the node (referred to as "graph analysis node") for graph analysis.

[0042] The graph analysis node in this step can be deployed in the same physical machine as the transaction node, or in different physical machines that can communicate through a network to form a distributed system. For the transaction node, it needs to be able to generate transaction logs (such as binlog). The heterogeneous attribute graph data structure refers to the fact that the points and edges in the graph structure have different types and attributes.

[0043] Step S2: The user defines the model mapping from the relational model of transactional data to the attribute graph model of graph data, i.e., data model mapping, and initializes the graph data of the graph analysis node.

[0044] The model mapping of transactional data to graph data in this step includes the mapping of database table names to points or edges of a certain type, and the mapping of attributes in the database table to attributes of the graph structure data.

[0045] Step S3: When the transaction node performs transaction commit, the transaction log generated by the transaction node is sent to the corresponding graph analysis node.

[0046] The transaction log in this step needs to include the database table name involved in the transaction, the primary key and value of the modified data, the modification type (such as insertion, deletion, update), and the sequence information (such as timestamp, data version number, etc.) at the time of transaction commit. The log is sent from the transaction node to the graph analysis node through a network or shared memory, etc.

[0047] Step S4: The log cleaner on the graph analysis node analyzes the log structure and converts it into a unified log format.

[0048] The log cleaner in this step unifies the transaction log into a unified log format containing necessary information, such as database table name, primary key and value, update type, and sequence information (such as timestamp, version number), etc.

[0049] Step S5: On the graph analysis node, the transaction log is recovered into updates to the graph data in sequence according to the data model mapping, and the graph data is updated using a concurrency control protocol.

[0050] In this step, the updates to the graph data need to be cleaned in sequence according to the sequence information in the log, and the graph data needs to be updated to ensure data consistency. The sequence granularity can be adjusted here (such as cleaning one by one or cleaning a fixed number of logs in parallel). According to the user-defined mapping of relational data to graph data, the data changes in the log are converted to changes to the points or edges and their attributes in the graph. At the same time, a concurrency control protocol (such as two-phase locking, optimistic concurrency control, multi-version concurrency control, etc.) needs to be used for management to solve the competition between updates to the graph data and user read operations.

[0051] Step S6: The client sends a graph analysis request to the server, and the server generates an execution plan and forwards the request to the involved graph analysis nodes. The server is responsible for receiving the graph analysis request, and will distribute the request to the relevant nodes according to the data involved in the request.

[0052] Step S7: The graph analysis nodes read data from the graph data and calculate according to the execution plan, and obtain each sub-result of the analysis request. In this step, the data reading needs to read a consistent data according to the concurrency control protocol.

[0053] Step S8: Merge the sub-results on each analysis node to generate a result set, and return the result set to the client. In this step, if the graph analysis task involves multiple graph analysis nodes, the local result sets on each node need to be merged before the result set is returned.

[0054] The transactional data in the application includes data stored in a relational model using a transactional database system or a stream system, etc.

[0055] The application also provides a graph analysis system for transactional data, which can be realized by executing the process steps of the graph analysis method for transactional data, i.e., the graph analysis method for transactional data can be understood by those skilled in the art as a preferred embodiment of the graph analysis system for transactional data. Specifically, referring to FIG. 1, the system specifically includes: Figure 2

[0056] The server and the client, the server includes a request scheduler (transaction request scheduler, graph analysis request scheduler) and a plurality of data nodes (transaction nodes and graph analysis nodes). The request scheduler is responsible for sending requests to appropriate data nodes, and the data nodes are responsible for processing transaction or graph analysis requests, reading and writing data, and calculating, etc. The client and the server adopt a client / server (C / S) mode to form a local area network by computer.

[0057] Implementation principle:

[0058] ​In the system implementation process, the user first specifies the system to establish a graph analysis node and define the data model mapping relationship. In the use process, the user sends a transaction request to the transaction scheduler in the system through the client, the transaction scheduler sends the transaction request to the corresponding transaction node (such as transaction node 0) according to the data division, the transaction node completes the transaction request and sends the log to the corresponding graph analysis node (such as graph analysis node 0) according to the data division, the log cleaner on the graph analysis node 0 converts the log into a unified format, and uses the information in the data model conversion mapping to convert the transaction log into data updates on the graph data and apply to the graph analysis node. At the same time, the graph analysis scheduler receives the graph analysis request sent by the user through the client, analyzes the request and generates a query plan. Send the request to the related graph analysis nodes (such as graph analysis nodes 0 and 1), and after the graph analysis nodes read the data and calculate the sub-results, return the sub-results to the scheduler, and the graph analysis scheduler returns the result set to the client after merging the sub-results.

[0059] Next, the application will be described in more detail.

[0060] As shown in Figure 1 , it is a flow chart of the transaction type data graph analysis method and system of the application. As shown in Figure 2 , it is a topological diagram of the deployment of the client and the server node. The following will be described in detail in combination with Figure 1 the relational data connection step:

[0061] In step S1, the user will specify how to establish a graph analysis node, including specifying which transaction nodes to establish corresponding graph analysis nodes, for example, to establish corresponding graph analysis nodes i (i = 0, 1, 2) for transaction nodes i. This means that the corresponding graph analysis node will be subjected to the data transmitted by the transaction node, and the stored data is also the same data stored in different data models.

[0062] In step S2, the user needs to specify how the transaction type data is mapped from the relational model to the property graph model. Including the need to specify the mapping of the database table to the point or edge type, the mapping of the column of the database table to the attribute type. For example, the table PERSON is mapped to the Person type point, the table ORDERS is mapped to the Orders type edge, and the NAME column in the table PERSON is mapped to the Name attribute of the Person point.

[0063] In step S3, the graph analysis node obtains a part of the initialization data from the corresponding transaction node according to the mapping relationship and converts it into graph structure data, and performs data initialization of the graph analysis node. For example, the graph analysis node i needs to obtain data from the transaction node i (i = 0, 1, 2) and convert it.

[0064] In step S4, when a transaction is committed on the transaction node, the transaction data generated (including how the data changes and the log order information) needs to be sent to the corresponding graph analysis node.

[0065] In step S5, the log cleaner on the graph analysis node analyzes the log received from the transaction node and converts it into a unified log format, including table name, primary key, value, update type and log order information.

[0066] In step S6, the graph data is updated according to the log order. The data in the log needs to be mapped according to the data model defined by the user in step S2 to convert the data from the relational model to the graph structure data, and the graph data in the graph analysis node is updated according to the concurrency control protocol (such as multi-version concurrency control).

[0067] In step S7, the graph analysis scheduler receives the user's graph analysis request (such as in the form of Gremlin graph query language, etc.), generates a query plan, and distributes it to the corresponding graph analysis nodes (such as to graph analysis nodes 1 and 2) according to the data division.

[0068] In step S8, on the graph analysis node, the data is read using the concurrency control protocol (such as the multi-version concurrency control protocol, and after obtaining a stable version number, the version number is used for data reading), and the calculation is performed according to the query plan.

[0069] In step S9, if the analysis task involves multiple graph analysis nodes, the partial result set needs to be merged on a certain graph analysis node (such as graph analysis node 1), and the final result set is returned to the client.

[0070] The embodiment of the present application provides a graph analysis method and system for transactional data, which uses the logs and backups of the transaction database to establish a new graph structure copy for graph analysis, i.e. stores the transactional data as a graph structure, uses the logs to update the graph data in real time, and performs graph analysis on the graph structure copy. Compared with the prior art, the present application can improve the performance and expressiveness of graph analysis on transactional data, while ensuring the consistency and timeliness of the data, and does not interfere with the original transaction execution.

[0071] Those skilled in the art know that, in addition to implementing the system provided by the present application and each device, module and unit thereof in the form of pure computer readable program code, the system provided by the present application and each device, module and unit thereof can also be implemented in the form of logic gates, switches, application specific integrated circuits, programmable logic controllers and embedded microcontrollers, etc. by logically programming the method steps to achieve the same functions. Therefore, the system provided by the present application and each device, module and unit thereof can be considered as a hardware component, and the devices, modules and units included therein for achieving various functions can also be considered as structures within the hardware component; the devices, modules and units for achieving various functions can also be considered as both software modules implementing methods and structures within hardware components.

[0072] The specific embodiments of the present application are described above. It needs to be understood that the present application is not limited to the specific embodiments described above, and various changes or modifications can be made by those skilled in the art within the scope of the claims, which does not affect the essential content of the present application. The embodiments of the present application and the features in the embodiments can be combined with each other in any manner without conflict.

Claims

1. A graph analysis method for transactional data, characterized in that, include: Step S1: Construct transaction nodes and graph analysis nodes according to user instructions; Step S2: Define the data model mapping and initialize the graph data of the graph analysis nodes; Step S3: When the transaction node commits a transaction, it sends the transaction log generated by the transaction node to the corresponding graph analysis node; Step S4: The log cleaner on the graph analysis node converts the transaction log into a unified format; Step S5: On the graph analysis node, the transaction log is restored sequentially to update the graph data according to the data model mapping, and the graph data is updated using the concurrency control protocol; Step S6: The client sends a graph analysis request to the server. The server generates an execution plan and forwards the request to the relevant graph analysis nodes. Step S7: The graph analysis node reads and calculates data from the graph data according to the execution plan to obtain the sub-results of the analysis request; Step S8: Merge the sub-results from each analysis node, generate a result set, and return the result set to the client; The data model mapping in step S2 includes: mapping database table names to points or edges of a certain type, and mapping attributes in the database table to graph structure data attributes. The transaction log in step S3 includes the database table names involved in the transaction, the primary keys and values ​​of the modified data, the modification type, and the order information when the transaction is committed. The transaction log will be sent from the transaction node to the graph analysis node via network or shared memory.

2. The graph analysis method for transactional data according to claim 1, characterized in that, In step S1, the graph analysis nodes and transaction nodes are deployed on the same physical machine or on different physical machines that can communicate via a network, forming a distributed system.

3. The graph analysis method for transactional data according to claim 1, characterized in that, In step S4, the log cleaner analyzes the structure of the transaction log and unifies it into a unified log format that includes database table names, primary keys and values, update types, and sequence information.

4. The graph analysis method for transactional data according to claim 1, characterized in that, In step S5, updating the graph data requires cleaning up the logs and updating the graph data in sequence according to the order information in the logs to ensure data consistency.

5. The graph analysis method for transactional data according to claim 1, characterized in that, In step S6, the server is responsible for receiving graph analysis requests and distributing the requests to relevant nodes based on the data involved in the requests.

6. The graph analysis method for transactional data according to claim 1, characterized in that, In step S7, the data reading needs to be consistent with the concurrency control protocol.

7. A graph analysis system for transactional data, used to execute the graph analysis method for transactional data according to any one of claims 1-6, characterized in that, include: Server and client; The server includes a request scheduler and several data nodes; The request scheduler is responsible for sending requests to appropriate data nodes, and the data nodes are responsible for processing transaction or graph analysis requests, reading and writing data, and performing calculations. The request scheduler includes: a transaction request scheduler and a graph analysis request scheduler; The data nodes include: transaction nodes and graph analysis nodes.

8. The graph analysis system for transactional data according to claim 7, characterized in that, The client and server adopt a client / server (C / S) model, and the computers are grouped into a local area network.

Images