Multi-dimensional index calculation method based on graph database

Abstract

This invention discloses a multi-dimensional indicator calculation method based on a graph database, comprising: defining an indicator graph storage model based on the graph database; registering parameterized calculation tasks corresponding to the indicators in the built-in computing framework of the graph database; when a user query request arrives, submitting the query transaction through a long connection channel established by the graph database driver, and the computing framework directly accessing local storage data to complete real-time calculation and return the results; when the original data in the graph database changes, the database trigger captures the change event and launches the calculation task in the same process; the user query trigger and the data update trigger share the same set of calculation tasks and graph traversal logic, realizing real-time multi-dimensional indicator calculation and updating with integrated storage and computation. The multi-dimensional indicator calculation method based on a graph database provided in this application only relies on the graph database and its own computing framework, simplifying the data processing chain and improving the efficiency of indicator output.

Description

Technical Field

[0001] This invention specifically relates to a method for calculating multi-dimensional indicators based on graph databases. Background Technology

[0002] In the field of big data analytics, "metrics" refer to quantitative data or status information used to measure, evaluate, and analyze business status. Through a metric system, enterprises can monitor their business operations in real time, thereby optimizing business processes and providing data support for management decisions. Specifically, scenarios such as anti-fraud auditing, power grid equipment status prediction, and financial risk assessment all involve a large number of multi-dimensional metrics that require real-time calculation. For example, when an account manager conducts a risk assessment on a customer's submitted approval application, they need to immediately calculate multi-dimensional metrics related to that application, including but not limited to "historical default rate," "relevant party credit score," and "number of fingerprint applications for the same device." These metrics often require aggregation, statistical analysis, and inference through association with massive historical datasets, ultimately outputting quantitative results that can be directly judged and used for decision-making by business stakeholders.

[0003] To perform the aforementioned multi-dimensional metric calculations that combine real-time and offline processing, traditional big data analytics systems typically require a combination of various heterogeneous frameworks and technologies. Especially in scenarios involving a mix of offline and real-time processing, distributed Spark / Flink computing engines and distributed OLAP database storage are often necessary, requiring complex architectures. The high demands on manpower, technical requirements, hardware resources, and operational maintenance make big data analytics extremely complex and expensive. Furthermore, because data flow involves numerous nodes—receiving a message from the server, processing it through parameter parsing in the response layer, database queries, execution of computational tasks by the Spark / Flink computing engine, and finally transmission back to the response layer—this entire process involves many services. The information exchange between these services inevitably involves network transmission latency, preventing the achievement of optimal performance in real-time computing scenarios.

[0004] In summary, while existing technologies can achieve multi-dimensional indicator calculations in the field of big data analytics, they suffer from significant drawbacks such as complex architecture, high cost, large latency, and difficulty in expansion. There is an urgent need for a new technology solution that can simplify the architecture, reduce costs, and simultaneously meet real-time performance requirements. Summary of the Invention

[0005] This invention provides a multi-dimensional index calculation method based on graph databases to solve the aforementioned technical problems, specifically adopting the following technical solution:

[0006] A method for calculating multi-dimensional indicators based on graph databases includes:

[0007] Define a graph storage model for metrics based on graph databases;

[0008] Register the parameterized computation task corresponding to the indicator in the built-in computation framework of the graph database. The task is written using Cypher query language combined with the graph database's custom computation process.

[0009] When a user query request arrives, the query transaction is submitted through a long connection channel established by the graph database driver. The computing framework directly accesses the local storage data to complete real-time calculations and returns the results.

[0010] When the original data in the graph database changes, the database trigger captures the change event and launches the computing task in the same process to incrementally update the affected metrics.

[0011] In this system, user query triggers and data update triggers share the same set of computing tasks and graph traversal logic, enabling real-time multi-dimensional indicator calculation and updates that integrate storage and computation.

[0012] Furthermore, the definition mentioned above, based on a graph database-based indicator graph storage model, includes:

[0013] Extract a list of metrics to be calculated based on the business scenario, and determine the business meaning, calculation dimension and calculation logic for each metric;

[0014] The indicator list is mapped to nodes, edges, and attributes in a graph database to form an indicator graph storage model and complete persistence.

[0015] Furthermore, before persistence is completed, the resulting index graph storage model is optimized. During the model optimization phase, the shortest path algorithm of the graph database is used to evaluate the traversal hop count of different modeling schemes, and the shortest hop count is used as the criterion to decide whether to further optimize the model.

[0016] Furthermore, the steps for establishing the indicator graph storage model further include: distinguishing between basic indicators and derived indicators, recording basic indicators with first-type nodes and derived indicators with second-type nodes in the graph model, and explicitly connecting derived indicator nodes with their dependent basic indicator nodes through dependency edges to form a weighted directed acyclic graph.

[0017] Furthermore, the parameterized computation task is written using a combination of Cypher declarative query language and custom computation procedures provided by the graph database, and registered in the graph database's metadata for joint invocation by query triggers and update triggers.

[0018] Furthermore, the establishment of the long connection channel includes: pre-establishing a TCP long connection between the client and the graph database, and reusing the session on the connection to submit multiple Cypher queries, thereby reducing handshake and serialization overhead.

[0019] Furthermore, the trigger directly invokes the graph database's built-in computing framework after capturing a data update event, without needing to send task requests to an external computing engine over the network.

[0020] Furthermore, the incremental update further includes: the trigger only passes the subgraph identifier affected by the changed node to the computing framework, and the computing framework performs a local traversal within the subgraph to complete the update of the affected metrics.

[0021] Furthermore, the graph database employs distributed storage, and before execution, the scheduler routes the subtasks to the nodes holding the corresponding shards according to the data locality principle to achieve parallel computing.

[0022] Furthermore, the graph database triggers share the same memory heap with the computing framework, and the triggers directly pass change events to the computing tasks through a zero-copy circular buffer, avoiding serialization and network transmission.

[0023] The advantage of this invention is that the multi-dimensional indicator calculation method based on graph databases provided does not rely on other complex big data components, but only on graph databases and their own computing framework. Graph databases can be directly used to connect with business services, which facilitates operation and maintenance management, simplifies the data processing chain, and improves the efficiency of indicator output.

[0024] The advantage of this invention lies in the multi-dimensional indicator calculation method based on graph databases. When calculating indicators requiring real-time multi-dimensional correlation, the graph database is used for correlation queries, and the graph traversal method within the computational framework is used to complete the indicator calculation. This integrated storage and computation structure allows the graph database to simultaneously handle storage, computation, querying, and updating functions as a single instance. Data does not need to flow across services, end-to-end latency is reduced to milliseconds, and system complexity and external dependencies are reduced. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 This is a schematic diagram of the multi-dimensional index calculation method based on graph database of this application. Detailed Implementation

[0027] Embodiments of the present invention are described in detail below. Examples of these embodiments are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0028] like Figure 1 The figure shown is a method for calculating multi-dimensional indicators based on graph databases according to this application, including:

[0029] S1: Define a graph storage model for metrics based on a graph database.

[0030] In graph databases, to ensure a more accurate and unambiguous storage model, it's essential to clearly define the storage model first. A better model definition leads to improved performance during computational tasks in the subsequent computational framework development.

[0031] In the embodiments of this application, the definition of a graph database-based index graph storage model includes:

[0032] Extract a list of metrics to be calculated based on the business scenarios, and determine the business meaning, calculation dimensions, and calculation logic for each metric. Specifically, analyze existing business scenarios and extract a list of metrics that need to be calculated in the actual scenarios. It is necessary to clearly define the business meaning, calculation dimensions, and calculation logic of each metric to ensure that the metric definitions fully match the business requirements.

[0033] The indicator list is mapped to nodes, edges, and attributes in a graph database to form an indicator graph storage model and complete its persistence. By analyzing the indicators in the indicator list, the elements to be stored in the graph database are extracted. Entities are the basic elements actually stored in the graph database. Examples include application documents, applicants, and the relationship between application documents and applicants. After forming the indicator graph storage model, it is inserted into the graph database. Specifically, the mapping process is not a simple one-to-one conversion from "table to node," but follows the "indicator-first" principle. All indicators to be calculated are listed first, and then the minimum set of entities required to support these indicators is derived. For frequently accessed composite indicators, their intermediate calculation results are materialized as nodes to reduce real-time computation.

[0034] In the embodiments of this application, before persistence is completed, the resulting index graph storage model is optimized. During the model optimization stage, the shortest path algorithm of the graph database is used to evaluate the traversal hop count of different modeling schemes, and the shortest hop count is used as the criterion to decide whether to further optimize the model.

[0035] Specifically, after modeling is completed, the shortest path algorithm is run for each metric to calculate the average number of hops. If the number exceeds a threshold, a prompt is made to adjust nodes or relationships. The system provides an automatic diagnostic script that outputs a "metric-hop count" lookup table, which modelers use to delete redundant relationships or add intermediate nodes, ensuring the graph model remains optimal and preventing performance degradation later on.

[0036] In the embodiments of this application, the step of establishing the indicator graph storage model further includes: distinguishing between basic indicators and derived indicators, recording basic indicators with first type nodes and derived indicators with second type nodes in the graph model, and explicitly connecting the derived indicator nodes with the basic indicator nodes they depend on through dependency edges to form a weighted directed acyclic graph.

[0037] Understandably, when establishing a metric graph storage model, it's necessary to distinguish between basic and derived metrics and clearly define the dependencies between them. Based on these requirements, this is solidified into a graph structure: basic metric nodes store the original metrics, derived metric nodes store aggregated or weighted results, and dependency edges carry weights or version numbers. At runtime, the system can trace back through dependency edges, recalculating only affected derived metrics, reducing CPU consumption. Adding new metrics doesn't require restarting tasks; simply adding nodes and edges is sufficient, significantly improving iteration efficiency. The directed acyclic graph avoids the risk of infinite loops caused by circular dependencies. Edge weights can be set to "computation latency" or "confidence level." When upstream data is delayed or confidence levels decrease, the system automatically reduces the weight of downstream derived metrics and displays a red exclamation mark on the UI to alert business users. This mechanism enables the metric chain to have self-explanatory and self-protective capabilities.

[0038] In summary, a "precise and unambiguous" model is not only reflected in the naming conventions of nodes, edges, and attributes, but more importantly, in the explicit expression of "metric semantics." Traditional relational models often hide metric semantics within SQL statements for multi-table JOINs. Once business rules change, the table structure, ETL scripts, and report SQL must be modified simultaneously, resulting in extremely high maintenance costs. This application uses a graph model to explicitly express "metric semantics" as a topological structure of nodes and edges: each business metric corresponds to a node, and the node only stores attributes directly related to its semantics; the dependencies between metrics are explicitly expressed as "dependency edges," which can carry weights, calculation formulas, or version numbers; when business rules change, only nodes need to be added or deleted, or dependency edges need to be adjusted, without modifying the underlying storage table structure. An additional benefit of this explicitness is "traceability": any anomaly in a metric result can be quickly located to the upstream basic data node through dependency edges, achieving root cause localization within minutes.

[0039] S2: Register parameterized computation tasks corresponding to the indicators in the built-in computation framework of the graph database. The tasks are written using Cypher query language combined with the graph database's custom computation process.

[0040] In the embodiments of this application, the parameterized computation task is written using a hybrid approach of Cypher declarative query language and custom computation procedures provided by the graph database, and registered in the graph database's metadata for joint invocation by query and update triggers. Specifically, the task script is written entirely using Cypher + custom procedures (such as Galaxybase PAR, Neo4j APOC) and stored in the system graph space. Both query and update triggers invoke the same procedure name, avoiding script duplication and version drift, and reducing maintenance costs.

[0041] Graph databases utilize Cypher for interaction. Compared to pre-defined HTTP interfaces, the diverse variations of Cypher statements allow for easier debugging and modification. Cypher statements are constructed based on business scenarios to execute specified tasks. Thanks to the dynamic nature of Cypher statements, multiple tasks can be concatenated. Due to the specific nature of business scenarios, functionalities frequently change during development; by adjusting the combination of Cypher statements, modified metric calculations can be performed, facilitating debugging and modification. A persistent connection driver is used to send statements to the graph database server, where the server processes and executes the Cypher statements.

[0042] S3: When a user query request arrives, the query transaction is submitted through a long connection channel established by the graph database driver. The computing framework directly accesses the local storage data to complete real-time calculations and returns the results.

[0043] When users query real-time metrics, traditional data computation requires pre-writing Spark / Flink computation tasks, which then submit the data to Spark / Flink for calculation. However, when using a graph database, parameterized computation tasks corresponding to the metrics are registered. Upon receiving the request at the response layer, the computation framework in the graph database can directly call the computation task and retrieve the corresponding stored data for direct computation, reducing communication and network transmission between services.

[0044] Traditional big data analytics frameworks often employ RESTful architectures, while the graph database in this application utilizes a Driver-based long-lived connection approach. In the implementation of this application, establishing the long-lived connection involves: pre-establishing a TCP long-lived connection between the client and the graph database, and reusing the session on this connection to submit multiple Cypher queries, thereby reducing handshake and serialization overhead. The Driver-based long-lived connection avoids TCP reuse issues. In practice, the client maintains the long-lived connection after initializing the Driver, and all queries reuse the same physical connection, reducing the number of connections and the number of server-side thread switches.

[0045] Traditional big data analytics frameworks, upon receiving a request at the response layer, package it into a new request and send it over the network to Spark / Flink for task execution. When Spark / Flink needs to access the database during computation, it retrieves the computational data from the database via JDBC over the network. In contrast, the graph database in this application, upon receiving a task, directly invokes the internal computation framework to trigger task execution. The necessary data can be directly retrieved from the database, eliminating the need for cross-network service calls and reducing inter-service communication and transmission. This results in superior performance for real-time graph metric computation scenarios.

[0046] S4: When the original data in the graph database changes, the database trigger captures the change event and starts a computing task in the same process to incrementally update the affected metrics.

[0047] Understandably, in traditional databases, when updating stored data, triggers can be set to initiate subsequent tasks. Because relational databases themselves do not participate in computation, each data update requires sending a request to a computation service, which then calculates the result and reconnects to the database or other message queue middleware to complete the data update. However, in graph databases, triggers can be used directly to initiate computation tasks within the computation framework, performing the computation and updating the data directly, without requiring information transfer between multiple services.

[0048] In the embodiments of this application, the trigger directly invokes the built-in computing framework of the graph database after capturing a data update event, without needing to send task requests to an external computing engine over the network. In practice, the trigger is registered on the AFTER UPDATE event of the node / edge. When an attribute changes, the trigger starts a computing task within the same process, eliminating network round trips, reducing latency from seconds to milliseconds, and eliminating the need for middleware such as Kafka and RPC, significantly reducing operational costs.

[0049] In the embodiments of this application, the incremental update further includes: the trigger only passes the subgraph identifiers affected by the changed nodes to the computing framework, and the computing framework performs a local traversal within the subgraph to complete the update of the affected metrics. In implementation, the trigger encapsulates the list of changed node IDs as event parameters. The computing task locates the subgraph using the ID index and performs BFS or shortest path computation only on that subgraph, avoiding full graph scanning.

[0050] In the embodiments of this application, the graph database triggers and the computing framework share the same memory heap. The triggers directly pass change events to the computing tasks through a zero-copy circular buffer, avoiding serialization and network transmission. The triggers directly call the internal computing framework, thus eliminating the need for cross-process communication. Preferably, through the shared memory circular buffer, after the trigger writes an event object, the computing task reads it directly using a pointer, without any copying.

[0051] In this application, user query triggers and data update triggers share the same set of computing tasks and graph traversal logic, realizing real-time multi-dimensional indicator calculation and update that integrates storage and computation.

[0052] In the computational framework of a graph database, each metric is registered with only one computational procedure. Whether triggered by a user query or a data update, this single procedure is ultimately invoked, rather than maintaining two sets of code or two scripts.

[0053] User queries trigger responses by taking "query parameters" (node ​​ID, time window, dimension, etc.) as input; data updates trigger responses by taking "change events" as input. The only difference between the two triggering methods is the parameters passed in; the underlying graph traversal, aggregation, and update logic is completely identical.

[0054] User query triggers and data update triggers share the same set of computation tasks and graph traversal logic. This allows for simultaneous support of both trigger scenarios through a single codebase and process, eliminating the cross-service and cross-network calls caused by the separation of "query tasks" and "incremental tasks" in traditional architectures. Due to this sharing mechanism, the storage and computation layers are compressed into the same graph database instance. Data can be computed without leaving the graph database; queries and updates are performed directly on local memory / disk, eliminating the need for external components such as Spark / Flink, message queues, and REST gateways. By sharing the same set of computation tasks and graph traversal logic, real-time multi-dimensional metric calculation and updates are achieved through "integrated storage and computation." During implementation, the graph database partitions nodes and edges into shards using hashing. The scheduler scans the execution plan before task startup and sends subtasks directly to the node holding that shard, avoiding data fetching across nodes.

[0055] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the above embodiments do not limit the present invention in any way, and all technical solutions obtained by equivalent substitution or equivalent transformation fall within the protection scope of the present invention.

Claims

1. A method for calculating multi-dimensional indicators based on graph databases, characterized in that, include: Define a graph storage model for metrics based on graph databases; Register the parameterized computation task corresponding to the indicator in the built-in computation framework of the graph database. The task is written using Cypher query language combined with the graph database's custom computation process. When a user query request arrives, the query transaction is submitted through a long connection channel established by the graph database driver. The computing framework directly accesses the local storage data to complete real-time calculations and returns the results. When the original data in the graph database changes, the database trigger captures the change event and launches the computing task in the same process to incrementally update the affected metrics. Among them, user query triggers and data update triggers share the same set of computing tasks and graph traversal logic, realizing real-time multi-dimensional indicator calculation and update in storage and computing integration; The definition of the indicator graph storage model based on graph database includes: Extract a list of metrics to be calculated based on the business scenario, and determine the business meaning, calculation dimension and calculation logic for each metric; The indicator list is mapped to nodes, edges, and attributes in a graph database to form an indicator graph storage model and complete persistence. Before persistence is completed, the resulting index graph storage model is optimized. During the model optimization phase, the shortest path algorithm is run for each index to calculate the average number of hops. If the number of hops exceeds the threshold, the system prompts for adjustment of nodes or relationships. The system outputs an index-hop count comparison table, which the modelers use to delete redundant relationships or add intermediate nodes. The shortest number of hops is used as the criterion to decide whether to further optimize the model.

2. The multi-dimensional index calculation method based on graph database according to claim 1, characterized in that, The steps for establishing the indicator graph storage model further include: distinguishing between basic indicators and derived indicators, recording basic indicators with first-type nodes and derived indicators with second-type nodes in the graph model, and explicitly connecting derived indicator nodes with their dependent basic indicator nodes through dependency edges, wherein the dependency edges carry weights, calculation formulas or version numbers, forming a unidirectional, acyclic weighted directed acyclic graph.

3. The multi-dimensional index calculation method based on graph database according to claim 1, characterized in that, The parameterized computation task is written using a combination of Cypher declarative query language and custom computation procedures provided by the graph database, and registered in the graph database's metadata for joint invocation by query triggers and update triggers.

4. The multi-dimensional index calculation method based on graph database according to claim 1, characterized in that, The establishment of the long connection channel includes: pre-establishing a TCP long connection between the client and the graph database, and reusing the session on the connection to submit multiple Cypher queries, thereby reducing handshake and serialization overhead.

5. The method for calculating multi-dimensional indicators based on graph databases according to claim 1, characterized in that, After capturing a data update event, the trigger directly calls the built-in computing framework of the graph database, without having to send task requests to an external computing engine over the network.

6. The method for calculating multi-dimensional indicators based on graph databases according to claim 1, characterized in that, The incremental update further includes: the trigger only passes the subgraph identifier affected by the changed node to the computing framework, and the computing framework performs a local traversal within the subgraph to complete the update of the affected metrics.

7. The method for calculating multi-dimensional indicators based on graph databases according to claim 1, characterized in that, The graph database uses distributed storage. Before execution, the scheduler routes the subtasks to the nodes holding the corresponding shards according to the data locality principle to achieve parallel computing.

8. The method for calculating multi-dimensional indicators based on graph databases according to claim 1, characterized in that, Graph database triggers share the same memory heap with the computing framework. Triggers directly pass change events to computing tasks through a zero-copy circular buffer, avoiding serialization and network transmission. During the index calculation process, the graph database's computing framework utilizes node attribute compression encoding and edge type bitmap indexing to skip nodes and edges that do not meet the filtering conditions when performing graph traversal.

Images