Storage method and query system for machine account resources and incidence relation data based on graph database

By using a graph database-based storage method and query system, the problems of data explosion and low performance in traditional resource allocation systems are solved, and efficient storage and querying of ledger resources and relational data are achieved.

CN118981467APending Publication Date: 2024-11-19NARI INFORMATION & COMM TECH
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202411047234.1
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-19

AI Technical Summary

Technical Problem

In traditional resource allocation systems, the storage of ledger resources and related data based on relational databases or document databases leads to an explosive growth in data volume, resulting in poor performance of cross-table or cross-collection joins and queries, which cannot meet the response requirements of real-world business.

Method used

By adopting a graph database-based storage method, and through graph storage methods, model design, data synchronization and query capabilities, efficient storage and querying of ledger resources and related relationship data can be achieved.

Benefits of technology

It enables efficient access to ledger resources and related data, solves the performance problems of traditional storage methods, and provides a convenient and fast data query system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118981467A_ABST
    Figure CN118981467A_ABST
Patent Text Reader

Abstract

The invention discloses a machine account resource and incidence relation data storage method based on a graph database and a query system thereof. The method comprises a graph storage method based on a general graph database; necessary and feasible model design conditions; the current storage structure design is adapted; shuffling a data structure, and finally landing in a graph database; landing in a graph database, and covering and correcting existing data; based on the resource configuration data basic graph query capability under the graph database graph storage scheme; the system comprises an application layer, a service layer and a data layer. According to the method, the characteristics of the graph database are utilized, a corresponding graph query language is used and packaged, and a convenient and quick-response data query system is constructed; the problems that historical data in a traditional resource configuration system is numerous, jumbled and disordered, not beneficial to long-time storage, low in query efficiency and the like are solved, and efficient access to the machine account resources and the association relationship historical data is achieved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of information platforms, and in particular to a method for storing ledger resources and associated relationship data based on a graph database and a query system thereof. Background Art

[0002] In the data processing process of traditional resource configuration systems, the storage of its ledger resources and associated relationship data is often based on relational databases or document databases. This leads to an explosive growth in the overall data volume and corresponding data operations when processing the corresponding historical data and attempting to persist it. In addition, when facing pattern queries on more complex associated links, because the storage mode of the relational database or document library itself is based on the resource table, it will cause a large number of cross-table or cross-collection joint queries, and the performance is extremely low, which cannot meet the response requirements of real business. Summary of the invention

[0003] Purpose of the invention: The purpose of the present invention is to provide a method for storing ledger resources and associated relationship data based on a graph database and a query system thereof.

[0004] Technical solution: The method for storing ledger resources and associated relationship data based on a graph database of the present invention comprises the following steps:

[0005] (1) Graph storage methods based on general graph databases, i.e., database systems with entities and their relationships as the main storage objects and corresponding storage methods;

[0006] (2) The object model's pre-inheritance characteristics, modeling rules, association characteristics, and corresponding necessary and feasible model design conditions based on the characteristics of the operation and maintenance object and the operation and maintenance requirements;

[0007] (3) Adapt the current storage structure design based on the object model's pre-inheritance characteristics, modeling rules, association characteristics, and corresponding necessary and feasible model design conditions;

[0008] (4) Incremental synchronization of ledger resources and associated relationship data of the resource configuration system, obtaining the changed content of ledger resources and associated relationship data from the upstream resource configuration system in real time or quasi-real time, and reorganizing the data structure through graph storage and corresponding persistence requirements, and finally implementing it in the graph database;

[0009] (5) The ledger resources and associated relationship data of the resource configuration system are fully synchronized. The real-time data slices of the ledger resources and associated relationship data are obtained from the upstream resource configuration system in a timed or quantitative detection manner. The data structure is reorganized through graph storage and corresponding persistence requirements, and finally implemented in the graph database, and the existing data is overwritten and corrected.

[0010] (6) The basic graph query capability of resource configuration data based on the graph database graph storage solution includes the following seven basic query capabilities and their availability requirements: the ability to add, delete, modify and query basic elements, the ability to add, delete, modify and query sparse associations, the ability to query and insert a large number of edges, the ability to query strict matches of indefinite length paths, the ability to query fuzzy matches of indefinite length paths, the ability to query detailed data of models and resources, and the ability to query statistical data of models and resources;

[0011] (7) The query function of existing historical data determines the query version based on the timestamp version number, performs related queries according to the actual time slice it faces, and thereby brings out the result set consisting of the ledger resources and related relationship data at the corresponding time point.

[0012] Furthermore, the graph storage method based on the general graph database in step (1) includes:

[0013] (1.1) The basic modeling method of general graph storage is based on entity and relationship modeling in the real world, including: graph, i.e., relationship graph, which is the logical separation unit of graph storage data; vertex, which generally refers to entity, i.e., the corresponding ledger resource in the resource allocation system; edge, which generally refers to the relationship between vertices; attribute, which is the business attribute contained in the vertex or edge;

[0014] (1.2) A universal graph storage query language that defines query conditions in a natural language-friendly way to express business query requirements;

[0015] (1.3) Complex relationship query and related algorithm capabilities. The library uses entities and relationships as basic units and provides various integrated algorithms for query and analysis, such as the shortest path algorithm and the minimum connected component algorithm.

[0016] (1.4) Graph process, i.e. graph traversal capability support based on graph storage, including: traversal generators for specific graphs, domain languages, and execution engines; functional data flow processes that transform type objects into other types of objects; traversal structured definition language for primitive graph semantics; parallel graph process processing system on distributed clusters; vertex code logic execution that communicates through message passing; parallel analysis MapReduce process for graph vertex computation process;

[0017] Furthermore, the single-step graph process classification of the graph process in the step (1.4) includes filtering class, mapping class, flat mapping class, side effect class and branch class.

[0018] Furthermore, in step (2), based on the characteristics of operation and maintenance objects and operation and maintenance requirements, traditional operation and maintenance objects and new operation and maintenance objects such as cloud environments coexist; object attributes meet the dual requirements of operation and maintenance management and asset maintenance; there is multiple inheritance between objects; there are customizable associations between objects; the runtime status of resources in the system is quickly and automatically reflected, and various types of topological view data that support customization are supported.

[0019] Furthermore, the step (2) comprises:

[0020] (2.1) The unidirectional constraint of the model inheritance relationship is that the direction is from the descendant model to the parent model;

[0021] (2.2) Models can only inherit from existing models;

[0022] (2.3) The unidirectional constraint of the association relationship between models, where the direction is from one model to another model described by its specific business meaning;

[0023] (2.4) A model in which a dependency exists must be directly or indirectly related to an independent model in which no dependency exists;

[0024] (2.5) The two models with dependency relationships are not interdependent;

[0025] (2.6) The inheritance relationship between models forms a directed acyclic graph;

[0026] (2.7) If the relationship between models forms a loop, then one of the relationships must be a nullable relationship.

[0027] Furthermore, the step (3) comprises:

[0028] (3.1) The model is the node, the inheritance and association relationships are the edges, and the addition, deletion, modification and query of the model are the related operation sets of the graph data set and the edge set;

[0029] (3.2) The graph is a collection of model abstractions, logically existing as the definitional superordinate of nodes and relationships;

[0030] (3.3) Nodes are divided into three categories: models, attributes, and instances. They have corresponding specific attributes such as name, encoding, value type, constraint, and nullable flag according to specific business conditions;

[0031] (3.4) Relationships are divided into four categories: model-to-model, model-to-attribute, model-to-instance, and instance-to-instance. Model-to-model includes inheritance, one-to-one association, and one-to-many association; model-to-attribute includes has association; model-to-instance includes generates association; and instance-to-instance includes universal association.

[0032] Furthermore, the step (4) comprises:

[0033] (4.1) Resource version judgment: Based on the resource data version number marked with a timestamp, the version of the ledger resources in the synchronized data and the stock resources in the graph database are judged. If the version number in the synchronized data is greater than the current version number, the subsequent processing is performed according to the following logic, otherwise the data is ignored;

[0034] (4.2) Deletion mark determination of resources. If it is marked as a resource to be deleted, the resource logical deletion mark is changed to the deletion state, and archiving is completed;

[0035] (4.3) Determine the new resource tag. If it is marked as a new resource to be added, the timestamp of the resource creation time is used as the version number to insert the new resource. If there is corresponding associated relationship data, it must be created and archived synchronously.

[0036] (4.4) Determine the update mark of the resource. If it is marked as a resource to be updated, the archive type of the existing resource and associated relationship data in the graph database is changed to a historical resource. The timestamp of the resource change time is used as the version number. The resource to be updated and the related associated relationship data are added to the graph database, and a historical record type association pointing to the archived resource is created.

[0037] Furthermore, the step (6) comprises:

[0038] (6.1) Add, delete, modify and query capabilities of basic elements: Support the addition, deletion, modification and query capabilities of independent points and edges when the number is large;

[0039] (6.2) Sparsely associated CRUD capabilities: Supports CRUD capabilities when there are a large number of nodes but a small number of edges between them;

[0040] (6.3) Large-scale edge query and insertion capabilities: Query and insertion capabilities support when there are large-scale edge sets on a single node;

[0041] (6.4) Query capability for strict matching of indefinite-length paths: When the query consists of indefinite-length paths, the query capability is supported by using explicit number of entries, labels, or other attribute conditions;

[0042] (6.5) Query capability for fuzzy matching of indefinite-length paths: When querying a path of indefinite length, the query is supported based on the characteristics of the starting node without limiting the number of hops.

[0043] (6.6) Model-resource detailed data query capability: According to the common model-resource query scenario, after constructing several layers of model queries, the query capability of resource details query is supported when there is a large resource node plane;

[0044] (6.7) Model-resource statistical data query capability: According to the common model-resource query scenario, after constructing several layers of model queries, the query capability of resource statistics based on conditions is supported when there is a large resource node plane.

[0045] Furthermore, the query function of the stock historical data in step (7) includes:

[0046] (7.1) Query historical ledger resources or associated data based on time intervals, and filter the corresponding time intervals through the version number of the ledger resources or associated data to complete the corresponding data query;

[0047] (7.2) Historical record-related data query based on ledger resource attributes: by specifying the ledger resource or ledger resource attribute value conditions, locking the corresponding ledger, and traversing its historical record type association to complete the corresponding data query;

[0048] (7.3) Based on the time slice, the ledger resource association network data query is carried out. By specifying the specific timestamp of the time slice, the ledger resource data within the range is filtered, and the corresponding association relationships are combined according to the result set to form an association network to complete the corresponding data query.

[0049] The ledger resource and association relationship data query system based on the graph database of the present invention includes an application layer, a service layer and a data layer;

[0050] The application layer includes:

[0051] Resource configuration system, which is used to provide real-time data acquisition capabilities for ledger resources and associated relationship data, including incremental data of data changes and scheduled or quantitative full data acquisition;

[0052] The upper-level query application system is used to provide the consumers of the business implementation with query services at the docking service layer to obtain relevant historical data;

[0053] The service layer includes:

[0054] The synchronization service system is used to obtain incremental data and timed or quantitative full data synchronization by connecting to the resource configuration system according to the above logic, and to perform corresponding data structure transformation according to the storage logic requirements;

[0055] The query service system is used to connect to the upper-level query application and provide historical data query capabilities for the ledger resources and associated relationship data stored in the system;

[0056] The data layer includes:

[0057] The graph database system is used to provide standard-compliant general graph database services and underlying graph storage capabilities.

[0058] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages: through the upstream and downstream integration of the resource configuration system and the graph database storage module, the resources and relationship data in the real-time resource configuration system are synchronized, and the historical resources and historical relationship specific data structures defined in the scheme of the present invention are reorganized and stored in the graph database, so as to realize the persistence of the historical data of the above data; at the same time, by utilizing the characteristics of the graph database, using the corresponding graph query language and encapsulating it, a convenient and fast-response data query system is constructed; it solves the problems of complex and disordered historical data in the traditional resource configuration system, which is not conducive to long-term preservation and low query efficiency, and realizes efficient access to ledger resources and historical data of relationship relationships. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 It is the overall system architecture diagram of the present invention;

[0060] Figure 2 It is a data synchronization flow chart of the present invention. DETAILED DESCRIPTION

[0061] The technical solution of the present invention is further described below in conjunction with the accompanying drawings.

[0062] like Figure 1 As shown, the ledger resource and association relationship data query system based on the graph database of the present invention includes an application layer, a service layer and a data layer;

[0063] The application layer includes:

[0064] Resource configuration system, which is used to provide real-time data acquisition capabilities for ledger resources and associated relationship data, including incremental data of data changes and scheduled or quantitative full data acquisition;

[0065] The upper-level query application system is used to provide the consumers of the business implementation with query services at the docking service layer to obtain relevant historical data;

[0066] The service layer includes:

[0067] The synchronization service system is used to obtain incremental data and timed or quantitative full data synchronization by connecting to the resource configuration system according to the above logic, and to perform corresponding data structure transformation according to the storage logic requirements;

[0068] The query service system is used to connect to the upper-level query application and provide historical data query capabilities for the ledger resources and associated relationship data stored in the system;

[0069] The data layer includes:

[0070] The graph database system is used to provide standard-compliant general graph database services and underlying graph storage capabilities.

[0071] like Figure 2 As shown, the data synchronization process of the present invention is as follows:

[0072] (1) Obtain resource data to be archived, actively initiate full data scan through synchronization service or passively receive incremental synchronization data from resource configuration system;

[0073] (2) Determine whether it is a full synchronization, based on the full mark in the initiator's request. If it is a full synchronization, jump to step (3); otherwise, jump to step (4);

[0074] (3) Synchronize resource version change judgment, by comparing the resource version number in the synchronized data with the latest version number in the graph database, to determine whether it is the latest resource. If it is the latest, jump to step (13) to archive, otherwise jump to step (4);

[0075] (4) Determine whether the data is a resource to be deleted, by judging the to-be-deleted mark in the synchronized data. If the judgment result is that the data is a resource to be deleted, jump to step (5); otherwise, jump to step (6);

[0076] (5) Logically delete the data corresponding to the resource to be deleted in the graph database, change the deletion mark to yes, and jump to step (13) for archiving;

[0077] (6) Determine whether it is a resource to be added, and judge it by the to-be-added mark in the synchronization data. If the judgment result is a resource to be added, jump to step (7); otherwise, jump to step (10);

[0078] (7) Obtain the creation time of the resource to be added, and use the creation time as the timestamp version number of the resource to be added;

[0079] (8) Insert the new resources, create the corresponding data nodes in the graph database, and complete the corresponding attribute creation according to the synchronized data attributes and the timestamp version number obtained in step 7;

[0080] (9) Insert the corresponding association relationship of the resource to be added, create the corresponding edge in the graph database according to the attachment association relationship in the synchronized data, jump to step (13), and complete the archiving;

[0081] (10) Obtain the update time of the resource to be updated, and use the update time as the timestamp version number of the resource to be updated;

[0082] (11) Change the corresponding data in the graph database of the resource to be updated and change its archive type to historical resource;

[0083] (12) Insert the resource to be updated, create the corresponding data node in the graph database, complete the corresponding attribute creation according to the synchronized data attributes and the timestamp version number obtained in step (10), and jump to step (9);

[0084] (13) Archiving is completed, and all the data from the above steps are stored in the database.

Claims

1. A method for storing ledger resources and associated relationship data based on a graph database, characterized in that: The steps include: (1) Graph storage methods based on general graph databases, i.e., database systems with entities and their relationships as the main storage objects and corresponding storage methods; (2) The object model's pre-inheritance characteristics, modeling rules, association characteristics, and corresponding necessary and feasible model design conditions based on the characteristics of the operation and maintenance object and the operation and maintenance requirements; (3) Adapt the current storage structure design based on the object model's pre-inheritance characteristics, modeling rules, association characteristics, and corresponding necessary and feasible model design conditions; (4) Incremental synchronization of ledger resources and associated relationship data of the resource configuration system, obtaining the changed content of ledger resources and associated relationship data from the upstream resource configuration system in real time or quasi-real time, and reorganizing the data structure through graph storage and corresponding persistence requirements, and finally implementing it in the graph database; (5) The ledger resources and associated relationship data of the resource configuration system are fully synchronized. The real-time data slices of the ledger resources and associated relationship data are obtained from the upstream resource configuration system in a timed or quantitative detection manner. The data structure is reorganized through graph storage and corresponding persistence requirements, and finally implemented in the graph database, and the existing data is overwritten and corrected. (6) The basic graph query capability of resource configuration data based on the graph database graph storage solution includes the following seven basic query capabilities and their availability requirements: the ability to add, delete, modify and query basic elements, the ability to add, delete, modify and query sparse associations, the ability to query and insert a large number of edges, the ability to query strict matches of indefinite length paths, the ability to query fuzzy matches of indefinite length paths, the ability to query detailed data of models and resources, and the ability to query statistical data of models and resources; (7) The query function of existing historical data determines the query version based on the timestamp version number, performs related queries according to the actual time slice it faces, and thereby brings out the result set consisting of the ledger resources and related relationship data at the corresponding time point.

2. The method for storing ledger resources and associated relationship data based on a graph database according to claim 1, characterized in that: The graph storage method in step (1) based on a general graph database includes: (1.1) The basic modeling method of general graph storage is based on entity and relationship modeling in the real world, including: graph, i.e., relationship graph, which is the logical separation unit of graph storage data; vertex, which generally refers to entity, i.e., the corresponding ledger resource in the resource allocation system; edge, which generally refers to the relationship between vertices; attribute, which is the business attribute contained in the vertex or edge; (1.2) A universal graph storage query language that defines query conditions in a natural language-friendly way to express business query requirements; (1.3) Complex relationship query and related algorithm capabilities. The library uses entities and relationships as basic units and provides various integrated algorithms for query and analysis, such as the shortest path algorithm and the minimum connected component algorithm. (1.4) Graph process, i.e. graph traversal capability support based on graph storage, including: traversal generators for specific graphs, domain languages, and execution engines; functional data flow processes that convert type objects into other types of objects; traversal structured definition language for original graph semantics; parallel graph process processing system on distributed clusters; vertex code logic execution that communicates through message passing; and parallel analysis MapReduce process for graph vertex computation process.

3. The method for storing ledger resources and associated relationship data based on a graph database according to claim 1, characterized in that: The single-step graph process classification of the graph process in the step (1.4) includes filtering class, mapping class, flat mapping class, side effect class and branch class.

4. The method for storing ledger resources and associated relationship data based on a graph database according to claim 1, characterized in that: In step (2), based on the characteristics of operation and maintenance objects and operation and maintenance requirements, traditional operation and maintenance objects and new operation and maintenance objects such as cloud environments coexist; object attributes meet the dual requirements of operation and maintenance management and asset maintenance; there are multiple inheritances between objects; there are customizable associations between objects; the runtime status of resources in the system is quickly and automatically reflected, and various types of topological view data that support customization are supported.

5. The method for storing ledger resources and associated relationship data based on a graph database according to claim 1, characterized in that: The step (2) comprises: (2.1) The unidirectional constraint of the model inheritance relationship is that the direction is from the descendant model to the parent model; (2.2) Models can only inherit from existing models; (2.3) The unidirectional constraint of the association relationship between models, where the direction is from one model to another model described by its specific business meaning; (2.4) A model in which a dependency exists must be directly or indirectly related to an independent model in which no dependency exists; (2.5) The two models with dependency relationships are not interdependent; (2.6) The inheritance relationship between models forms a directed acyclic graph; (2.7) If the relationship between models forms a loop, then one of the relationships must be a nullable relationship.

6. The method for storing ledger resources and associated relationship data based on a graph database according to claim 1, characterized in that: The step (3) comprises: (3.1) The model is the node, the inheritance and association relationships are the edges, and the addition, deletion, modification and query of the model are the related operation sets of the graph data set and the edge set; (3.2) The graph is a collection of model abstractions, logically existing as the definitional superordinate of nodes and relationships; (3.3) Nodes are divided into three categories: models, attributes, and instances. They have corresponding specific attributes such as name, encoding, value type, constraint, and nullable flag according to specific business conditions; (3.4) Relationships are divided into four categories: model-to-model, model-to-attribute, model-to-instance, and instance-to-instance. Model-to-model includes inheritance, one-to-one association, and one-to-many association; model-to-attribute includes has association; model-to-instance includes generates association; and instance-to-instance includes universal association.

7. The method for storing ledger resources and associated relationship data based on a graph database according to claim 1, characterized in that: The step (4) comprises: (4.1) Resource version judgment: Based on the resource data version number marked with a timestamp, the version of the ledger resources in the synchronized data and the stock resources in the graph database are judged. If the version number in the synchronized data is greater than the current version number, the subsequent processing is performed according to the following logic, otherwise the data is ignored; (4.2) Deletion mark determination of resources. If it is marked as a resource to be deleted, the resource logical deletion mark is changed to the deletion state, and archiving is completed; (4.3) Determine the new resource tag. If it is marked as a new resource to be added, the timestamp of the resource creation time is used as the version number to insert the new resource. If there is corresponding associated relationship data, it must be created and archived synchronously. (4.4) Determine the update mark of the resource. If it is marked as a resource to be updated, the archive type of the existing resource and associated relationship data in the graph database is changed to a historical resource. The timestamp of the resource change time is used as the version number. The resource to be updated and the related associated relationship data are added to the graph database, and a historical record type association pointing to the archived resource is created.

8. The method for storing ledger resources and associated relationship data based on a graph database according to claim 1, characterized in that: The step (6) comprises: (6.1) Add, delete, modify and query capabilities of basic elements: Support the addition, deletion, modification and query capabilities of independent points and edges when the number is large; (6.2) Sparsely associated CRUD capabilities: Supports CRUD capabilities when there are a large number of nodes but a small number of edges between them; (6.3) Large-scale edge query and insertion capabilities: Query and insertion capabilities support when there are large-scale edge sets on a single node; (6.4) Query capability for strict matching of indefinite-length paths: When the query consists of indefinite-length paths, the query capability is supported by using explicit number of entries, labels, or other attribute conditions; (6.5) Query capability for fuzzy matching of indefinite-length paths: When querying a path of indefinite length, the query is supported based on the characteristics of the starting node without limiting the number of hops. (6.6) Model-resource detailed data query capability: According to the common model-resource query scenario, after constructing several layers of model queries, the query capability of resource details query is supported when there is a large resource node plane; (6.7) Model-resource statistical data query capability: According to the common model-resource query scenario, after constructing several layers of model queries, the query capability of resource statistics based on conditions is supported when there is a large resource node plane.

9. The method for storing ledger resources and associated relationship data based on a graph database according to claim 1, characterized in that: The query function of the stock historical data in step (7) includes: (7.1) Query historical ledger resources or associated data based on time intervals, and filter the corresponding time intervals through the version number of the ledger resources or associated data to complete the corresponding data query; (7.2) Historical record-related data query based on ledger resource attributes: by specifying the ledger resource or ledger resource attribute value conditions, locking the corresponding ledger, and traversing its historical record type association to complete the corresponding data query; (7.3) Based on the time slice, the ledger resource association network data query is carried out. By specifying the specific timestamp of the time slice, the ledger resource data within the range is filtered, and the corresponding association relationships are combined according to the result set to form an association network to complete the corresponding data query.

10. A ledger resource and association relationship data query system based on a graph database, characterized in that: Includes application layer, service layer and data layer; The application layer includes: Resource configuration system, which is used to provide real-time data acquisition capabilities for ledger resources and associated relationship data, including incremental data of data changes and scheduled or quantitative full data acquisition; The upper-level query application system is used to provide the consumers of the business implementation with query services at the docking service layer to obtain relevant historical data; The service layer includes: The synchronization service system is used to obtain incremental data and timed or quantitative full data synchronization by connecting to the resource configuration system according to the above logic, and to perform corresponding data structure transformation according to the storage logic requirements; The query service system is used to connect to the upper-level query application and provide historical data query capabilities for the ledger resources and associated relationship data stored in the system; The data layer includes: The graph database system is used to provide standard-compliant general graph database services and underlying graph storage capabilities.

Citation Information

Cited By

  • Big data-oriented data correction fusion system and method

    CN120892420A

  • A big data-oriented data correction fusion system and method

    CN120892420B