Data warehouse system based on graph database and construction method

Abstract

The invention discloses a data warehouse system based on a graph database and a construction method, the system comprises a graph data storage layer and a platform service layer, the graph data storage layer comprises a physical large graph storage module and a business sub-graph storage module; the physical large graph storage module adopts a distributed graph storage engine and is responsible for storing enterprise core data elements and association relationships thereof; a graph database is adopted as a bottom storage facility; the service sub-graph storage module provides an independent sub-graph storage space for each service application based on the data of the physical large graph storage module; dynamic graph mode expansion is supported, business departments are allowed to add specific attributes and relationships, and customized business sub-graphs are formed; the platform service layer extracts data from each business system through a high-performance ETL engine provided by a graph database, and stores the data to the physical large graph storage module after data governance service cleaning and conversion; the method has the beneficial effect that the flexibility and adaptability of business application are remarkably improved.

Description

Technical Field

[0001] This invention belongs to the field of data warehouse technology, specifically relating to a data warehouse system and construction method based on graph database. Background Technology

[0002] A data warehouse is a system for storing and managing large amounts of data to support scenarios such as business intelligence, decision support, and historical analysis. It provides a unified data view by integrating data from multiple heterogeneous data sources to facilitate analysis and data mining.

[0003] Traditional data warehouses employ a layered, closed-loop architecture with clearly defined responsibilities for each layer, forming a complete chain of "data access - integration - storage - application." The core structure typically consists of four layers: The first layer is the data source layer, whose main function is to access internal and external structured data, such as data from business systems like ERP, CRM, and order systems, as well as CSV files and log databases, providing raw data for subsequent processing. The second layer is the ETL layer (extract-transform-load), which pulls full or incremental data from the data source on an hourly or daily basis. After processing such as cleaning (duplicate removal, completion), standardization (unifying field formats), and association (multi-source data concatenation), the data is loaded into the storage layer. Commonly used components include Informatica, DataStage, and HiveETL. The third layer is the storage layer (core layer), which includes the ODS layer, DW layer, and DM layer: The ODS layer (operational data storage) stores copies of the original data, with a structure consistent with the data source, primarily used for data traceability; the DW layer (data warehouse layer) is organized by "customer domain" and "transaction domain." The first layer integrates data from subject areas and stores detailed data using a relational model (such as the third normal form). The second layer (DM layer) is geared towards specific business scenarios such as sales analysis and risk control analysis, and uses a star or snowflake schema (fact table + dimension table) to store pre-summarized data. Common storage media include Oracle, Teradata (structured storage), and HDFS (batch storage). The third layer is the application layer, which provides SQL query interfaces for upper-layer tools and supports the query and report generation needs of BI tools (such as Tableau and Power BI).

[0004] Traditional data warehouses operate around three core mechanisms designed for "offline batch processing": First, dimensional modeling principles. Taking the Kimball architecture as an example, a star schema is built using "fact tables + dimension tables"—the fact table stores quantitative indicators such as transaction amount and order number, while the dimension tables store descriptive attributes such as time, region, and customer type, thereby reducing the number of table joins and improving batch statistical efficiency. Second, ETL batch scheduling mechanisms, where data processing is executed on a periodic task basis (e.g., extracting the full transaction data of the previous day at midnight every day), updating the storage layer data through "full coverage" or "incremental appending," such as partitioning the DW layer by date, with new data directly written to the new partition. Third, offline computing engine support, relying on batch processing engines such as HiveMR and Spark Batch to perform data transformation and statistics. The computing logic is specifically optimized for "large volume, low timeliness data," such as processing TB-level historical data at once, rather than pursuing real-time response. Therefore, traditional solutions have the following problems:

[0005] Question 1: Insufficient real-time performance and poor computational performance

[0006] Data takes several hours to become available from generation (e.g., transaction data for the day can only be viewed the next day). The root causes, from an architectural and principle perspective, are threefold: First, the ETL layer uses "periodic scheduling" rather than real-time stream processing, with data extraction intervals at least on the order of hours, making it impossible to capture real-time data updates from user transactions, app behavior logs, and other data sources. Second, the storage layer (DW layer, DM layer) uses a "partition append" mode (e.g., partitioning by day), meaning new data can only be written after batch ETL is complete, preventing real-time updates. Third, the batch processing engine (e.g., Hive MR) takes a long time to start (up to minutes), does not support "millisecond / second" real-time queries, and cannot respond quickly even when real-time data is integrated.

[0007] Question 2: Poor scalability and high cost

[0008] Once data volume exceeds the petabyte (PB) level, expansion becomes time-consuming (from hours to days), hardware costs increase exponentially, and business operations are easily interrupted during the expansion process. The root causes are related to storage architecture and resource configuration logic: First, the storage layer relies on "static sharding / vertical scaling." Traditional data warehouses (such as Teradata) use "vertical scaling" (upgrading single-node CPUs and memory), which has physical performance limitations. Even distributed data warehouses like Hive use "static sharding by time / topic" (such as partitioning by year and month), requiring a full migration of historical shard data during expansion (e.g., migrating 2023 data from 3 nodes to 5 nodes), which is time-consuming and interrupts offline tasks. Second, computing and storage are coupled. Traditional data warehouses are mostly "computing-storage integrated" architectures (such as Oracle RAC), requiring simultaneous expansion of computing and storage resources during expansion, making independent expansion on demand impossible (e.g., if storage alone is insufficient, computing nodes still need to be purchased), resulting in resource waste.

[0009] Question 3: Insufficient application flexibility and poor adaptation to business perspectives

[0010] For example, it only supports analysis based on "predefined dimensions" (such as sales queries based on "time-region-product"). Adding new dimensions (such as "payment method") requires reconstructing the data model. Furthermore, it cannot meet the personalized needs of departments (such as operations requiring "hourly-store-user profile" data, and finance requiring "daily-subject-cost center" data). The root cause lies in the storage layer design and modeling perspective: First, the DM layer adopts a "pre-aggregation design," pre-aggregating data according to fixed dimensions (such as "monthly-region") to improve query efficiency. Adding new dimensions requires redesigning the fact table and dimension table, and rerunning historical data, a process that can take several days. Second, the modeling perspective is "global." Traditional data warehouses model according to "enterprise-level subject domains" (such as the "transaction domain" of the DW layer covering all company transaction data), without considering departmental "personalized perspectives" (such as operations focusing on "user activity dimension," and finance focusing on "tax dimension"). This results in departmental users needing to piece together data across multiple data marts, leading to low application efficiency. Summary of the Invention

[0011] In view of the technical deficiencies mentioned in the background art, the purpose of this invention is to provide a data warehouse system and construction method based on graph database, aiming to at least partially solve one of the technical problems in the prior art.

[0012] To achieve the above objectives, in a first aspect, embodiments of the present invention provide a data warehouse system based on a graph database. The system includes a graph data storage layer and a platform service layer. The graph data storage layer includes a physical large graph storage module and a business sub-graph storage module.

[0013] The physical large image storage module is used for:

[0014] It adopts a distributed graph storage engine to store the core data elements of the enterprise and their relationships, and supports efficient access to trillions of node-edge data.

[0015] A graph database is used as the underlying storage facility;

[0016] The business subgraph storage module is used for:

[0017] Based on the data from the physical large graph storage module, it provides independent subgraph storage space for each business application; and supports dynamic graph mode expansion, allowing business departments to add unique attributes and relationships to form customized business subgraphs;

[0018] The platform service layer is used to extract data from various business systems through the high-performance ETL engine provided by the graph database, and after data cleaning and transformation by the data governance service, it is stored in the physical large graph storage module.

[0019] As one specific implementation of this application, the extracted data, after being cleaned by a data governance service, specifically includes:

[0020] Leveraging the built-in batch write tools of graph databases, efficient data transfer from heterogeneous business systems to graph databases can be achieved.

[0021] During data cleaning and transformation, a two-layer data processing mechanism is adopted to ensure that the quality of data is controllable during the access and storage stages.

[0022] As one specific implementation of this application, the two-layer data processing mechanism specifically includes:

[0023] The first layer is to support users to pass in custom SQL statements through graph database when reading data from the business system relational database, so as to directly achieve source-level dirty data filtering and field format standardization during the data reading stage;

[0024] The second layer is the storage layer verification. When data is written to the graph database, the kernel will perform mandatory consistency verification, including data integrity verification, data type consistency verification, and edge data constraint verification.

[0025] As a specific implementation of this application, the advanced architecture of the graph database's native distributed graph storage engine is used to adopt an intelligent sharding strategy based on graph structure to automatically divide the data into multiple shards and distribute them across the nodes of the cluster. This storage mechanism not only efficiently organizes nodes, edges and attributes through the principle of data locality, reducing cross-node query overhead, but also improves data availability and load balancing by using a multi-replica mechanism.

[0026] As one specific implementation of this application, the platform service layer also provides:

[0027] The access control service is used to implement fine-grained access control, supporting column-level, row-level, and graph-structure-level access control.

[0028] Operation and maintenance monitoring services are used to provide operation and maintenance capabilities such as cluster management, performance monitoring, and fault recovery.

[0029] The Computation Engine service integrates various computing modes, including interactive queries, batch analysis, and real-time graph algorithms.

[0030] As one specific implementation of this application, the process for forming a customized business sub-graph is as follows:

[0031] First, during the permission allocation phase, the permission management capabilities of the graph database are used to configure fine-grained data access permissions for each user; attribute-level permission control is supported, which restricts users to access only specific fields through permission policies, while other data is not visible, thereby ensuring data security and compliance.

[0032] Secondly, in the subgraph generation stage, user-specific subgraphs are constructed using two methods;

[0033] Logical subgraphs are generated by the graph database's built-in graph computing engine, mapping partial data from the physical graph based on query or graph traversal operations, without requiring physical storage;

[0034] The physical subgraph extracts data using graph query language or user-defined functions, and uses data import tools or APIs to materialize and store the subgraph data as an independent graph space; all subgraphs are strictly controlled by the permission system and are only visible to the users to which they belong.

[0035] Finally, in the data application and feedback phase, business applications can perform computational tasks on the physical subgraph; they can receive data unique to the upstream application department and write it into the subgraph in real time through the data writing interface provided by the graph database, and participate in the graph computation or analysis process; the calculated indicator data can be persisted to the physical subgraph, and the result data can be sent back to the physical graph through the API interface, realizing cross-subgraph data reuse and unified management, forming a complete data closed loop.

[0036] Secondly, embodiments of the present invention also provide a method for constructing a data warehouse based on a graph database, applied to a graph database-based data warehouse system as described in the first aspect, the system comprising a graph data storage layer and a platform service layer; the method includes the following steps:

[0037] Data is extracted from various business systems using the high-performance ETL engine provided by the graph database. After being cleaned and transformed by the data governance service, the data is written to the graph database for storage.

[0038] A distributed graph storage engine is used to store and write core enterprise data elements and their relationships, in order to generate a physical graph that serves as an enterprise-level data foundation and fully preserves the core data entities and their relationships.

[0039] Based on the physical graph data, it provides independent subgraph storage space for each business application; and supports dynamic graph mode extension, allowing business departments to add unique attributes and relationships to form customized business subgraphs.

[0040] The technical solutions provided in the embodiments of the present invention have the following beneficial effects:

[0041] 1. At the storage technology level, a native distributed graph database is adopted to replace the multi-layered heterogeneous storage architecture of the traditional data warehouse, realizing the intensive reconstruction of the storage architecture; the decentralized storage management, data transformation and computing acceleration capabilities are integrated into a unified engine, significantly reducing system complexity and operation and maintenance costs;

[0042] 2. Propose an innovative architecture of "physical large map - business sub-map", which subverts the traditional three-layer structure of data warehouse ODS-DW-DM. The dual-mode architecture of "physical large map - business sub-map" solves the problem of rigidity of pre-aggregation model, and the business sub-map realizes on-demand adjustment from the business perspective;

[0043] 3. Unifying governance and agility, ensuring consistency with the physical big picture, empowering business sub-graphs, releasing business agility under standardized governance, overcoming the problems of insufficient application flexibility and poor adaptation to business perspectives, while ensuring the timeliness of analysis, realizing new dimensions of instant support capabilities, department-level customized analysis capabilities, and multi-application parallel development capabilities, significantly improving the flexibility and adaptability of business applications. Attached Figure Description

[0044] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below.

[0045] Figure 1 This is an architecture diagram of a data warehouse system based on a graph database provided in an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of a data governance service provided in an embodiment of the present invention;

[0047] Figure 3 This is a schematic flowchart illustrating anti-fraud and internal auditing as examples provided in an embodiment of the present invention;

[0048] Figure 4 This is a flowchart of a data warehouse construction method based on a graph database provided in an embodiment of the present invention. Detailed Implementation

[0049] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0050] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0051] JDBC (Java Database Connectivity) is an application programming interface in the Java language that specifies how client programs access databases, providing methods such as querying and updating data in the database.

[0052] It should be noted that, unless otherwise stated, the technical terms used in this embodiment have the common meaning as understood in the relevant technical field.

[0053] Please refer to Figure 1 , Figure 2 This invention provides a data warehouse system based on a graph database. The system includes a graph data storage layer and a platform service layer. The graph data storage layer includes a physical large graph storage module and a business subgraph storage module.

[0054] The physical large image storage module is used for:

[0055] It adopts a distributed graph storage engine to store the core data elements of the enterprise and their relationships, and supports efficient access to trillions of node-edge data.

[0056] A graph database is used as the underlying storage facility;

[0057] The business subgraph storage module is used for:

[0058] Based on the data from the physical large graph storage module, it provides independent subgraph storage space for each business application; and supports dynamic graph mode expansion, allowing business departments to add unique attributes and relationships to form customized business subgraphs;

[0059] The platform service layer is used to extract data from various business systems through the high-performance ETL engine provided by the graph database, and after data cleaning and transformation by the data governance service, it is stored in the physical large graph storage module.

[0060] In this embodiment, the extracted data, after being cleaned by a data governance service, specifically includes:

[0061] Leveraging the built-in batch write tools of graph databases, efficient data transfer from heterogeneous business systems to graph databases can be achieved.

[0062] During data cleaning and transformation, a two-layer data processing mechanism is adopted to ensure that the quality of data is controllable during the access and storage stages;

[0063] The data governance service includes a data cleaning, transformation, and standardization pipeline, and adopts a distributed parallel processing architecture.

[0064] Furthermore, the two-layer data processing mechanism specifically includes:

[0065] The first layer, when reading data from the business system's relational database, uses a graph database to support users passing in custom SQL statements (e.g., SELECT cleaned_id, CAST(amount AS FLOAT) FROM raw_table WHEREamount IS NOT NULL), enabling source-level dirty data filtering and field format standardization directly during the data reading phase.

[0066] The second layer is the storage layer verification. When data is written to the graph database, the kernel will perform mandatory consistency verification, including data integrity verification, data type consistency verification, and edge data constraint verification.

[0067] Specifically, data integrity verification: ensure that the primary key field is not null;

[0068] Data type consistency check: Reject writing data that violates the type definition (such as writing a string into an integer field);

[0069] Edge data constraint verification: The start and end points of an edge must both exist before the relationship can be written.

[0070] The above two-layer mechanism ensures that the quality of data is controllable during the access and storage stages.

[0071] During implementation, distributed storage and partitioning are used:

[0072] The data is ultimately persisted to a graph database. Leveraging the advanced architecture of the graph database's native distributed graph storage engine, the system employs an intelligent sharding strategy based on the graph structure, automatically dividing the data into multiple shards and distributing them across the nodes of the cluster. This storage mechanism not only efficiently organizes vertices, edges, and attributes through the principle of data locality, reducing cross-node query overhead, but also utilizes a multi-replica mechanism to improve data availability and load balancing. With this architecture, the graph database can significantly optimize the performance of depth graph traversal, multi-hop queries, and complex pattern matching, greatly reducing query latency, thereby comprehensively improving the system's retrieval efficiency and scalability in high-concurrency, large-scale graph data scenarios.

[0073] When applied, the platform service layer also provides:

[0074] The access control service is used to implement fine-grained access control, supporting column-level, row-level, and graph-structure-level access control.

[0075] Operation and maintenance monitoring services are used to provide operation and maintenance capabilities such as cluster management, performance monitoring, and fault recovery.

[0076] The Computation Engine service integrates various computing modes, including interactive queries, batch analysis, and real-time graph algorithms.

[0077] In this embodiment, the business department applies for data access permissions through the permission management service, extracts relevant data from the physical map, and can selectively add department-specific data to form a business sub-map; the process of forming a customized business sub-map is as follows:

[0078] First, during the permission allocation phase, the permission management capabilities of the graph database are used to configure fine-grained data access permissions for each user; attribute-level permission control is supported, and permission policies restrict users to accessing only specific fields (such as only the account field), while other data is not visible, thereby ensuring data security and compliance;

[0079] Secondly, in the subgraph generation stage, user-specific subgraphs are constructed using two methods;

[0080] Logical subgraphs are generated by the graph database's built-in graph computing engine, mapping partial data from the physical graph based on query or graph traversal operations, without requiring physical storage;

[0081] The physical subgraph extracts data using graph query language or user-defined functions, and uses data import tools or APIs to materialize and store the subgraph data as an independent graph space; all subgraphs are strictly controlled by the permission system and are only visible to the users to which they belong.

[0082] Finally, in the data application and feedback phase, business applications can perform computational tasks on the physical subgraph; they can receive data unique to the upstream application department and write it into the subgraph in real time through the data writing interface provided by the graph database, and participate in the graph computation or analysis process; the calculated indicator data can be persisted to the physical subgraph, and the result data can be sent back to the physical graph through the API interface, realizing cross-subgraph data reuse and unified management, forming a complete data closed loop.

[0083] For ease of understanding of the present invention, please refer to Figure 3 The flowchart shown uses anti-fraud and internal audit as examples to illustrate the implementation path of a typical application scenario for a graph data warehouse architecture. It should be noted that this flowchart only uses anti-fraud and internal audit as example scenarios; in actual applications, the graph warehouse architecture has universal scalability—by independently building dedicated subgraphs for different business scenarios, diverse business query needs can be supported.

[0084] In summary, the main innovations of this technical solution compared to traditional data warehouses are as follows:

[0085] At the storage technology level, this solution replaces the multi-layered heterogeneous storage architecture of traditional data warehouses (such as relational databases, HDFS, and columnar storage) with a native distributed graph database, achieving a centralized reconstruction of the storage architecture. This technology replaces the traditional composite storage architecture of data warehouses (OLTP database + OLAP data warehouse + big data platform) with a unified graph storage engine, compressing the original 5-7 layer architecture into a single storage layer and eliminating cross-system data migration processes. Simultaneously, based on the autonomous operation and maintenance capabilities of the graph database, it achieves automated data balancing (built-in intelligent partitioning algorithm), self-healing fault recovery (node ​​switching <30 seconds), and unified monitoring, significantly reducing the manpower required for operation and maintenance. Through fundamental architectural reconstruction, this solution integrates distributed storage management, data transformation, and computational acceleration capabilities into a unified engine, significantly reducing system complexity and operation and maintenance costs.

[0086] The high availability and autonomous operation and maintenance capabilities of native distributed graph databases primarily rely on their distributed data storage and multi-replica mechanisms. Based on these two capabilities, the failure of a single node in a distributed graph database cluster will not affect the normal operation of the overall service, provided the replica data is complete. Simultaneously, the graph database's built-in monitoring process continuously monitors the running status of cluster nodes. Once a node anomaly is detected, the failed node service is automatically restarted. After the node recovers, a data balancing process is automatically triggered, fully realizing self-healing and data recovery without manual intervention, thus reducing operational costs.

[0087] By proposing an innovative "Physical Large Graph - Business Subgraph" architecture, this architecture overturns the traditional "ODS-DW-DM" three-layer structure of data warehouses, fundamentally solving the shortcomings of traditional data warehouses in terms of data application flexibility and business adaptability. This architecture uses a "physical large graph" as the enterprise-level data foundation, fully preserving core data entities and relationships. Simultaneously, it achieves on-demand data extraction and application through a "business subgraph" mechanism, breaking through the limitations of traditional pre-aggregation models and supporting dynamic data organization based on business scenarios. At the technical implementation level, this architecture possesses three breakthrough capabilities: first, it supports dynamic data extraction from the physical large graph according to business needs, allowing the addition of department-specific data to form independent subgraphs; second, it implements fine-grained access control based on a graph database; and third, it utilizes high-performance graph traversal capabilities to ensure real-time data access. These features enable various business applications to build their own dedicated analysis environments in parallel. This architecture directly addresses two core pain points of traditional data warehouses: the difficulty of dimensional expansion caused by pre-aggregation models, and the inability of global perspective modeling to meet the personalized needs of departments. By ensuring the consistency and integrity of enterprise-level data through physical maps, and empowering business departments with data autonomy through business sub-maps, the system achieves new dimensions of real-time support capabilities, department-level customized analysis capabilities, and multi-application parallel development capabilities while ensuring data governance standards, significantly improving the flexibility and adaptability of business applications.

[0088] The innovative business subgraph mechanism supports business departments in flexibly building their own dedicated data analysis environments. Each application can dynamically extract data from the physical graph in real time to form independent subgraphs according to scenario requirements, and can supplement them with department-specific data sources. This architecture achieves a breakthrough in real-time stream computing capabilities—when an application needs to process real-time data streams generated by upstream systems, it can directly access the business subgraph for instant computation, with the results fed back to downstream systems in real time. Simultaneously, computation results with reusable value can be automatically synchronized to the physical graph storage, forming a closed-loop data circulation system of "real-time access - instant computation - result feedback - value accumulation," ensuring the timeliness of analysis while continuously enriching the enterprise's core data assets.

[0089] Graph warehouses systematically address the four major pain points of traditional data warehouses—"architectural redundancy, rigid models, lack of real-time capabilities, and inefficient collaboration"—through distributed graph storage, dynamic subgraph mechanisms, and real-time closed-loop design, achieving a next-generation data architecture paradigm of "global governance and local agility."

[0090] The above solution has the following beneficial effects:

[0091] 1. At the storage technology level, a native distributed graph database is adopted to replace the multi-layered heterogeneous storage architecture of the traditional data warehouse, realizing the intensive reconstruction of the storage architecture; the decentralized storage management, data transformation and computing acceleration capabilities are integrated into a unified engine, significantly reducing system complexity and operation and maintenance costs;

[0092] 2. Propose an innovative architecture of "physical large map - business sub-map", which subverts the traditional three-layer structure of data warehouse ODS-DW-DM. The dual-mode architecture of "physical large map - business sub-map" solves the problem of rigidity of pre-aggregation model, and the business sub-map realizes on-demand adjustment from the business perspective;

[0093] 3. Unifying governance and agility, ensuring consistency with the physical big picture, empowering business sub-graphs, releasing business agility under standardized governance, overcoming the problems of insufficient application flexibility and poor adaptation to business perspectives, while ensuring the timeliness of analysis, realizing new dimensions of instant support capabilities, department-level customized analysis capabilities, and multi-application parallel development capabilities, significantly improving the flexibility and adaptability of business applications.

[0094] Reference Figure 4 Based on the same inventive concept, this invention also provides a method for constructing a data warehouse based on a graph database, applied to the graph database-based data warehouse system described above, the system including a graph data storage layer and a platform service layer; the method includes the following steps:

[0095] S101 extracts data from various business systems through the high-performance ETL engine provided by the graph database, and after being cleaned and transformed by the data governance service, the data is written to the graph database for storage.

[0096] S102 uses a distributed graph storage engine to store and write core enterprise data elements and their relationships, in order to generate a physical graph that is an enterprise-level data foundation and fully preserves the core data entities and their relationships.

[0097] S103, based on physical large graph data, provides independent subgraph storage space for each business application; and supports dynamic graph mode extension, allowing business departments to add unique attributes and relationships to form customized business subgraphs.

[0098] Furthermore, the extracted data, after being cleaned by data governance services, specifically includes:

[0099] Leveraging the built-in batch write tools of graph databases, efficient data transfer from heterogeneous business systems to graph databases can be achieved.

[0100] During data cleaning and transformation, a two-layer data processing mechanism is adopted to ensure that the quality of data is controllable during the access and storage stages.

[0101] In implementation, the method further includes: using the advanced architecture of the graph database's native distributed graph storage engine, adopting an intelligent sharding strategy based on graph structure to automatically divide the data into multiple shards and distribute them across the nodes of the cluster; this storage mechanism not only efficiently organizes nodes, edges and attributes through the principle of data locality, reducing cross-node query overhead, but also improves data availability and load balancing by utilizing a multi-replica mechanism.

[0102] The process for forming a customized business sub-graph is as follows:

[0103] First, during the permission allocation phase, the permission management capabilities of the graph database are used to configure fine-grained data access permissions for each user; attribute-level permission control is supported, which restricts users to access only specific fields through permission policies, while other data is not visible, thereby ensuring data security and compliance.

[0104] Secondly, in the subgraph generation stage, user-specific subgraphs are constructed using two methods;

[0105] Logical subgraphs are generated by the graph database's built-in graph computing engine, mapping partial data from the physical graph based on query or graph traversal operations, without requiring physical storage;

[0106] The physical subgraph extracts data using graph query language or user-defined functions, and uses data import tools or APIs to materialize and store the subgraph data as an independent graph space; all subgraphs are strictly controlled by the permission system and are only visible to the users to which they belong.

[0107] Finally, in the data application and feedback phase, business applications can perform computational tasks on the physical subgraph; they can receive data unique to the upstream application department and write it into the subgraph in real time through the data writing interface provided by the graph database, and participate in the graph computation or analysis process; the calculated indicator data can be persisted to the physical subgraph, and the result data can be sent back to the physical graph through the API interface, realizing cross-subgraph data reuse and unified management, forming a complete data closed loop.

[0108] It should be noted that for a more detailed description of the workflow of the method embodiments, please refer to the aforementioned system embodiments section, which will not be repeated here.

[0109] The entire plan has the following beneficial effects:

[0110] 1. At the storage technology level, a native distributed graph database is adopted to replace the multi-layered heterogeneous storage architecture of the traditional data warehouse, realizing the intensive reconstruction of the storage architecture; the decentralized storage management, data transformation and computing acceleration capabilities are integrated into a unified engine, significantly reducing system complexity and operation and maintenance costs;

[0111] 2. Propose an innovative architecture of "physical large map - business sub-map", which subverts the traditional three-layer structure of data warehouse ODS-DW-DM. The dual-mode architecture of "physical large map - business sub-map" solves the problem of rigidity of pre-aggregation model, and the business sub-map realizes on-demand adjustment from the business perspective;

[0112] 3. Unifying governance and agility, ensuring consistency with the physical big picture, empowering business sub-graphs, releasing business agility under standardized governance, overcoming the problems of insufficient application flexibility and poor adaptation to business perspectives, while ensuring the timeliness of analysis, realizing new dimensions of instant support capabilities, department-level customized analysis capabilities, and multi-application parallel development capabilities, significantly improving the flexibility and adaptability of business applications.

[0113] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0114] In the embodiments provided in this application, it should be understood that the disclosed system and construction method can be implemented in other ways. For example, the embodiments described above are merely illustrative. For instance, the division of layers is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. In addition, the mutual coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices or units, or it may be an electrical, mechanical or other form of connection.

[0115] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing unit, or each module can exist physically separately, or two or more modules can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0116] If the integrated module is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0117] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A data warehouse system based on a graph database, characterized in that, The system includes a graph data storage layer and a platform service layer. The graph data storage layer includes a physical large graph storage module and a business sub-graph storage module. The physical large image storage module is used for: It adopts a distributed graph storage engine to store the core data elements of the enterprise and their relationships, and supports efficient access to trillions of node-edge data. A graph database is used as the underlying storage facility; The business subgraph storage module is used for: Based on the data from the physical large graph storage module, it provides independent subgraph storage space for each business application; and supports dynamic graph mode expansion, allowing business departments to add unique attributes and relationships to form customized business subgraphs; The platform service layer is used to extract data from various business systems through the high-performance ETL engine provided by the graph database, and after data cleaning and transformation by the data governance service, it is stored in the physical large graph storage module.

2. The data warehouse system based on a graph database as described in claim 1, characterized in that, The extracted data, after being cleaned by data governance services, specifically includes: Leveraging the built-in batch write tools of graph databases, efficient data transfer from heterogeneous business systems to graph databases can be achieved; During data cleaning and transformation, a two-layer data processing mechanism is adopted to ensure that the quality of data is controllable during the access and storage stages.

3. A data warehouse system based on a graph database as described in claim 2, characterized in that, The two-layer data processing mechanism specifically includes: The first layer is to support users to pass in custom SQL statements through graph database when reading data from the business system relational database, so as to directly achieve source-level dirty data filtering and field format standardization during the data reading stage; The second layer is the storage layer verification. When data is written to the graph database, the kernel will perform mandatory consistency verification, including data integrity verification, data type consistency verification, and edge data constraint verification.

4. A data warehouse system based on a graph database as described in claim 2, characterized in that, Through the advanced architecture of its native distributed graph storage engine, the graph database adopts an intelligent sharding strategy based on graph structure to automatically divide the data into multiple shards and distribute them across the nodes of the cluster. This storage mechanism not only efficiently organizes nodes, edges, and attributes through the principle of data locality, reducing cross-node query overhead, but also improves data availability and load balancing by utilizing a multi-replica mechanism.

5. A data warehouse system based on a graph database as described in any one of claims 1 to 4, characterized in that, The platform service layer also provides: The access control service is used to implement fine-grained access control, supporting column-level, row-level, and graph-structure-level access control. Operation and maintenance monitoring services are used to provide operation and maintenance capabilities such as cluster management, performance monitoring, and fault recovery. The Computation Engine service integrates various computing modes, including interactive queries, batch analysis, and real-time graph algorithms.

6. A data warehouse system based on a graph database as described in claim 5, characterized in that, The process for generating customized business sub-graphs is as follows: First, during the permission allocation phase, the permission management capabilities of the graph database are used to configure fine-grained data access permissions for each user; attribute-level permission control is supported, which restricts users to access only specific fields through permission policies, while other data is not visible, thereby ensuring data security and compliance. Secondly, in the subgraph generation stage, user-specific subgraphs are constructed using two methods; Logical subgraphs are generated by the graph database's built-in graph computing engine, mapping partial data from the physical graph based on query or graph traversal operations, without requiring physical storage; The physical subgraph extracts data using graph query language or user-defined functions, and uses data import tools or APIs to materialize and store the subgraph data as an independent graph space; all subgraphs are strictly controlled by the permission system and are only visible to the users to which they belong. Finally, in the data application and feedback phase, business applications can perform computational tasks on the physical subgraph; they can receive data unique to the upstream application department and write it into the subgraph in real time through the data writing interface provided by the graph database, and participate in the graph computation or analysis process; the calculated indicator data can be persisted to the physical subgraph, and the result data can be sent back to the physical graph through the API interface, realizing cross-subgraph data reuse and unified management, forming a complete data closed loop.

7. A method for constructing a data warehouse based on a graph database, characterized in that, The method is applied to a graph database-based data warehouse system as described in claim 1, the system comprising a graph data storage layer and a platform service layer; the method comprises the following steps: Data is extracted from various business systems using the high-performance ETL engine provided by the graph database. After being cleaned and transformed by the data governance service, the data is written to the graph database for storage. A distributed graph storage engine is used to store and write core enterprise data elements and their relationships, in order to generate a physical graph that serves as an enterprise-level data foundation and fully preserves the core data entities and their relationships. Based on the physical graph data, it provides independent subgraph storage space for each business application; and supports dynamic graph mode extension, allowing business departments to add unique attributes and relationships to form customized business subgraphs.

8. The data warehouse construction method based on graph database as described in claim 7, characterized in that, The extracted data, after being cleaned by data governance services, specifically includes: Leveraging the built-in batch write tools of graph databases, efficient data transfer from heterogeneous business systems to graph databases can be achieved; During data cleaning and transformation, a two-layer data processing mechanism is adopted to ensure that the quality of data is controllable during the access and storage stages.

9. A data warehouse construction method based on a graph database as described in claim 7 or 8, characterized in that, The method further includes: Through the advanced architecture of its native distributed graph storage engine, the graph database adopts an intelligent sharding strategy based on graph structure to automatically divide data into multiple shards and distribute them across the nodes of the cluster. This storage mechanism not only efficiently organizes nodes, edges and attributes through the principle of data locality, reducing cross-node query overhead, but also improves data availability and load balancing by using a multi-replica mechanism.

10. A data warehouse construction method based on a graph database as described in claim 9, characterized in that, The process for generating customized business sub-graphs is as follows: First, during the permission allocation phase, the permission management capabilities of the graph database are used to configure fine-grained data access permissions for each user; attribute-level permission control is supported, which restricts users to access only specific fields through permission policies, while other data is not visible, thereby ensuring data security and compliance. Secondly, in the subgraph generation stage, user-specific subgraphs are constructed using two methods; Logical subgraphs are generated by the graph database's built-in graph computing engine, mapping partial data from the physical graph based on query or graph traversal operations, without requiring physical storage; The physical subgraph extracts data using graph query language or user-defined functions, and uses data import tools or APIs to materialize and store the subgraph data as an independent graph space; all subgraphs are strictly controlled by the permission system and are only visible to the users to which they belong. Finally, in the data application and feedback phase, business applications can perform computational tasks on the physical subgraph; they can receive data unique to the upstream application department and write it into the subgraph in real time through the data writing interface provided by the graph database, and participate in the graph computation or analysis process; the calculated indicator data can be persisted to the physical subgraph, and the result data can be sent back to the physical graph through the API interface, realizing cross-subgraph data reuse and unified management, forming a complete data closed loop.

Images