Enterprise Data Operation Multi-Source Data Virtualization Access and Automated Indicator Management Methods and Systems

Abstract

This application relates to a method and system for multi-source data virtualization access and automated indicator management in enterprise data operations, belonging to the field of computer technology. It aims to solve the problems of fragmented indicator definitions, frequent logical conflicts, static dependency management, and rigid coupling of verification rules in existing technologies. The method includes: constructing a unified virtualized data access layer to integrate heterogeneous data sources; decomposing indicators into versionable semantic atoms and registering them to an atomic knowledge base; dynamically constructing a directed acyclic graph of indicator dependencies based on semantic atom combinations; dynamically binding data quality rules to semantic atoms; parsing indicator requests through a virtualized query engine, traversing the graph in reverse to aggregate dependencies and rules, generating and executing local queries, and automatically verifying the quality of the resulting data. This application achieves unified indicator definitions, automatic dependency propagation, embedded quality verification, and transparent multi-source access, significantly improving data consistency, governance efficiency, and system flexibility.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, specifically relating to a method and system for enterprise data operation multi-source data virtualization access and automated indicator management. Background Technology

[0002] In enterprise data operations, the integration and unified management of multi-source heterogeneous data are crucial for supporting digital decision-making. However, existing technical solutions have serious shortcomings in indicator governance, leading to poor data consistency and low decision credibility, mainly in the following three aspects:

[0003] First, the definition of the same business metric (such as "user activity") is scattered across different systems, ETL scripts, and reporting tools, lacking a unified semantic standard, which leads to frequent cross-departmental data conflicts. For example, the different definitions of "high-net-worth customers" in different systems of a certain bank directly cause customer segmentation bias.

[0004] Secondly, existing indicator libraries mostly adopt static management, which cannot dynamically track the dependencies between indicators. When the underlying data or basic indicators change, it takes several hours to conduct manual impact analysis, and the error rate is extremely high, which seriously hinders the agile evolution of the indicator system.

[0005] Third, quality rules are often hard-coded and bound to specific indicators. New business scenarios require development intervention, the launch cycle is long, rule verification is delayed, and problems are often only exposed in the final reporting stage.

[0006] Therefore, there is an urgent need in this field for an innovative solution that can achieve atomic definition of indicators, dynamic management of dependencies, and embedded quality verification, in order to fundamentally solve the above problems. Summary of the Invention

[0007] The purpose of this invention is to overcome the technical defects of existing enterprise data operations, such as fragmented indicator definitions, frequent logical conflicts, static dependency management, and rigid coupling of verification rules, and to provide a method and system for multi-source data virtualization access and automated indicator management in enterprise data operations.

[0008] This invention constructs a semantic atomic model for metrics, breaking down complex business metrics into indivisible, independently versionable semantic atoms, and building a dynamic metric dependency graph based on this. When an underlying semantic atom changes, the system automatically propagates the impact along the graph and provides real-time alerts for all affected derived metrics. Simultaneously, this invention dynamically binds data quality verification rules to semantic atoms, automatically aggregating relevant rules through a virtualized query engine during metric calculation. This achieves automated, context-aware data quality control across the entire metric chain, fundamentally ensuring the unified definition, logical consistency, and calculation accuracy of key business metrics across the enterprise.

[0009] On the one hand, this invention provides a method for enterprise data operation multi-source data virtualization access and automated indicator management, including:

[0010] Through the heterogeneous data source virtualization adaptation module, a unified virtualized data access layer is established for multiple business databases, data warehouses, data lakes and external application interface data sources distributed within the enterprise; the virtualized data access layer maps the physical tables, fields and data interfaces of each heterogeneous data source into a set of standardized virtual logical data entities.

[0011] The semantic atomicity parsing and registration module receives indicator definition instructions input by business personnel, parses and decomposes these instructions into a set of semantic atoms. Semantic atoms include atomic entities, atomic behaviors, atomic modifiers, and atomic operators. Atomic entities refer to core objects in the business domain, atomic behaviors refer to actions or events that occur in atomic entities, atomic modifiers are used to limit the attributes, states, or time ranges of atomic entities or atomic behaviors, and atomic operators are preset aggregation, filtering, or transformation functions. All parsed semantic atoms are assigned globally unique identifiers and registered and stored in the atomic knowledge base.

[0012] The indicator dependency dynamic graph construction module is used to construct and maintain a directed acyclic graph of indicator dependencies in the graph database based on the combination and calculation logic of semantic atoms in the indicator definition instructions. The nodes in the graph represent semantic atoms or composite indicators composed of combinations of semantic atoms, and the directed edges in the graph represent the calculation dependencies and data lineage between nodes.

[0013] The data quality rule dynamic generation and binding module defines data quality verification rules and logically binds these rules to specific semantic atoms or combinations of atoms in the atomic knowledge base. The data quality verification rules include data type constraints, value range constraints, non-empty constraints, and cross-atomic logical consistency constraints.

[0014] The virtualized query generation and execution engine receives metric query requests initiated by external systems or users. It first parses the query request, locates the corresponding composite metric node in the dynamic graph of metric dependencies, then traverses the graph backwards along all dependent semantic atomic nodes, collecting data quality verification rules bound to all nodes along the traversal path. Next, it compiles the calculation logic of the composite metric with the collected semantic atomic definitions to generate one or more local query language instructions for heterogeneous data sources, and executes them through the heterogeneous data source virtualization adaptation module. Upon receiving the query results, the virtualized query generation and execution engine automatically verifies the result data using the collected data quality verification rules, and returns the verified data as the final metric value.

[0015] The indicator lifecycle and version control module performs version control and state management on each semantic atom in the atomic knowledge base and the composite indicators composed of them. Any modification to a semantic atom will create a new version. At the same time, the indicator dependency dynamic graph construction module automatically identifies this change and marks all composite indicators that reference the old version atom in the graph, triggering consistency conflict warnings or automatic update processes. State management includes the flow control between draft, active, obsolete and archived states.

[0016] As one embodiment of the present invention, establishing a unified virtualized data access layer through a heterogeneous data source virtualization adaptation module specifically includes:

[0017] Configure and instantiate dedicated connectors for different data source types, including a general database direct connection driver for relational databases, a dedicated application interface adapter for data warehouses, a data parser for structured documents in a data lake, and a representational state transition application interface client for external services.

[0018] The heterogeneous data source virtualization adaptation module has a built-in unified metadata model, which is used to describe data entities, attributes and their relationships in a standardized structure. After connecting to the physical data source, the dedicated connector automatically captures its metadata information and converts it into an instance of the unified metadata model according to the mapping rules predefined by the business logic, generating and persistently storing a mapping table from physical model to logical model.

[0019] When a structural change notification is received from the underlying physical data source, the heterogeneous data source virtualization adaptation module automatically triggers a metadata synchronization task, updates the mapping table, and verifies whether the change affects the defined virtual logical data entities. If there is an impact, the module for constructing the dynamic graph of metric dependencies is notified to perform an impact analysis.

[0020] As one embodiment of the present invention, the indicator semantic atomic parsing and registration module parses and decomposes the indicator definition instruction into semantic atoms, specifically including:

[0021] It provides a domain-specific language for business personnel to describe the computational logic of metrics using a domain-specific language with a predefined grammatical structure; the grammatical structure of the domain-specific language predefines the declaration and usage syntax of atomic entities, atomic behaviors, atomic modifiers and atomic operators.

[0022] The indicator semantic atomicity parsing and registration module has a built-in parser based on a domain-specific language grammar. The parser uses a recursive descent parsing algorithm to parse the input indicator definition instruction string into an abstract syntax tree. The non-leaf nodes of the abstract syntax tree represent computational logic and compositional relationships, while the leaf nodes represent specific semantic atoms.

[0023] Traverse the abstract syntax tree, extract all leaf nodes, and query the atomic knowledge base. If the semantic atom already exists, establish a reference relationship. If it is a new semantic atom, assign it a globally unique identifier, version number, creator information, and timestamp, and store its structured data in the atomic knowledge base. The atomic knowledge base is stored using a document-oriented database. Each semantic atom is represented by an independent document object, whose attributes include atom type, business definition, technical scope, responsible person, and data source information.

[0024] As one embodiment of the present invention, the indicator dependency dynamic graph construction module constructs and maintains a directed acyclic graph of indicator dependencies, specifically including:

[0025] In a graph database, atomic indicator nodes and composite indicator nodes are defined. Atomic indicator nodes directly correspond to semantic atoms in the atomic knowledge base, while composite indicator nodes represent new indicators that are combined by computational logic from one or more atomic indicator nodes or other composite indicator nodes.

[0026] Define a class of "dependent" directed edges. When creating a new composite index, based on its structure in the abstract syntax tree, start from the node representing the composite index and create "dependent" directed edges pointing to all its directly dependent atomic index nodes or composite index nodes.

[0027] The graph construction process is incremental; whenever a new indicator is registered or the definition of an existing indicator is modified, the indicator dependency dynamic graph construction module only needs to create or update the relevant nodes and edges in the graph, without rebuilding the entire graph; when the definition of a semantic atom changes, the indicator dependency dynamic graph construction module creates a node representing the new version of the atom in the graph, and automatically creates edges pointing to the new version of the atom node for all composite indicators that depend on the atom according to the dependency edges in the indicator dependency dynamic graph, while retaining the edges pointing to the old version nodes and marking them as "pending update".

[0028] As one embodiment of the present invention, the process of compiling and generating native query language instructions by the virtualization query generation and execution engine specifically includes:

[0029] The virtualized query generation and execution engine constructs a logical query plan, which represents the complete calculation process of the indicators in a tree structure. Its leaf nodes are the required atomic indicators, and the intermediate nodes are the calculation operations represented by the atomic operators.

[0030] The virtualized query generation and execution engine uses the mapping table from the physical model to the logical model to replace all virtual logical data entities and their attributes in the logical query plan with the table names and field names in the corresponding physical data source, thereby generating a physical query plan.

[0031] The virtualized query generation and execution engine calls the corresponding query language compiler based on the data source type involved in the physical query plan. The compiler converts the general physical query plan into a local query language script for the specific data source. The script can be a structured query language, a multidimensional expression query language, or other proprietary query language.

[0032] During compilation, the virtualized query generation and execution engine also performs query optimizations, including predicate pushdown, join order rearrangement, and materialized view selection, to improve query execution efficiency.

[0033] On the other hand, a multi-source data virtualization access and automated indicator management system for enterprise data operations is provided, which includes:

[0034] The heterogeneous data source virtualization adaptation module is used to establish a unified virtualized data access layer for multiple business databases, data warehouses, data lakes and external application interface data sources distributed within an enterprise, and to map the physical structure of each heterogeneous data source into a set of standardized virtual logical data entities.

[0035] The indicator semantic atomicity parsing and registration module connects with the heterogeneous data source virtualization adaptation module. It is used to receive and parse indicator definition instructions, decompose them into semantic atoms composed of atomic entities, atomic behaviors, atomic modifiers and atomic operators, and register and store the semantic atoms in the atomic knowledge base.

[0036] The indicator dependency dynamic graph construction module, connected to the indicator semantic atomization parsing and registration module, is used to construct and maintain a directed acyclic graph of indicator dependencies in the graph database, representing the computational dependencies and data lineage relationships between indicators, based on the combination and computational logic of semantic atoms.

[0037] The data quality rule dynamic generation and binding module is connected to the indicator semantic atomic parsing and registration module. It is used to define data quality verification rules and logically bind the rules to specific semantic atoms or combinations of atoms in the atomic knowledge base.

[0038] The virtualized query generation and execution engine connects with the heterogeneous data source virtualization adaptation module, the indicator dependency dynamic graph construction module, and the data quality rule dynamic generation and binding module. It is used to receive indicator query requests, parse and locate the corresponding nodes in the graph, traverse the graph in reverse to collect dependent semantic atoms and bound data quality rules, compile and generate local query language instructions for heterogeneous data sources and issue them for execution, and finally use the collected rules to automatically verify the query results and return the final indicator value.

[0039] The indicator lifecycle and version control module is connected to the indicator semantic atomic parsing and registration module and the indicator dependency dynamic graph construction module. It is used to perform version control and status management on all semantic atoms and composite indicators, and automatically transmit the impact and issue early warnings in the graph when the atomic definition changes.

[0040] In one embodiment of the present invention, the system further includes an indicator conflict and consistency detection unit, which is connected to the indicator dependency dynamic graph construction module; the unit is triggered periodically or when the indicator definition changes, and performs the following operations:

[0041] Traverse the dynamic graph of metric dependencies to detect whether there are circular dependencies, and immediately issue an alert if they exist.

[0042] Scan the atomic knowledge base to identify semantic atoms with similar business definitions but different technical terms or computational logic; calculate the semantic similarity of the atomic definition text and combine it with its contextual connection patterns in the graph to discover potential synonymous but heterogeneous indicators, and push them to data governance personnel for adjudication.

[0043] When a composite metric depends on semantic atoms from different virtual logical data entities, the metric conflict and consistency detection unit automatically checks whether these entities have primary key associations or business logic associations at the physical level. If not, the composite metric is marked as "high risk," indicating a data silo computing problem.

[0044] As one embodiment of the present invention, the virtualized query generation and execution engine integrates a distributed caching component. When an indicator is successfully calculated and verified, the hash value of its query request, the compiled local query language instruction, and the calculation result are stored in the distributed caching component, and a cache expiration time calculated based on the data source update frequency is set. When the same indicator query request is received subsequently, the virtualized query generation and execution engine prioritizes obtaining the result from the cache, thereby significantly reducing the query pressure on the underlying data source and shortening the response time.

[0045] In summary, this application includes at least one of the following beneficial technical effects:

[0046] 1. By thoroughly breaking down the definition of indicators into indivisible semantic atoms and managing them in a centralized and versioned manner, the problem of fragmentation and inconsistency of indicator definitions in different systems and tools is fundamentally solved, ensuring the uniformity of understanding and calculation methods of core business concepts across the enterprise.

[0047] 2. By utilizing a dynamic indicator dependency graph constructed from a graph database, the system enables the visualization and automated management of data lineage and computational dependencies among indicators. When any basic semantic atom definition changes, the system can track and transmit its impact to all downstream derived indicators in real time and accurately. This transforms the time-consuming and error-prone manual investigation process in traditional technologies into millisecond-level automated analysis and early warning, greatly improving the efficiency and accuracy of data governance.

[0048] 3. This invention decouples and dynamically binds data quality rules with the core unit of indicator definition, namely semantic atoms, so that data quality verification is no longer an isolated and delayed link, but is embedded in the entire process of indicator calculation; the virtualized query engine automatically aggregates all relevant rules before executing the query, realizing context-aware, end-to-end automated data quality assurance, which can intercept and alert at the first time a data problem occurs, effectively avoiding erroneous decisions caused by data quality problems.

[0049] 4. By using a heterogeneous data source virtualization adaptation layer and a virtualized query generation and execution engine, the complexity and heterogeneity of the underlying data storage are shielded, providing a unified and standardized indicator service interface for upper-layer data consumption. This achieves the separation of indicator definition and physical implementation, so that changes in business logic do not require modifications to the underlying data architecture, and vice versa. This significantly enhances the flexibility, scalability, and maintainability of the entire data operation system. Attached Figure Description

[0050] Figure 1 This is a schematic diagram of the overall technical solution architecture of the enterprise data operation multi-source data virtualization access and automated indicator management method and system proposed in this invention;

[0051] Figure 2 This is a schematic diagram illustrating the core principle framework of the indicator semantic atomization model and dynamic dependency graph construction in this invention;

[0052] Figure 3 This is a flowchart illustrating the data quality rule aggregation and automated verification logic of the virtualized query generation and execution engine in this invention. Detailed Implementation

[0053] To further illustrate the technical means and effects adopted by the present invention in order to achieve the intended purpose, the following detailed description is provided in conjunction with the accompanying drawings and preferred embodiments, based on specific implementation methods of the present invention.

[0054] Reference Figures 1 to 3As shown, this invention provides a method and system for multi-source data virtualization access and automated indicator management in enterprise data operations, aiming to solve the technical problems of fragmented indicator definitions, frequent logical conflicts, static management of dependencies, and rigid coupling of verification rules in enterprise data operations.

[0055] Enterprise data operation multi-source data virtualization access and automated indicator management methods include:

[0056] Step S1: Establish a unified virtualized data access layer through the heterogeneous data source virtualization adaptation module;

[0057] Step S2: Receive and parse the indicator definition instruction through the indicator semantic atomization parsing and registration module, decompose it into a set of semantic atoms and register and store them in the atomic knowledge base;

[0058] Step S3: Using the indicator dependency dynamic graph construction module, construct and maintain a directed acyclic graph of indicator dependencies in the graph database based on the combination logic of semantic atoms;

[0059] Step S4: Define data quality verification rules through the data quality rule dynamic generation and binding module, and logically bind the rules with specific semantic atoms or combinations of atoms;

[0060] Step S5: Receive the indicator query request through the virtualized query generation and execution engine, parse and locate the corresponding node in the map, traverse backwards to collect the dependent semantic atoms and bound data quality rules, compile and generate local query language instructions for heterogeneous data sources and issue them for execution, and finally use the collected rules to automatically verify the query results and return the final indicator value.

[0061] Step S6: Through the indicator lifecycle and version control module, version control and status management are performed on all semantic atoms and composite indicators, and the impact is automatically transmitted and early warning is issued in the graph when the atomic definition changes.

[0062] In step S1, a unified virtualized data access layer is established for multiple business databases, data warehouses, data lakes and external application interface data sources distributed within the enterprise through the heterogeneous data source virtualization adaptation module; the virtualized data access layer maps the physical tables, fields and data interfaces of each heterogeneous data source into a set of standardized virtual logical data entities.

[0063] First, the heterogeneous data source virtualization adaptation module configures and instantiates dedicated connectors for different data source types, including a general database direct connection driver for relational databases, a dedicated application interface adapter for data warehouses, a data parser for structured files in data lakes, and a representational state transition application interface client for external services.

[0064] Each dedicated connector has embedded metadata retrieval logic, which can automatically scan the physical model of the target data source and extract table names, field names, field types, primary and foreign key relationships, and comment information when connecting for the first time or receiving a structural change notification.

[0065] Subsequently, the heterogeneous data source virtualization adaptation module calls the built-in unified metadata model converter to convert the captured heterogeneous metadata into standardized virtual logical data entity descriptions according to the mapping rules predefined by the business logic.

[0066] The unified metadata model adopts a three-layer structure: the first layer is the logical domain, which corresponds to the enterprise business domain such as the customer domain and the order domain; the second layer is the logical entity, which corresponds to the core business object such as the customer and the order; and the third layer is the logical attribute, which corresponds to the specific characteristics of the object such as the customer number and the order amount.

[0067] After the conversion is completed, the heterogeneous data source virtualization adaptation module generates and persists a mapping table from physical model to logical model. This mapping table records the precise correspondence between each physical field and logical attribute, including field aliases, data type conversion rules, and null value handling strategies.

[0068] When the underlying physical data source undergoes structural changes, such as adding fields or modifying field types, the heterogeneous data source virtualization adaptation module automatically triggers a metadata synchronization task to re-fetch the metadata of the changed parts, update the mapping table, and verify whether the change affects the defined virtual logical data entities. If there is an impact, such as the physical field on which a logical attribute depends being deleted, the heterogeneous data source virtualization adaptation module immediately sends an impact analysis request to the indicator dependency dynamic graph construction module to initiate the downstream indicator dependency link re-examination process.

[0069] In step S2, the indicator semantic atomization parsing and registration module receives the indicator definition instructions input by business personnel, and parses and decomposes the indicator definition instructions into a set of semantic atoms. Semantic atoms include atomic entities, atomic behaviors, atomic modifiers, and atomic operators. Atomic entities refer to core objects in the business domain, atomic behaviors refer to actions or events that occur in atomic entities, atomic modifiers are used to limit the attributes, states, or time ranges of atomic entities or atomic behaviors, and atomic operators are preset aggregation, filtering, or transformation functions. All parsed semantic atoms are assigned globally unique identifiers and registered and stored in an atomic knowledge base.

[0070] The metric semantic atomicity parsing and registration module provides a domain-specific language whose grammatical structure strictly defines the declaration syntax of four types of semantic atoms. For example, a complete metric definition instruction can be expressed as: "Calculate the number of active users who have completed payments in the past thirty days", where "user" is the atomic entity, "complete payment" is the atomic action, "in the past thirty days" is the atomic modifier, and "count" is the atomic operator.

[0071] The built-in parser of the indicator semantic atomic parsing and registration module uses a recursive descent parsing algorithm to parse the input natural language instruction string word by word and construct an abstract syntax tree. The root node of the abstract syntax tree is the indicator name, and its child nodes are, in order, operator nodes, behavior nodes, entity nodes and modifier nodes.

[0072] After parsing is complete, the indicator semantic atomic parsing and registration module traverses all leaf nodes of the abstract syntax tree and extracts semantic atoms one by one.

[0073] For each extracted atom, the indicator semantic atomization parsing and registration module first queries the atom knowledge base to determine whether it already exists. If it exists, a reference relationship is directly established, and the current indicator's dependency on that atom is recorded. If it is a new semantic atom, a globally unique identifier is assigned to it. This identifier is composed of the atom type code, business domain code, sequence number, and timestamp hash value to ensure global uniqueness and traceability.

[0074] Meanwhile, the semantic atomicity parsing and registration module assigns an initial version number "one" to each new atom and records the creator information, creation time, business definition text, technical description, responsible person, and data source information. All semantic atoms are stored as independent document objects in an atomic knowledge base composed of a document-oriented database. Each document contains fields such as atom type, identifier, version number, status, definition text, technical implementation logic, associated data source, and reference count.

[0075] Taking the business metric "number of active users in the past 30 days" as an example, the implementation process of this invention is demonstrated:

[0076] Business personnel input the command using a domain-specific language (DSL): "Calculate the number of active users who have completed payments in the past thirty days." Here, "user" is the atomic entity, "complete payment" is the atomic action, "past thirty days" is the atomic modifier, and "count" is the atomic operator.

[0077] The parser employs a recursive descent parsing algorithm to parse instructions into an abstract syntax tree. Leaf nodes extract semantic atoms (such as the atomic entity "user" and the atomic action "complete payment"). The parser queries the atomic knowledge base; if the atom does not exist, a new atom is registered and assigned a globally unique identifier (such as "ENT_USER_001" or "ACT_PAYMENT_001").

[0078] Create a composite metric node “Number of active users in the past 30 days” in the graph database, and create “dependent” directed edges that point to dependent atomic nodes (such as “User”, “Completed payment”).

[0079] Bind data quality rules to the atomic entity "User", such as "User ID NOT null constraint"; bind rules to the atomic behavior "Complete Payment", such as "Payment Amount is Greater Than 0".

[0080] The virtualized query generation and execution engine receives query requests, traverses the graph in reverse to collect dependency atoms and rules, and generates a logical query plan. Using a mapping table from the physical model to the logical model, the logical plan is converted into an SQL query script. After execution, data quality rules are applied to validate the results, such as checking that the user ID is not null.

[0081] Define the rule "If the order status is 'completed', then a corresponding payment record must exist", and automatically verify the consistency of business logic after the query is executed.

[0082] In step S3, the indicator dependency dynamic graph construction module is used to construct and maintain a directed acyclic graph of indicator dependencies in the graph database according to the combination and calculation logic of semantic atoms in the indicator definition instructions. The nodes in the graph represent semantic atoms or composite indicators composed of combinations of semantic atoms, and the directed edges in the graph represent the calculation dependencies and data lineage between nodes.

[0083] The indicator dependency dynamic graph construction module predefines two types of node labels in the graph database: atomic indicator nodes and composite indicator nodes.

[0084] Atomic indicator nodes directly correspond to semantic atoms in the atomic knowledge base, and their attributes include atomic identifier, version number, and status; composite indicator nodes represent new indicators composed of one or more atomic indicator nodes or other composite indicator nodes through computational logic, and their attributes include indicator name, calculation expression, creation time, and status.

[0085] When a new indicator registration request arrives, firstly, the indicator dependency dynamic graph construction module creates a composite indicator node representing the indicator in the graph database; then, the indicator dependency dynamic graph construction module parses the abstract syntax tree of the indicator, identifies all atomic indicator nodes or composite indicator nodes that it directly depends on, and creates "dependent on" directed edges pointing to these dependent nodes starting from the newly created composite indicator node.

[0086] Each edge carries a dependency type attribute, such as "aggregate dependency", "filter dependency", or "transformation dependency".

[0087] The graph construction process is incremental, only dealing with the newly added or changed parts, without the need for a full reconstruction. When the definition of a semantic atom changes, for example, the time range of "active users" is changed from "past seven days" to "past thirty days", the metric lifecycle and version control module will create a new version document for that atom.

[0088] At this point, the indicator dependency dynamic graph construction module detects the version update event in the atomic knowledge base, immediately creates a node representing the new version of the atom in the graph database, and traverses all composite indicator nodes that reference the old version of the atom, creating edges for them pointing to the new version of the atomic node. At the same time, the old edges are marked as "pending update" and the consistency conflict warning process is triggered to notify the relevant responsible persons to confirm whether to switch to the new version.

[0089] In step S4, data quality verification rules are defined through the data quality rule dynamic generation and binding module, and the data quality verification rules are logically bound to specific semantic atoms or combinations of atoms in the atomic knowledge base. The data quality verification rules include data type constraints, value range constraints, non-empty constraints, and cross-atomic logical consistency constraints.

[0090] The data quality rule dynamic generation and binding module provides a rule definition interface, allowing data governance personnel to configure data quality verification rules for each semantic atom. For example, binding "not null constraint" and "uniqueness constraint" to the "customer number" attribute in the atomic entity "customer"; binding a value range constraint of "greater than zero" to the "payment amount" field in the atomic behavior "complete payment"; binding a data type constraint of "time format is ISO8601" to the atomic modifier "within the past thirty days". More complex rules, such as cross-atomic logical consistency constraints, can be defined as: "If the order status is 'completed', then a corresponding payment record must exist".

[0091] All data quality verification rules are stored in the form of structured rule objects, which include rule type, applicable atom identifier, verification logic expression and severity level. The binding relationship between data quality verification rules and semantic atoms is realized through the "binding rule list" field in the atom knowledge base. This field stores the set of rule identifiers applicable to the atom. When a composite indicator depends on multiple atoms, its final set of data quality verification rules is the union of all binding rules of the dependent atoms.

[0092] In step S5, the virtualized query generation and execution engine receives indicator query requests initiated by external systems or users. The engine first parses the query request, locates the corresponding composite indicator node in the dynamic graph of indicator dependencies, then traverses all dependent semantic atom nodes in reverse along the graph, and collects data quality verification rules bound to all nodes along the traversal path. Next, it compiles the calculation logic of the composite indicator with the collected semantic atom definitions to generate one or more local query language instructions for heterogeneous data sources, and executes them through the heterogeneous data source virtualization adaptation module. After receiving the query results, the engine automatically verifies the result data using the collected data quality verification rules and returns the verified data as the final indicator value.

[0093] After receiving a query request, the virtualized query generation and execution engine first searches for the corresponding composite indicator node in the dynamic graph of indicator dependencies. Once found, the virtualized query generation and execution engine starts a depth-first search, traversing backwards along all "dependent" edges until all leaf nodes, i.e. atomic indicator nodes, are reached. During this process, the virtualized query generation and execution engine synchronously collects the data quality rules bound to each atomic node, forming a complete set of rules.

[0094] Subsequently, the virtualized query generation and execution engine constructs a logical query plan, which represents the complete calculation process of the index in a tree structure. Its leaf nodes are the required atomic indices, and the intermediate nodes are the calculation operations represented by the atomic operators, such as summation, counting, and averaging.

[0095] Next, the virtualization query generation and execution engine queries the mapping table from the physical model to the logical model, replaces all virtual logical data entities and their attributes in the logical query plan with the table names and field names in the corresponding physical data source, and generates the physical query plan.

[0096] Then, the virtualized query generation and execution engine calls the corresponding query language compiler based on the data source type involved in the physical query plan. For example, if the data source is a relational database, the structured query language compiler is called; if it is a cube, the multidimensional expression query language compiler is called.

[0097] The compiler translates a general physical query plan into a local query language script for a specific data source. During compilation, the virtualized query generation and execution engine performs query optimizations, including pushing filter conditions down to the data source as far as possible, rearranging the order of multi-table joins to reduce the size of intermediate result sets, and selecting available materialized views to replace the original tables to speed up the query.

[0098] The optimized script is distributed to the corresponding data source for execution through the heterogeneous data source virtualization adaptation module. After execution, the virtualization query generation and execution engine receives the original result set and applies the collected data quality verification rules one by one. For example, it checks whether the values ​​are within the allowed range, whether the time format is correct, and whether there are combinations that violate business logic. If all rules pass the verification, the result is returned as the final indicator value. If any rule fails the verification, the error details are recorded and a verification failure status code is returned, and an alarm notification is triggered at the same time.

[0099] In step S6, the full lifecycle and version control module of indicators performs version control and state management on each semantic atom in the atomic knowledge base and the composite indicators composed of them. Any modification to a semantic atom will create a new version. At the same time, the indicator dependency dynamic graph construction module automatically recognizes this change and marks all composite indicators that reference the old version atom in the graph, triggering a consistency conflict warning or automatic update process. State management includes the flow control between draft state, active state, obsolete state and archived state.

[0100] The indicator lifecycle and version control module maintains a state machine for each semantic atom and composite indicator. When initially created, the state is "draft state," which is only visible to the creator. After approval, the state changes to "active state," which can be referenced by all users. When business rules change, users cannot directly modify the active state atom. Instead, they must create a new draft based on the current version, modify it, and submit it for approval. After approval, the new version becomes active, and the old version changes to "obsolete state." Obsolete state atoms are still retained in the knowledge base, but new references are prohibited.

[0101] After the retention period configured by the administrator, it can be manually converted to "archived state" for historical traceability only. The state transition process is recorded in the operation log, including the operator, time, reason and the state before and after. When an atom changes from active state to obsolete state, the indicator lifecycle and version control module immediately notifies the indicator dependency dynamic graph construction module. The latter marks all composite indicators that depend on the obsolete atom as "risk indicators" in the graph and prompts "the dependency has been obsolete and it is recommended to update" when the user queries.

[0102] Based on the above methods, this invention provides an enterprise data operation multi-source data virtualization access and indicator automated management and control system, including a heterogeneous data source virtualization adaptation module, an indicator semantic atomic parsing and registration module, an indicator dependency dynamic graph construction module, a data quality rule dynamic generation and binding module, a virtualized query generation and execution engine, and an indicator full life cycle and version management module. Each module communicates through a standard application programming interface and shares an atomic knowledge base and graph database.

[0103] The heterogeneous data source virtualization adaptation module is responsible for shielding the differences in the underlying data sources and providing a unified logical view.

[0104] The indicator semantic atomicity parsing and registration module serves as the entry point for indicator definition, ensuring that all indicators are built based on an atomic model.

[0105] The dynamic graph construction module for indicator dependencies maintains the lineage relationships between indicators in real time, supporting impact analysis and tracing.

[0106] The module for dynamically generating and binding data quality rules embeds quality assurance into the indicator definition process.

[0107] The virtualized query generation and execution engine serves as the execution hub, integrating query compilation, distribution, execution, and verification.

[0108] The full lifecycle and version control module for indicators ensures the orderly evolution of indicator assets.

[0109] In addition, it includes an indicator conflict and consistency detection unit, which periodically scans the graph and knowledge base to detect circular dependencies, synonymous heterogeneous atoms, and data silo computing issues, and generates governance recommendation reports.

[0110] The virtualized query generation and execution engine also integrates a distributed caching component to cache the results of frequently queried metrics. The cache key is the hash value of the query request, and the cache value includes the compiled query script and result data. The cache expiration time is dynamically calculated based on the update frequency of the underlying data source to ensure a balance between data timeliness and query performance.

[0111] In summary, this invention constructs a closed-loop automated indicator management system by combining semantic atomization, dependency graphing, rule dynamics, and execution virtualization technologies, fundamentally solving the problem of indicator consistency in enterprise data operations.

[0112] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the present invention can be implemented in other specific forms without departing from the spirit or essential characteristics of the present invention. Therefore, the embodiments should be regarded as exemplary and non-limiting in all respects.

[0113] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment includes only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for enterprise data operation multi-source data virtualization access and index automatic control, characterized in that, Comprise: A unified virtualization data access layer is established by the heterogeneous data source virtualization adaptation module, comprising: Configure and instantiate special connectors for different data source types, including a general database direct connection driver for relational databases, a special application program interface adapter for data warehouses, a data parser for structured files in data lakes, and a representational state transfer application program interface client for external services; The heterogeneous data source virtualization adaptation module has a unified metadata model built in, which is used to describe data entities, attributes and their relationships in a standardized structure; After connecting to the physical data source, the special connector automatically captures its metadata information, and converts it into an instance of the unified metadata model according to the mapping rules defined by the business logic in advance, generating and persistently storing a mapping table from the physical model to the logical model; When receiving a structure change notification of the underlying physical data source, the heterogeneous data source virtualization adaptation module automatically triggers a metadata synchronization task, updates the mapping table, and checks whether the change affects the defined virtual logical data entity, if there is an impact, notify the index dependency relationship dynamic graph construction module for impact analysis. The index semantic atomization analysis and registration module parses and disassembles the index definition instruction into semantic atoms, including: 2.The enterprise data operation multi-source data virtualization access and index automation management and control method of claim 1, wherein, Provide a domain-specific language for business personnel to describe index calculation logic using a domain-specific language with a predetermined grammar structure; The grammar structure of the domain-specific language defines the declaration and use syntax of atomic entities, atomic behaviors, atomic modifiers, and atomic operators in advance; ​ ​ ​ 3.The enterprise data operation multi-source data virtualization access and index automation management and control method of claim 1, wherein, ​ ​ The index semantic atom resolution and registration module is internally provided with a syntax analyzer based on a domain-specific language grammar, which is used to parse the input index definition instruction string into an abstract syntax tree; a non-leaf node of the abstract syntax tree represents a calculation logic and a combination relationship, and a leaf node represents a specific semantic atom; All leaf nodes are extracted by traversing the abstract syntax tree, and the atom knowledge base is queried; if the semantic atom already exists, a reference relationship is established; if it is a new semantic atom, a globally unique identifier, a version number, creator information, and a timestamp are assigned to it, and its structured data is stored in the atom knowledge base; the atom knowledge base is stored by using a document type database, and each semantic atom is represented by an independent document object, and the attributes include atom type, business definition, technical standard, responsible person, and data source information.

4. The enterprise data operation multi-source data virtualization access and index automation management and control method according to claim 3, characterized in that, The index dependency dynamic graph construction module constructs and maintains an index dependency directed acyclic graph, including: In the graph database, define atomic index nodes and composite index nodes; the atomic index node directly corresponds to the semantic atom in the atom knowledge base, and the composite index node represents a new index composed of one or more atomic index nodes or other composite index nodes through calculation logic combination; Define a class of "depends on" directed edges, when a new composite index is created, according to its structure in the abstract syntax tree, starting from the node representing the composite index, create "depends on" directed edges pointing to all directly dependent atomic index nodes or composite index nodes; The construction process of the graph is incremental, whenever a new index is registered, or the definition of an existing index is modified, the index dependency dynamic graph construction module only needs to create or update related nodes and edges in the graph, without the need to rebuild the entire graph; when the definition of a semantic atom is changed, the index dependency dynamic graph construction module creates a node representing the new version of the atom in the graph, and according to the dependency edges in the index dependency dynamic graph, automatically creates edges pointing to the new version of the atom node for all composite indexes that depend on the atom, while retaining the edges pointing to the old version node and marking them as "to be updated".

5. The enterprise data operation multi-source data virtualization access and index automation management and control method according to claim 1, characterized in that, The data quality rule dynamic generation and binding module logically binds data quality check rules to specific semantic atoms or atomic combinations, including: Bind non-empty constraints and uniqueness constraints to the attributes in the atomic entity; Bind value range constraints to the numerical fields in the atomic behavior; Bind data type constraints to the atomic modifiers; Define cross-atomic logical consistency constraints, which are used to check the business logic consistency between multiple semantic atoms; All data quality check rules are stored in the form of structured rule objects, and the association between data quality check rules and semantic atoms is realized through the binding rule list field in the atom knowledge base. 6.The enterprise data operation multi-source data virtualization access and index automation management and control method according to claim 1, characterized in that, The virtualized query generation and execution engine compiles to generate local query language instructions, including: The virtualized query generation and execution engine builds a logical query plan, which represents the complete calculation process of the index in a tree structure, and the leaf nodes are the required atomic indexes, and the intermediate nodes are the calculation operations represented by atomic operators; The virtualized query generation and execution engine replaces all virtual logical data entities and their attributes in the logical query plan with table names and field names in the corresponding physical data source according to the mapping table from the physical model to the logical model, to generate a physical query plan; The virtualized query generation and execution engine calls a corresponding query language compiler according to the type of the data source involved in the physical query plan; the compiler converts the general physical query plan into a local query language script specific to the data source, and the script is a structured query language, a multi-dimensional expression query language, or other proprietary query language; During the compilation process, the virtualized query generation and execution engine also performs query optimization.

7. The enterprise data operation multi-source data virtualization access and index automation management and control method according to claim 1, characterized in that, The module also performs version control and state management on all semantic atoms and composite indicators through the indicator full life cycle and version management module, including: A state machine is maintained for each semantic atom and composite indicator, and the states include a draft state, an active state, a discarded state, and an archived state; Any modification operation on a semantic atom creates a new version, and the state of the old version changes to the discarded state; When a semantic atom changes from the active state to the discarded state, the module notifies the indicator dependency dynamic graph construction module to mark all composite indicators that depend on the discarded semantic atom as risk indicators in the graph, and prompts the dependent items when the user queries that the dependent items have been discarded.

8. The enterprise data operation multi-source data virtualization access and index automation management and control method according to claim 7, characterized in that, After receiving the query result, the virtualized query generation and execution engine uses the collected data quality verification rules to automatically verify the result data, including: checking whether the numerical value is within the allowed range; verifying whether the time format meets the preset standard; checking whether there is a combination that violates the business logic; If all data quality verification rules pass, the result is returned as the final indicator value; if any rule fails, the error details are recorded and a verification failure status code is returned, and an alarm notification is triggered.

9. An enterprise data operation multi-source data virtualization access and index automation management and control system, characterized in that, The module also performs version control and state management on all semantic atoms and composite indicators through the indicator full life cycle and version management module, including: The heterogeneous data source virtualization adaptation module is used to establish a unified virtualization data access layer for multiple business databases, data warehouses, data lakes, and external application program interface data sources distributed within an enterprise; The indicator semantic atomization analysis and registration module is connected with the heterogeneous data source virtualization adaptation module, and is used to receive and analyze indicator definition instructions, decompose them into semantic atoms composed of atomic entities, atomic behaviors, atomic modifiers, and atomic operators, and register and store the semantic atoms in the atomic knowledge base; The indicator dependency dynamic graph construction module is connected with the indicator semantic atomization analysis and registration module, and is used to construct and maintain an indicator dependency directed acyclic graph representing the calculation dependency and data lineage relationship between indicators in a graph database according to the combination and calculation logic of the semantic atoms; The data quality rule dynamic generation and binding module is connected with the indicator semantic atomization analysis and registration module, and is used to define data quality verification rules and logically bind the data quality verification rules with specific semantic atoms or atomic combinations in the atomic knowledge base. The virtualization query generation and execution engine is connected with the heterogeneous data source virtualization adaptation module, the index dependency dynamic graph construction module and the data quality rule dynamic generation and binding module, and is used for receiving an index query request, analyzing and locating to a corresponding node in the graph, traversing the graph in reverse to collect dependent semantic atoms and bound data quality rules, compiling to generate local query language instructions for the heterogeneous data source and executing, and finally automatically checking the query result by using the collected rules and returning a final index value; The index full life cycle and version management module is connected with the index semantic atomization analysis and registration module and the index dependency dynamic graph construction module, and is used for performing version control and state management on all semantic atoms and composite indexes, and when the atom definition is changed, automatically conducting influence conduction and early warning in the graph.

Images