A dynamic management method of asset data

Abstract

The application relates to the technical field of asset data management, and discloses a dynamic management method for asset data, which comprises the following steps: acquiring data flow events of each data node of a real-time asset graph and acquiring implicit boundary changes; generating an asset update instruction of a target data object; updating a state attribute of the target data object, and starting from the target data object to traverse downstream dependent data objects and execute cascaded state updating; when detecting a currently-traversed downstream dependent data, judging that a circular dependency occurs and skipping the downstream dependent data object; single-time cascaded state updating, triggering quantity fusing when the number of downstream nodes exceeds a quantity threshold; stopping automatic downward traversal when the traversal depth exceeds a depth limit; concurrent cascaded state updating; and triggering stubborn cycle alarm when the number of times that a same circular path is detected within a preset time exceeds a preset number of times. The application improves the stability and reliability of an asset state updating process.

Description

Technical Field

[0001] This invention relates to the field of asset data management technology, and more specifically, to a dynamic management method for asset data. Background Technology

[0002] With the development of application scenarios such as data asset entry, data governance, and data element circulation, enterprises need to continuously manage their data assets. Data assets typically include structured database tables, unstructured files, API (Application Programming Interface) data streams, data processing tasks, and business reports. To ensure the accuracy of data asset catalogs, asset maps, and asset status information, it is necessary to update the corresponding asset status promptly when data assets change and synchronously transmit the status changes to related dependent assets.

[0003] Existing data asset management systems primarily employ two methods for asset status maintenance. One method is periodic scanning, which involves periodically scanning data warehouse metadata, asset catalogs, or lineage relationships using scheduled tasks and updating asset status based on the scan results. This method relies on a fixed scanning cycle, and changes in asset status occurring between scans cannot be detected in a timely manner, resulting in a lag in asset status updates. The other method is event-triggered update technology, which monitors explicit structural changes such as database DDL (Data Definition Language) operations and metadata change events, triggering asset status updates when changes are detected. While this method improves update real-time performance, it typically only focuses on explicit changes such as table structure adjustments and object additions / deletions, lacking effective detection capabilities for implicit status changes affecting asset availability and integrity, such as permission revocation, data processing job failures, and business system status notifications. This can easily lead to inconsistencies between the asset status in the asset graph and the actual operational status. However, asset dependencies in real-world business environments are often complex, with some assets potentially forming circular dependencies. When using the traditional recursive propagation method for state updates, problems such as repeated traversal, infinite recursion, and loss of control of the update link can easily occur, thus affecting the stability and reliability of the asset state update process.

[0004] Therefore, it is necessary to design a dynamic management method for asset data to solve the problems existing in the current technology. Summary of the Invention

[0005] In view of this, the present invention proposes a dynamic management method for asset data, which aims to solve the problems of repeated traversal, infinite recursion, and uncontrolled update links in the existing technology, thereby affecting the stability and reliability of the asset status update process.

[0006] This invention proposes a dynamic management method for asset data, comprising:

[0007] Construct a real-time asset graph; acquire data stream events for each data node of the real-time asset graph, and obtain implicit boundary changes based on the data stream events;

[0008] Based on the data stream events and implicit boundary changes, an asset update instruction for the target data object is generated;

[0009] According to the asset update instruction, update the state attributes of the target data object, and traverse the downstream dependent data objects starting from the target data object to perform cascading state updates;

[0010] During the traversal, a set of visited nodes is maintained. When a downstream dependent data object currently being traversed is detected to exist in the set of visited nodes, a circular dependency is determined and the downstream dependent data object is skipped.

[0011] During the execution of the cascading state update, for a single cascading state update, if the number of downstream nodes exceeds the number threshold, a circuit breaker is triggered; if the traversal depth exceeds the depth limit, automatic downward traversal is stopped; for concurrent cascading state updates, if the number of times the same loop path is detected exceeds the preset number within a preset time, a stubborn loop alarm is triggered.

[0012] Furthermore, the real-time asset map includes:

[0013] The system uses a distributed graph database to define node labels to distinguish data object types and employs directed edges to record lineage and computational dependencies between nodes. It receives changed data through a dedicated synchronization interface and submits it in batches according to a preset time window. It receives node change events through a distributed message queue and incrementally generates and updates the real-time asset graph.

[0014] Furthermore, when acquiring the data stream events of each node in the real-time asset graph, the following are included:

[0015] The real-time asset graph collects data stream events from each data node using probes, including a binlog parser, an audit log parser, and a message queue subscriber. The data stream events include metadata change events, data access behavior events, and business status events. The data stream events are then converted into standardized change vectors.

[0016] Furthermore, when obtaining implicit boundary changes, the following are included:

[0017] Based on the permission revocation event in the metadata change event, by monitoring the change log of the database authorization table and comparing the permission bits before and after the change, when it is determined that the permission has changed from authorized to unauthorized, the implicit boundary change of data access failure is identified.

[0018] Based on the data processing job failure event in the data access behavior event, the job failure reason code is extracted and classified by the failure classifier. When it is determined to be a permanent failure, the implicit boundary change of the bloodline break is determined.

[0019] Based on the aforementioned business status events, asset status change notifications issued by the business system are obtained by subscribing to message queues or adapting gateways to determine the implicit boundary changes that have occurred in asset availability or integrity.

[0020] Furthermore, when generating asset update instructions for the target data object, the following are included:

[0021] The standardized change vector is matched with the rules in the business rule base to determine the expected state of the target data object; based on the expected state, an asset update instruction for the target data object is generated.

[0022] Furthermore, when generating asset update instructions for the target data object, the following are included:

[0023] Before generating the asset update instruction, the current state of the target data object is queried to determine whether migration from the current state to the expected state is permitted; if the check shows permission is granted, the asset update instruction for the target data object is generated; if the check shows permission is denied, the instruction generation is intercepted and an alarm is triggered.

[0024] Furthermore, when traversing downstream dependent data objects starting from the target data object and performing cascading state updates, the process includes:

[0025] For downstream dependent data objects that are not skipped, recursively check whether their state has become invalid due to changes in the state of upstream data objects.

[0026] Furthermore, when recursively checking whether its state has become invalid due to a state change of the upstream data object, it also includes:

[0027] For downstream dependent data objects determined to be invalid, if they are critical business objects, a state reassessment event is generated and pushed to a message queue for asynchronous processing; if they are non-critical business objects, a reassessment task is generated and stored in a cache queue for batch state reassessment.

[0028] Furthermore, after determining that a circular dependency exists and skipping downstream dependent data objects, the process includes:

[0029] Add the skipped downstream dependent data objects and all downstream nodes to the delay re-evaluation queue; re-execute the cascading state update starting from the skipped downstream dependent data objects until the downstream dependent data objects are no longer in the loop path or the maximum number of retries is reached.

[0030] Furthermore, for concurrent cascading state updates, this includes:

[0031] Limit the maximum number of cascading state update processes that can be executed in parallel at the same time; limit the same target data object from being repeatedly updated by multiple concurrent processes.

[0032] Compared with existing technologies, the advantages of this invention are as follows: It simultaneously monitors metadata changes, data access behaviors, and business status events, clarifying their technical significance, achieving real-time capture of asset changes across all dimensions, eliminating the coverage blind spots of traditional technologies, and ensuring high synchronization between the asset graph and the actual state. During the traversal process, it maintains a "set of visited nodes," skipping nodes when circular dependencies are detected to avoid infinite recursion. This enables safe and stable cascading state updates in real, complex graphs (including circular dependencies), preventing system crashes and expanding the applicability of the technical solution. It integrates a triple optimization mechanism of priority degradation / sleep, cold start preheating, and clustering batch matching, avoiding inaccurate rule priority allocation during cold starts; automatically eliminating low-frequency rules during long-term operation, saving computing resources; and reducing the amount of matching computation through clustering batch matching. Attached Figure Description

[0033] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:

[0034] Fig. 1 A flowchart illustrating the dynamic asset data management method provided in this embodiment of the invention;

[0035] Fig. 2 A flowchart of cascading state update provided for an embodiment of the present invention. Detailed Implementation

[0036] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specified, embodiments and features in the embodiments of the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0037] In some embodiments of this application, see Figs. 1-2 As shown, a dynamic management method for asset data is proposed, including:

[0038] Construct a real-time asset graph; acquire data stream events for each data node in the real-time asset graph, and obtain implicit boundary changes based on the data stream events;

[0039] Based on data stream events and implicit boundary changes, generate asset update instructions for the target data object;

[0040] According to the asset update instruction, update the state attributes of the target data object, and traverse the downstream dependent data objects starting from the target data object to perform cascading state updates;

[0041] During the traversal, maintain a set of visited nodes. When it is detected that the downstream dependent data object being traversed exists in the set of visited nodes, a circular dependency is determined and the downstream dependent data object is skipped.

[0042] During the execution of cascading state updates, for a single cascading state update, if the number of downstream nodes exceeds the number threshold, a circuit breaker is triggered; if the traversal depth exceeds the depth limit, automatic downward traversal is stopped; for concurrent cascading state updates, if the number of times the same loop path is detected exceeds the preset number within a preset time, a stubborn loop alarm is triggered.

[0043] The asset boundary refers to the effective management scope of a data asset, encompassing a logical space comprised of attributes such as storage location, access permissions, and business affiliation. Data assets are collections of digitized information with potential or actual economic value, including but not limited to structured database tables, unstructured files, API interface data streams, ETL (Extract, Transform, Load) tasks, and business reports. Implicit boundary changes are state changes that do not manifest through explicit structural alterations (such as CREATE / ALTER statements) but actually affect asset availability, integrity, or lineage. Examples include data access failures due to permission revocation, lineage breaks due to ETL job failures, and asset state change notifications issued by business systems. Standardized change vectors convert raw events from different sources and formats (such as database logs, audit logs, and business messages) into JSON format data with a fixed field structure, facilitating unified processing by the subsequent rule engine. Cascading state updates automatically trigger the reassessment and updating of the state of downstream dependent assets when the state of a data asset changes, based on lineage or computational dependencies between assets. Circular dependency occurs when two or more assets form a closed-loop dependency relationship in an asset dependency graph. For example, asset A depends on asset B, and asset B directly or indirectly depends on asset A. Traditional cascading update methods are prone to infinite recursion or system crashes in this scenario.

[0044] Specifically, in this embodiment, a real-time asset graph based on a graph database is first constructed to represent the dependencies and affiliations between different data objects. Nodes in the graph represent different data objects, including but not limited to database tables, API interfaces, ETL tasks, business documents, and business reports; edges in the graph represent the relationships between nodes, including lineage dependencies, computational dependencies, and affiliations.

[0045] Preferably, the real-time asset graph is constructed using a high-throughput distributed graph database, such as Neo4jCluster, JanusGraph, or other graph storage systems with graph traversal and incremental update capabilities. Each node distinguishes different types of data objects through node labels and stores the basic information of the objects in the form of attribute key-value pairs. The basic information includes at least the creation time, department, status code, version number, and timestamp.

[0046] Among them, lineage dependencies are used to represent the path of data flowing from source objects to derived objects, while computational dependencies are used to represent the relationships between tasks that read, process, produce, or invoke data. By modeling these relationships with directed edges, the graph can accurately reflect the actual dependency structure between data objects, providing a foundation for subsequent state propagation and cascading updates.

[0047] Furthermore, the real-time asset graph is maintained using an incremental update mechanism. During graph updates, node change data is received through a dedicated synchronization interface and submitted in batches according to a preset time window. This ensures real-time performance while reducing the performance overhead of frequent writes. Preferably, the time window is set to 50 milliseconds to 200 milliseconds. Simultaneously, node change events can be received through a distributed message queue to improve event throughput in high-concurrency scenarios, ensuring the real-time asset graph continuously reflects the current asset status.

[0048] It should be noted that existing graph databases are typically used only to store static dependencies between assets, or to perform simple single-step queries, such as finding the direct downstream nodes of a given node. When performing cascading state updates, existing methods often assume that asset dependencies are directed acyclic graphs and directly propagate state changes using a tree-like recursive approach. This lacks a real-time detection and avoidance mechanism for circular dependencies, which can easily lead to infinite recursion in complex dependency scenarios.

[0049] In contrast, this invention leverages the efficient breadth-first search capability of graph databases to dynamically maintain a set of visited nodes at the application layer. When traversing downstream dependent nodes starting from the target node, for each visited node, it is determined whether it already exists in the set of visited nodes: if it already exists, a circular dependency is identified, the node is immediately skipped, and the corresponding log is recorded to avoid infinite recursion; if it does not exist, the node is added to the set of visited nodes, and traversal continues. Through this method, this invention can achieve secure and stable cascading updates in complex graph environments.

[0050] Compared to using recursive CTE (Common Table Expression) queries in relational databases for loop detection, this invention leverages the graph traversal advantages of graph databases. In complex graphs with a large number of nodes and deep dependency levels, it reduces traversal latency, making loop dependency detection and cascading state updates more suitable for real-world business scenarios. For example, in a scenario where a listed company is implementing data asset table entry, its core data assets include a "Customer Information Table," a "Sales Order Details Table," a "Product Dimension Table," and a "Revenue Summary View." The "Revenue Summary View" depends on the "Sales Order Details Table" and the "Product Dimension Table," and the "Sales Order Details Table" in turn depends on the "Customer Information Table," used to associate customer geographic information and partition revenue.

[0051] In the graph, all the above data objects exist in the form of nodes, and directed edges represent data flow or computational dependencies between nodes. When the industry classification of a customer in the "Customer Information Table" changes, it may trigger a change in the partition classification in the "Sales Order Details Table," which in turn affects the statistical results of the "Revenue Summary View," and ultimately affects the measurement amount of data assets in the financial statements.

[0052] Specifically, lightweight probes are deployed on the host machine of each data node or adjacent collection nodes to monitor and collect data stream events in real time. The collected raw events are then uniformly converted into standardized change vectors for subsequent processing by the rule engine. Data stream events include at least metadata change events, data access behavior events, and business status events. By uniformly collecting and standardizing these three types of events, this invention can cover implicit boundary changes that are difficult to perceive with traditional technologies, including but not limited to data access failures caused by permission revocation, lineage breaks caused by ETL job failures, and asset status change notifications issued by business systems, thereby achieving real-time perception of asset availability, integrity, and effectiveness.

[0053] Metadata change events are used to characterize changes in metadata attributes such as data object structure, permissions, and ownership. These events can be obtained by parsing database transaction logs, specifically using change data capture techniques, such as Debezium to capture MySQL binary logs or PostgreSQL write-ahead logs, and mapping log fields to corresponding metadata attributes according to preset parsing rules.

[0054] In one implementation, the parsing rules can be configured as a YAML file to define the mapping relationship between log fields and metadata attributes. For example, ALTER TABLE operations can be mapped to table structure change events, and GRANT or REVOKE related operations can be mapped to permission change events. Permission changes are a key source for identifying implicit boundary changes in this invention. For instance, when data governance personnel revoke the finance department's query permissions for a specified field in a data table, although the data table structure remains unchanged, its access boundary has changed. This invention can identify this permission revocation event as a valid implicit boundary change and generate a corresponding standardized change vector for subsequent state reassessment.

[0055] Data access behavior events are used to characterize the access status of data objects in the computing engine or business processing. These events can be obtained by parsing the audit logs of computing engines such as Spark and Flink. The log content includes at least the accessing user, object identifier, operation type, timestamp, and error code.

[0056] In one implementation, audit logs in a distributed environment can be collected and aggregated using log collection tools such as Fluentd to reduce duplicate events and improve collection efficiency. For example, an ETL task periodically reads data from an upstream table to update downstream data assets. If the task fails to read data consecutively due to changes in upstream field constraints, the corresponding failure records in the audit log will be identified as data access anomalies by a probe and further converted into standardized change vectors to trigger a reassessment of the downstream asset lineage status. Data access behavior events include not only successful access records but also access failures, timeouts, and permission denials, which reflect changes in the data link status. In this way, the present invention can identify processing link failures that traditional technologies that only focus on structural changes cannot detect.

[0057] Among them, business status events are used to characterize changes in asset status, job status, or quality alarm information issued by the business system. These events can be obtained by subscribing to a message queue, or by receiving raw status notifications through a business system adaptation gateway and then uniformly converting them into a standard format.

[0058] In one implementation, business status events can be encapsulated using the CloudEvents specification to improve cross-system compatibility. For example, when an upstream "sales order ETL job" fails due to an abnormal data format in the source system, the business system can publish a business status event of type "job failed." After the probe captures this event, it can convert it into a standardized change vector and trigger a status reassessment of downstream data assets such as the "revenue summary view," temporarily changing its status from "available" to "pending refresh" or "untrusted."

[0059] This also includes the transformation of standardized change vectors, which uniformly converts raw events collected from different sources and in different formats into standardized change vectors. The standardized change vectors preferably use JSON (JavaScript Object Notation) format, whose fields include at least a unique event identifier, event type, event source, event timestamp, target object information, and change details. Specifically, the event type distinguishes between metadata change events, data access behavior events, and business status events; the event source identifies the specific data source from which the event originates, such as MySQL binary logs, Spark audit logs, or message queue topics; the target object information identifies the affected data object and its type; and the change details describe the changed fields, old values, and new values. Through this unified structure, events from different sources can all be processed uniformly by the subsequent rule engine, thereby reducing the complexity of multi-source heterogeneous access.

[0060] In a preferred embodiment, the probe is written in Golang, resulting in a small program size. It can be deployed in a containerized manner on the host machine where the data node resides, or run as a system service. The probe performs only read-only data collection on the target database or computing engine, requiring neither plugin installation nor service restart.

[0061] To improve adaptability in complex environments, a degradation mechanism for probe operation is also provided: for operating systems that do not support eBPF (extended Berkeley Packet Filter), it can automatically switch to ETW (Event Tracing for Windows) based acquisition mode, or achieve equivalent event capture by parsing the Windows Event Log and performance counters; for legacy systems that cannot directly deploy probes, a bypass acquisition adapter can be used to obtain change events by periodically polling the system table, parsing text log files, or calling external interfaces, and uniformly convert them into standardized change vectors.

[0062] Furthermore, to prevent event loss due to network outages or message queue congestion, the probe can be configured with a circular buffer to cache recent events and asynchronously persist them to the local disk. The probe can also periodically send heartbeat messages carrying the sequence number of the largest sent event, enabling the central service to identify missing intervals and initiate retransmission requests. After network recovery, the probe retransmits the events corresponding to the missing intervals from the local cache or disk logs based on the retransmission requests, thus ensuring the integrity of event transmission.

[0063] To reduce false positive and false negative rates, this invention sets up refined judgment logic for three types of key events. For permission revocation events, the probe monitors the change logs of the database system's authorization table, compares the permission status before and after the change, and only determines a valid permission revocation event when the permission changes from "authorized" to "unauthorized" and the change operation type is REVOKE or DROP USER; for normal failures such as temporary permission expiration, only logs are recorded without triggering alarms. For ETL job failure events, the probe extracts the job failure reason code by parsing the computing engine audit logs and classifies the failure into temporary failure, permanent failure, and business logic failure using a built-in failure classifier. For temporary failures, the probe can wait for a retry before making a judgment; for permanent failures, a change vector is directly generated and subsequent status processing is triggered, thereby avoiding false judgments caused by short-term fluctuations. For business status events, when the business system supports a unified event protocol, the probe can directly subscribe to the message queue; when the business system does not support this protocol, the original status notification can be converted into a standard event format through a lightweight adaptation gateway before processing, thereby reducing the cost of business system transformation.

[0064] Through the above methods, this invention can uniformly collect and standardize the processing of metadata change events, data access behavior events, and business status events, achieving synchronous perception of the three boundary dimensions of data assets: "permissions—processing—production." Specifically, metadata change events cover changes at the structural and permission levels, data access behavior events cover changes at the processing link and access status levels, and business status events cover changes at the production process and business notification levels. These three types of events together constitute a complete perception system for implicit boundary changes.

[0065] Specifically, the acquired standardized change vectors are processed according to rules to determine whether the target data object needs to undergo a state change. When the state change is legal, an asset update instruction is generated, and when the state change is illegal, an alarm is triggered.

[0066] A two-layer verification mechanism is adopted. First, the standardized change vector is matched with the rules in the business rule base by the adaptive rule engine to determine the expected state of the target data object.

[0067] In one implementation, the business rule base is written in a rule language, such as Drools, and stored in a distributed configuration center to support online editing, version management, and dynamic deployment. The rule engine can use Drools version 8.0 and achieve efficient matching of rule patterns through the Rete algorithm.

[0068] When performing rule matching, the edit distance algorithm can be used to calculate the similarity between the change attributes in the standardized change vector and the rule conditions. For text-type fields, the edit distance between the change attribute string and the rule condition string is calculated, and then the similarity is calculated based on the relationship between the edit distance and the longest string length. For numeric fields, normalized Euclidean distance is used to calculate similarity. For date-type fields, the reciprocal of the time difference is used as the similarity measure. The matching threshold can be configured through the web console, with a preferred default value of 80%, and a weighted scoring method is used, where keyword matching, numeric range matching, and timestamp proximity each have different weights.

[0069] To improve the operating efficiency of the rule engine in large-scale rule bases and high-throughput scenarios, this invention further sets up two types of optimization mechanisms.

[0070] First, a cold start warm-up mechanism. Upon startup, historical standardized change vector logs for a preset duration, such as historical data from the past 7 days, are loaded from persistent storage, and an untriggered counter is associated with each rule. This counter records the number of times a rule is not triggered during continuous event processing. During the warm-up phase, the trigger count of each rule in the historical log is counted, and the untriggered counter is initialized to the total number of events within the historical window minus the trigger count of that rule. Subsequently, an initial priority is set based on the historical trigger frequency of the rules, with rules having higher trigger frequencies having higher initial priority. Preferably, the priority is divided into six levels, with level one being the highest and level six the lowest. This mechanism avoids the problem of all rules being in the same default state during a cold start, making it difficult to distinguish between high-frequency and low-frequency rules, thus ensuring the rule engine maintains high processing efficiency from the very beginning.

[0071] Second, clustering batch matching optimization. Received standardized change vectors are clustered online based on the hash value of attribute fields or edit distance similarity. Specifically, for each arriving change vector, its similarity to the currently active cluster center is calculated; if the similarity exceeds a preset threshold, the vector is assigned to the current cluster; otherwise, a new cluster center is created based on the vector. Change vectors within the same cluster share the same intermediate rule matching results. That is, a complete rule matching is first performed on the cluster center, the matching result is cached in memory, and then other vectors within the cluster directly reuse this cached result to generate corresponding status instructions in batches. In this way, the computational cost of rule matching can be reduced from the order of magnitude of "number of change vectors multiplied by the number of rules" to the order of magnitude of "number of clusters multiplied by the number of rules," thereby significantly improving throughput.

[0072] When the match between a change vector and a rule in the rule base exceeds a preset threshold, the rule engine generates a status instruction containing the expected asset status. The status instruction preferably uses JSON format, and its fields include at least a unique instruction identifier, a target object identifier, a set of expected status key-value pairs, and the instruction's validity period.

[0073] Furthermore, rule priorities can be dynamically adjusted. Each rule is associated with an untriggered counter. After processing each standardized change vector, the counter for untriggered rules is incremented by one. When the counter exceeds a preset threshold, the rule's priority is automatically downgraded. If a rule remains untriggered for an extended period, it enters a dormant state and no longer participates in matching, only being periodically woken up for checks. When a rule is retriggered, its priority is restored and the counter is reset. This mechanism further improves the rule engine's operational efficiency and reduces the overhead of invalid matching.

[0074] After the rules engine determines the expected state of the target data object, it queries the historical state sequence of the target data object in the current asset graph based on the state instruction, and verifies whether the migration from the current state to the expected state is legal.

[0075] Specifically, the dynamic graph layer can adopt a two-tier storage strategy of "memory plus disk": the state of hot objects is cached in the Redis key-value store system, and the cache validity period can be set to 1 to 2 hours; the full state is stored in the graph database. When performing a state query, the Redis cache is checked first. If it does not exist in the cache, it is loaded from the graph database and the cache is updated. The query result includes at least the state value and version number.

[0076] The system checks whether a transition from the current state to the expected state is permitted based on a predefined set of state transition constraint rules. State transition rules can be defined using a finite state machine model, and the state transition matrix can be defined through a configuration file in XML or JSON Schema format. To accommodate asset types of varying sizes and complexities, a hierarchical definition strategy is preferred for state transition rules.

[0077] For the basic state machine, no more than 15 core states can be defined for each asset type. The transition matrix can be edited through a visual web tool and an XML configuration file can be automatically generated. The configuration file is also validated to check for unreachable states or illegal loops.

[0078] For extended states, if users need to define more than 15 states, or need to configure conditional migration rules, such as allowing migration from "Published" to "Archived" only when the user role is auditor, then it is supported to write custom migration rules using Groovy scripts and dynamically load the scripts into the rule engine. At the same time, a sandbox mechanism is used to restrict high-risk operations of the scripts, such as prohibiting file system access and network calls.

[0079] In addition, an offline conflict detection tool can be provided to periodically scan all migration rules to check for the existence of circular migration paths, such as A→B→C→A, or conflict definitions where the same event in the same state is mapped to multiple different target states, and generate conflict reports for administrators to correct.

[0080] The state transition matrix explicitly specifies which target states are allowed and prohibited from being transitioned to from each current state. Taking the data asset entry scenario as an example, the state of a data asset can include "Draft," "Reviewed," "Published," "Expired," and "Pending Refresh." Transitioning directly from the "Published" state to the "Draft" state is not allowed, but transitioning to the "Expired" or "Pending Refresh" states is permitted. Transitioning back to the "Published" state from the "Pending Refresh" state is allowed after the refresh is completed. The configuration also supports using the asterisk wildcard to represent any state. To reduce runtime parsing overhead, the verification logic can be implemented through pre-compiled rule classes.

[0081] When the state transition verification passes, i.e. when the transition from the current state to the expected state is explicitly allowed in the transition matrix, an asset update instruction is generated; when the state transition verification fails, an alarm is triggered, the current state, expected state, occurrence time and rule identifier that triggered the operation are recorded, and alarm information is pushed through multiple channels such as email, SMS, and WeChat Work. Different notification frequencies can be set according to the alarm level, with P0 being the highest level.

[0082] Finally, the generated asset update instruction is sent to the next step; if the state migration is deemed invalid and an alarm is triggered, the alarm handling process is initiated, where the administrator or subsequent processes will manually confirm, correct the rules, or repair the state.

[0083] Specifically, based on the received asset update instructions, the state attributes of the target data object are updated and a corresponding historical version snapshot is generated. At the same time, a cascading state update process is triggered based on the dependencies between data objects to achieve consistent maintenance of data assets in a complex dependency network.

[0084] In one implementation, state updates are implemented using an optimistic locking mechanism. Specifically, each data object includes at least a version number field and a timestamp field. During an update, the current version number is first read and compared with the version number stored in the database. If the version numbers match, a state update operation is performed, and the version number is incremented. If the version numbers do not match, a concurrency conflict is identified, and the update is retried or an error message is returned to the upper layer. This mechanism effectively ensures data consistency in high-concurrency scenarios.

[0085] After the state update is complete, a historical version snapshot is generated for the target data object. The historical version snapshot uses an incremental storage method, recording only the changed attribute fields and their values ​​before and after the change, and is associated with a globally incrementing version number.

[0086] In one implementation, snapshot data is stored in a distributed file system according to preset path rules, such as using a directory structure like "object identifier / year / month / version number.json". Simultaneously, a complete version chain is preserved to support restoring the complete state of data objects at any historical version number at a given point in time.

[0087] This mechanism enables the accurate restoration of the data asset status at any point in time during data asset entry or auditing scenarios, thereby meeting compliance auditing and traceability requirements.

[0088] After updating the state of the target data object, a cascading state update process is initiated based on the dependencies maintained in the graph database. This process is used to propagate changes in the state of upstream assets to their downstream dependent assets and to perform failure assessments and state re-evaluations on the downstream assets.

[0089] Downstream dependent data objects are identified by starting from the target data object and using a breadth-first search algorithm to traverse all its direct and indirect downstream dependent data objects based on the lineage and dependency relationships stored in the graph database.

[0090] In one implementation, to avoid the traversal range from expanding indefinitely, a maximum search depth limit is set, preferably 5 layers, with a maximum configurable limit of 20 layers. The graph database edge index structure keeps the single-hop query complexity at a constant level, thereby ensuring traversal efficiency.

[0091] Circular dependency detection and avoidance involves maintaining a set of visited nodes during traversal, preferably using a hash table structure. When visiting each downstream node, it first checks if the node already exists in the visited set: if it does, a circular dependency is identified, the node and its subsequent paths are immediately skipped, and a circular dependency log is recorded; if it doesn't exist, the node is added to the visited set, and traversal of its dependent nodes continues. To further enhance system stability, after detecting a circular dependency, the relevant nodes can be added to a delayed re-evaluation queue, where a background thread periodically (e.g., every 5 minutes) re-triggers the evaluation process, with a maximum retry count (e.g., 3 times). When the same circular path is frequently triggered within a preset time window (e.g., more than 10 times in 1 hour), it is identified as a stubborn circular dependency, triggering a high-priority alarm and suspending the automatic processing of that path. This mechanism avoids the problems of infinite recursion, stack overflow, or system crashes that occur with traditional cascading updates when circular dependencies exist.

[0092] The failure state recursive determination is as follows: for downstream dependent data objects that have not been skipped, recursively determine whether their state has failed due to the state change of the upstream target data object.

[0093] In one implementation, failure determination rules are configured using a scripting language, such as Groovy scripts, to support flexible business logic definitions. Failure determination conditions include, but are not limited to: when the data quality score of an upstream data asset decreases by more than a preset threshold, the downstream asset is determined to be failed; when the permission status of an upstream data asset changes from "authorized" to "revoked," all its directly or indirectly dependent assets are determined to be failed. To reduce redundant calculation overhead, failure determination results can be cached in memory, with a cache validity period set to 5 to 10 minutes.

[0094] The downstream status reassessment triggering mechanism employs differentiated processing strategies based on business importance for downstream dependent data objects deemed invalid: For critical business objects (e.g., data assets directly impacting financial statements), standardized change vectors are immediately generated and pushed to a message queue (e.g., Kafka) for asynchronous processing by the rule engine; for non-critical business objects, they are added to a reassessment task queue (e.g., a Redis queue), with scheduled tasks triggering the status reassessment process in batches. To prevent a single upstream node change from causing a "snowball effect" of downstream node updates, circuit breaking and rate limiting mechanisms are implemented at the cascading update entry point, including but not limited to: quantity circuit breaking, which refers to a control mechanism where, during the cascading status update process, when the number of downstream nodes to be updated exceeds a preset threshold, the system suspends the current cascading update process and transfers the remaining nodes to a delayed processing queue or splits them into multiple sub-tasks for execution. When the number of nodes involved in a single cascading exceeds a preset threshold (e.g., 1000), a circuit breaker is triggered and updates are paused; Depth limit: Nodes exceeding the maximum search depth are no longer automatically updated and are converted to manual review tasks; Concurrency control: The number of cascading update processes executed concurrently at the same time is limited by a thread pool and distributed lock mechanism.

[0095] In summary, by simultaneously monitoring metadata changes, data access behaviors, and business status events, and clarifying their technical significance, this approach achieves real-time, multi-dimensional capture of asset changes, eliminating blind spots in traditional technologies and ensuring high synchronization between the asset graph and the actual state. Maintaining a "set of visited nodes" during traversal and skipping nodes when circular dependencies are detected avoids infinite recursion. This enables safe and stable cascading state updates in real-world complex graphs (including circular dependencies), preventing system crashes and expanding the applicability of the technical solution. Furthermore, it integrates a triple optimization mechanism: priority degradation / sleep, cold start preheating, and clustering batch matching. This prevents inaccurate rule priority allocation during cold starts, automatically eliminates low-frequency rules during long-term operation, saving computational resources, and significantly reduces matching computation through clustering batch matching.

[0096] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the protection scope of the present invention.

Claims

1. A method for dynamic management of asset data, characterized in that, include: Build a real-time asset graph; Obtain the data stream events of each data node in the real-time asset map, and obtain the implicit boundary changes based on the data stream events; Based on the data stream events and implicit boundary changes, an asset update instruction for the target data object is generated; According to the asset update instruction, update the state attributes of the target data object, and traverse the downstream dependent data objects starting from the target data object to perform cascading state updates; During the traversal, a set of visited nodes is maintained. When a downstream dependent data object currently being traversed is detected to exist in the set of visited nodes, a circular dependency is determined and the downstream dependent data object is skipped. During the execution of the cascaded state update, for a single cascaded state update, when the number of downstream nodes exceeds the number threshold, a circuit breaker is triggered; when the traversal depth exceeds the depth limit, automatic downward traversal is stopped; for concurrent cascaded state updates; When the same loop path is detected more than a preset number of times within a preset time, a persistent loop alarm is triggered. When obtaining implicit boundary changes, the following are included: Based on the permission revocation event in the metadata change event, by monitoring the change log of the database authorization table and comparing the permission bits before and after the change, when it is determined that the permission has changed from authorized to unauthorized, the implicit boundary change of data access failure is identified. Based on data processing job failure events in data access behavior events, the job failure reason code is extracted and classified by a failure classifier. When it is determined to be a permanent failure, the implicit boundary change that caused the bloodline break is identified. Based on business status events, asset status change notifications issued by business systems are obtained by subscribing to message queues or adapting gateways to determine implicit boundary changes that have occurred in asset availability or integrity.

2. The dynamic management method for asset data according to claim 1, characterized in that, The real-time asset map includes: The system uses a distributed graph database to define node labels to distinguish data object types and employs directed edges to record lineage and computational dependencies between nodes. It receives changed data through a dedicated synchronization interface and submits it in batches according to a preset time window. It receives node change events through a distributed message queue and incrementally generates and updates the real-time asset graph.

3. The dynamic management method for asset data according to claim 1, characterized in that, When acquiring data stream events for each node of the real-time asset graph, the following are included: The real-time asset graph collects data stream events from each data node using probes, including a binlog parser, an audit log parser, and a message queue subscriber. The data stream events include metadata change events, data access behavior events, and business status events. The data stream events are then converted into standardized change vectors.

4. The dynamic management method for asset data according to claim 3, characterized in that, When generating asset update instructions for the target data object, the following are included: The standardized change vector is matched with the rules in the business rule base to determine the expected state of the target data object; based on the expected state, an asset update instruction for the target data object is generated.

5. The dynamic management method for asset data according to claim 4, characterized in that, When generating asset update instructions for the target data object, the following are included: Before generating the asset update instruction, the current state of the target data object is queried to determine whether migration from the current state to the expected state is permitted; if the check shows permission is granted, the asset update instruction for the target data object is generated; if the check shows permission is denied, the instruction generation is intercepted and an alarm is triggered.

6. The dynamic management method for asset data according to claim 1, characterized in that, When traversing downstream dependent data objects starting from the target data object and performing cascading state updates, the process includes: For downstream dependent data objects that have not been skipped, recursively checking whether their state has become invalid due to state changes of upstream data objects.

7. The dynamic management method for asset data according to claim 6, characterized in that, When recursively checking whether its state has become invalid due to a state change of an upstream data object, it also includes: For downstream dependent data objects determined to be invalid, if they are critical business objects, a state reassessment event is generated and pushed to a message queue for asynchronous processing; if they are non-critical business objects, a reassessment task is generated and stored in a cache queue for batch state reassessment.

8. The dynamic management method for asset data according to claim 1, characterized in that, After determining that a circular dependency exists and skipping downstream dependent data objects, the process includes: Add the skipped downstream dependent data objects and all downstream nodes to the delay re-evaluation queue; re-execute the cascading state update starting from the skipped downstream dependent data objects until the downstream dependent data objects are no longer in the loop path or the maximum number of retries is reached.

9. The dynamic management method for asset data according to claim 1, characterized in that, When performing concurrent cascading state updates, the following are included: Limit the maximum number of cascading state update processes that can be executed in parallel at the same time; limit the same target data object from being repeatedly updated by multiple concurrent processes.

Images