Data asset identification system and method based on multi-dimensional rule engine blood relationship analysis

Abstract

The invention relates to the technical field of data management and information, and discloses a data asset identification system and method based on multi-dimensional rule engine blood relationship analysis, and the method comprises the steps: obtaining static code information and dynamic operation tracking information of to-be-analyzed data; constructing a theoretical data blood relationship map based on the static code information; constructing a real-time data blood relationship map based on the dynamic operation tracking information; generating a fusion data consanguinity map by adopting a time window alignment strategy; constructing a five-dimensional data asset identification rule model; starting a blood relationship driven multi-dimensional rule engine; executing the multi-dimensional rule engine, and evaluating and determining the nature of the to-be-analyzed data; in the evaluation process, a conflict resolution mechanism is started. According to the method, the multi-dimensional rule engine is driven through dynamic and static combined blood relationship analysis, and conflict judgment is performed by using blood relationship information.

Description

Technical Field

[0001] This application relates to the fields of data governance and information technology, and in particular to a data asset identification method and system based on multi-dimensional rule engine lineage analysis. Background Technology

[0002] With the rapid development of big data, artificial intelligence, and cloud computing technologies, data has become a core strategic asset for enterprises and organizations. However, in complex, heterogeneous, and dynamically changing data environments, accurately, efficiently, and automatically identifying, inventorying, and managing massive amounts of data assets has become a serious challenge in the field of data governance. Traditional data asset management relies heavily on manual sorting, manual tagging, and static metadata management, which suffers from inefficiency, untimely updates, incomplete coverage, and susceptibility to errors, making it difficult to meet the agility requirements of modern enterprise data ecosystems.

[0003] To address these challenges, existing technologies have proposed a series of solutions for automated data asset management and identification. However, after in-depth research, the applicant found that while existing technologies have made significant progress in the automated identification and lineage analysis of data assets, when using multi-dimensional rules to identify data assets, rules from different dimensions may produce contradictory or inconsistent evaluation conclusions, and existing technologies generally lack effective conflict arbitration mechanisms. For example, structural rules based on content pattern matching may identify a data column as sensitive personal information, while rules based on metadata definitions may identify it as a common business identifier. In such cases, existing technologies often rely on preset static priorities or require manual intervention for adjudication, which not only reduces the degree of automation but also affects the accuracy and reliability of the final identification results. In particular, existing technologies fail to fully utilize the real-world usage context of data during runtime, i.e., dynamic lineage information, as an authoritative basis for resolving such rule conflicts, resulting in limited confidence in the identification.

[0004] Therefore, how to construct a mechanism that can utilize high-fidelity fused data lineage information to not only drive a multi-dimensional rule engine for accurate evaluation, but also to use dynamic lineage information for automated arbitration when rule conflicts occur, so as to further improve the accuracy, automation and confidence of data asset identification, is a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The objective of this application is to provide a data asset identification method based on multi-dimensional rule engine lineage analysis, comprising: acquiring static code information and dynamic runtime tracking information of the data to be analyzed; constructing a theoretical data lineage graph based on the static code information through abstract syntax tree parsing; constructing a real-time data lineage graph based on the dynamic runtime tracking information by capturing the actual flow path of the data during runtime through an observability framework; fusing the theoretical data lineage graph and the real-time data lineage graph using a time window alignment strategy to generate a fused data lineage graph; and constructing a five-dimensional data asset identification rule model, wherein the five-dimensional data asset identification rule model includes structural rules based on regular expressions and metadata rules based on a data dictionary. The five-dimensional data asset identification rule model employs a multi-dimensional rule engine driven by data lineage. This engine takes the fused data lineage graph as input and uses it as the basis for decision-making regarding the statistical and semantic rules. The multi-dimensional rule engine is then executed to evaluate and characterize the data to be analyzed. During the evaluation process, a conflict resolution mechanism is activated. When different dimensions of the five-dimensional data asset identification rule model produce contradictory identification conclusions, the dynamic tracking information recorded in the fused data lineage graph is used as the arbitration basis for automated decision-making and adjudication, ultimately outputting the data asset identification result.

[0006] By adopting the above technical solution, this application drives a multi-dimensional rule engine through dynamic and static lineage analysis and uses lineage information for conflict adjudication. This not only makes the data asset identification process more intelligent and context-aware, but also significantly improves the confidence and reliability of the final identification results through an automated conflict resolution mechanism. It reduces human intervention, improves the overall efficiency of data governance, and solves the technical problem of how to improve the accuracy and automation of data asset identification in complex data environments.

[0007] Furthermore, the generation of the fused data lineage graph includes: setting a predetermined time window; within the predetermined time window, associating, matching, and aligning the theoretical links representing data relationships at design time in the theoretical data lineage graph with the tracking trajectories representing data flow during actual operation in the real-time data lineage graph, ensuring the consistency between static structure and dynamic instances, thereby constructing the fused data lineage graph.

[0008] By adopting the above technical solution, it is possible to ensure that the statically designed structure corresponds one-to-one with the dynamically running instance, thereby constructing a high-fidelity fused data lineage map, providing more reliable and accurate input data for the subsequent rule engine.

[0009] Furthermore, the static code information includes structured query language scripts, job configuration files, and business logic processing code; the abstract syntax tree parsing is used to extract the dependencies between the data source, data transformation logic, and data destination from the static code information.

[0010] By adopting the above technical solutions, the integrity and breadth of the theoretical pedigree map are ensured, and it can cover the entire chain design of data processing.

[0011] Furthermore, the dynamic runtime tracing information is captured by deploying and integrating an agent or software development kit of the observability framework. The dynamic runtime tracing information includes data access requests, call chains of data processing functions, message queue transmission records, and data migration paths between different computing nodes.

[0012] By adopting the above technical solutions, cross-platform dynamic data tracking was achieved, ensuring the accuracy, real-time nature, and richness of detail in real-time pedigree data.

[0013] Furthermore, the five-dimensional data asset identification rule model is a unified and quantitative decision formula, wherein: the structural rules match the format and pattern of data content using regular expressions; the metadata rules verify data definitions by comparing predefined data dictionaries, technical metadata, and business metadata; the statistical rules analyze the statistical characteristics of the data, including numerical distribution, cardinality, null value rate, and access frequency; the semantic rules infer the business meaning of the data based on the upstream source and downstream consumption relationship of the data in the fused data lineage graph; and the compliance rules determine the sensitivity level and compliance risk of the data based on built-in laws and regulations, industry standards, or internal enterprise data security policies.

[0014] By adopting the above technical solutions, a comprehensive, structured, and quantifiable evaluation framework is provided, which can conduct in-depth analysis of data assets from multiple perspectives, including technology and business.

[0015] Furthermore, the activation of the lineage-driven multi-dimensional rule engine, using the fused data lineage graph as input, provides a decision-making basis for the statistical and semantic rules in the five-dimensional data asset identification rule model. This includes: using the dynamic statistical information of nodes in the fused data lineage graph as input parameters for the statistical rules, wherein the dynamic statistical information includes access frequency, update cycle, and data volume changes; and using the parent-child relationship, influence relationship, and business domain affiliation of neighboring nodes among data nodes in the fused data lineage graph as the basis for context inference by the semantic rules.

[0016] By adopting the above technical solutions, the rules are no longer isolated and static, but can dynamically adjust the evaluation logic according to the actual use of data and the context, thereby improving the accuracy of identification.

[0017] Furthermore, the conflict resolution mechanism includes: monitoring whether the five-dimensional data asset identification rule model outputs two or more inconsistent or contradictory classification labels or attribute conclusions when evaluating the same data object; if so, prioritizing the retrieval of dynamic operation tracking information directly related to the data object from the fused data lineage map; using the real usage scenario, processing logic, and flow path reflected in the dynamic operation tracking information as the highest priority evidence to arbitrate the two or more inconsistent or contradictory classification labels or attribute conclusions, and selecting the conclusion that best matches the highest priority evidence as the final identification result.

[0018] By adopting the above technical solutions, an automated and fact-based adjudication process was established, which greatly improved the accuracy of identification results in complex and ambiguous scenarios.

[0019] Furthermore, it also includes a feedback enhancement step: the label information of the data asset identification result is reverse-labeled onto the corresponding data node in the fused data lineage graph, thereby enriching the business semantics of the fused data lineage graph and providing more accurate contextual information for subsequent tasks. The label information of the data asset identification result includes asset type, business affiliation, and sensitivity level.

[0020] By adopting the above technical solutions, a closed-loop learning and enhancement system was created. By continuously enriching the business semantics of the graph, more accurate and richer contextual information is provided for subsequent asset identification and data governance tasks, thereby achieving continuous system evolution and performance improvement.

[0021] Furthermore, the method also includes setting configurable weight coefficients for each dimension in the five-dimensional data asset identification rule model. These weight coefficients are used to perform weighted calculations on the evaluation results of each dimension in the absence of conflict, generating a comprehensive confidence score to quantify the reliability of the identification results.

[0022] By adopting the above technical solution, a flexible way is provided to quantify the overall credibility of the identification results, allowing data governance personnel to adjust the importance of different dimensions according to business needs.

[0023] The second objective of this application is to provide a data asset identification system based on multi-dimensional rule engine lineage analysis, comprising: a data information acquisition unit for acquiring static code information and dynamic operation tracking information of the data to be analyzed; a data lineage analysis unit for constructing and fusing a fused data lineage graph based on the static code information and dynamic operation tracking information, wherein the data lineage analysis unit integrates a static abstract syntax tree parsing module, a dynamic observability tracking module, and a time window alignment fusion module; and a rule modeling and management unit for constructing and maintaining five-dimensional data asset identification rules including structural rules, metadata rules, statistical rules, semantic rules, and compliance rules. The model includes a core decision engine unit configured to receive the fused data lineage map as input, provide decision-making basis for the statistical and semantic rules in the five-dimensional data asset identification rule model, and evaluate and characterize the data to be analyzed; a conflict resolution unit, linked with the core decision engine unit, used to call the dynamic operation tracking information in the fused data lineage map as arbitration basis when contradictory conclusions arise in various dimensions of the five-dimensional data asset identification rule model, and to perform automated decision-making and adjudication; and an asset identification result output unit, used to output the data asset identification result after evaluation by the core decision engine unit and adjudication by the conflict resolution unit. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of the data asset identification method based on multi-dimensional rule engine lineage analysis in this application.

[0025] Figure 2 This is a block diagram of the data asset identification system based on lineage analysis using a multi-dimensional rule engine, as described in this application. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are only for explaining this application and are not intended to limit the scope of protection of this application.

[0027] This application constructs a comprehensive and accurate fusion data lineage map through a dynamic and static combined lineage analysis method. This fusion data lineage map not only drives a multi-dimensional rule engine to make intelligent decisions, but also uses dynamic running tracking information as the highest priority arbitration basis when rules conflict, thereby achieving automated identification of high-confidence data assets.

[0028] The following section will use a typical business scenario of a fintech company as an example to elaborate on the implementation details of each step of the method in this application and the collaborative working mechanism of each unit of the system, so as to fully demonstrate the technical connotation of this application.

[0029] Specifically, a fintech company's data platform carries massive amounts of user data, transaction data, and credit data. In order to comply with laws and regulations such as the "Personal Information Protection Law of the People's Republic of China," the company urgently needs to automatically inventory all data assets in its data platform and accurately identify which data categories belong to sensitive personal information, such as ID card numbers, mobile phone numbers, and bank card numbers, in order to implement differentiated data anonymization, encryption, and access control strategies.

[0030] like Figure 1 and Figure 2 As shown in the figure, the data asset identification method based on multi-dimensional rule engine lineage analysis in this application includes the following steps.

[0031] S01: Obtain static code information and dynamic runtime tracing information of the data to be analyzed.

[0032] This application employs a data information acquisition unit to collect comprehensive information from the data platform. Specifically, the information acquisition unit initiates static code information collection and dynamic runtime tracking information collection in parallel.

[0033] Specifically, the static code information collection process scans all data processing-related code assets in the company's code repository, including but not limited to: structured query language (SQL) scripts used to build data models in the data warehouse, such as DDL statements for creating tables and views and DML statements for data processing; job configuration files used to implement data extraction, transformation, and loading in the data integration platform, such as job definitions in Kettle or DataX; and business logic processing code embedded in various microservices or business systems, such as the parts of Java or Python code that involve database operations.

[0034] Simultaneously, dynamic runtime tracking information is collected. This collection process involves deploying and integrating a standardized observability framework called OTel (OpenTelemetry) into various runtime components of the data platform in a non-intrusive or low-intrusive manner. These runtime components include databases, microservices, message queues, and big data computing engines. This observability framework can capture and report the actual activity trajectory of data during system runtime in a unified format. This activity trajectory information is rich in detail, including the initiator of the data access request, the target table and column, the operation type (read / write), the call chain of the data processing function, the message queue transmission records, and the specific migration path of the data between different computing nodes. This information constitutes the true lineage of the data's "runtime."

[0035] After the information collection is completed, the data lineage analysis unit is used to construct and merge a fused data lineage map based on static code information and dynamic runtime tracking information. The data lineage analysis unit integrates a static abstract syntax tree parsing module, a dynamic observability tracking module, and a time window alignment fusion module.

[0036] S02: Based on static code information, a theoretical data lineage map is constructed through abstract syntax tree parsing.

[0037] Specifically, the static abstract syntax tree parsing module receives static code information and uses mature abstract syntax tree parsing technology to extract the dependencies between the data source, data transformation logic, and data destination from the static code information. For example, for a SQL script, it can be decomposed into a tree structure to accurately identify the source table, target table, join conditions, and column-level transformation logic in the query, thereby constructing a theoretical data lineage graph based on the design intent. This theoretical data lineage graph depicts how data should flow from the source to the destination in the design.

[0038] S03: Based on dynamic operation tracking information, through an observability framework, capture the actual flow path of data during operation and construct a real-time data lineage map.

[0039] Specifically, the Dynamic Observability Tracing module processes a large amount of real-time tracing logs collected from OpenTelemetry's standardized observability framework. This module cleans and transforms the raw log streams, mapping them to standardized entities (nodes) and relationships (edges) in a graph database. For example, database queries are transformed into "query event" nodes and relationships are established with "user" nodes, "service" nodes, and "data table" nodes to form a real-time data lineage graph. This real-time data lineage graph accurately records how data was actually accessed and processed at a specific point in time.

[0040] S04: Using a time window alignment strategy, theoretical data lineage maps and real-time data lineage maps are fused to generate a fused data lineage map.

[0041] Specifically, since the theoretical data lineage graph is static while the real-time data lineage graph is dynamic, inconsistencies may exist between the two. For example, code may have been modified but not deployed, or there may be temporary manual data operations. To address this issue, the time window alignment and fusion module employs a time window alignment strategy. A predetermined time window is set, such as the past 24 hours. Within this window, an entity mapping rule base is used for matching. This rule base contains a series of logics, such as matching table names in static code with access entities in dynamic logs; it also associates temporary tables or aliases in loading jobs through contextual logic. After successful alignment, the time window alignment and fusion module appends dynamic information as attributes to the corresponding links and nodes in the theoretical data lineage graph. This dynamic information includes the precise execution time of the job, the number of data rows processed, and the access frequency. Conversely, if a link in the theoretical data lineage graph has never appeared in the dynamic logs for a long period, it may be marked as "dormant" or "abandoned." Through this fusion process, a fused data lineage map was finally generated, which includes both macro-level design blueprints and micro-level operational realities.

[0042] S05: Construct a five-dimensional data asset identification rule model.

[0043] Specifically, after the data lineage graph is constructed, the core decision engine unit begins to evaluate the data assets to be analyzed. The evaluation is based on a five-dimensional data asset identification rule model predefined by the rule modeling and management unit. The five-dimensional data asset identification rule model includes structural rules based on regular expressions, metadata rules based on data dictionaries, statistical rules based on data distribution and access patterns, semantic rules based on upstream and downstream data relationships, and compliance rules based on industry standards and security policies.

[0044] Specifically, structural rules use regular expressions to match the format and pattern of data content. Structural rules focus on the inherent patterns of data content. The system will call a pre-built regular expression library to sample and scan the column data in the data table. For example, the regular expression ^1[3-9]\d{9}$ matches Chinese mobile phone numbers, the regular expression (^\d{15}$)|(^\d{18}$)|(^\d{17}(\d|X|x)$) matches 15-digit or 18-digit ID card numbers, and the regular expression ^[a-zA-Z0-9_-]+@[a-zA-Z0-9_-]+(\.[a-zA-Z0-9_-]+)+$ matches email addresses.

[0045] Suppose that after scanning, it is found that 95% of the content in a certain column of the data table matches the regular expression for a mobile phone number, this dimension will be output with a high score and a preliminary conclusion: structurally highly suspected to be a mobile phone number.

[0046] The metadata rules dimension verifies data definitions by comparing them with predefined data dictionaries, technical metadata, and business metadata. It checks data-related metadata information. The system automatically queries the enterprise-level metadata center or data dictionary. For example, it can check whether the name of a column in the data table contains keywords such as "phone", "mobile", or "tel", whether the column's comment explicitly states "contact phone number" or "user mobile phone", and whether its defined data type is a string. If so, the column will be commented as "user contact information". This dimension will output a high score, concluding that the metadata definition points to contact information.

[0047] The statistical rule dimension is used to analyze the statistical characteristics of the data and incorporate lineage information. The statistical characteristics include numerical distribution, cardinality, null value rate, and access frequency.

[0048] Specifically, basic statistical analysis is used by the system to calculate the cardinality of a column in a data table. If the cardinality is close to the total number of rows, it indicates that the data is unique and meets the characteristics of a personal identifier. At the same time, the null value rate is calculated. An extremely low null value rate may indicate that the field is a business-critical field.

[0049] Outlier detection can employ the IQR (Interquartile Range) method to detect data format anomalies. For example, by statistically analyzing the length of all strings in a column and calculating their first quartile (Q1) and third quartile (Q3), if most values ​​are 11 units long (i.e., the length of a phone number), and a small number of outliers are 5 or 20 units long, these values ​​will be flagged. The specific logic of IQR is: calculate IQR = Q3 - Q1; any data with a length less than Q1 - 1.5 * IQR or greater than Q3 + 1.5 * IQR is considered an outlier.

[0050] The core decision engine unit extracts dynamic statistical information related to a column in the data table from the fused data lineage graph. The dynamic statistical information includes access frequency, update cycle, data volume changes, etc. For example, the average daily access frequency, update cycle, and related data volume changes of this column in the past month. If it is found that this column is frequently read by multiple core business systems, this will greatly increase its importance score.

[0051] The semantic rule dimension relies entirely on the fused data lineage graph to infer the business meaning of the data. The core decision engine analyzes the position and connection relationship of column data in the data table in the fused data lineage graph, such as upstream source analysis or downstream consumption analysis.

[0052] Specifically, upstream source analysis traces the data source of this column, revealing that it originates from a message queue topic specifically designed for handling new user registration events. Downstream consumption analysis tracks the data's destination, finding that it is consumed by multiple downstream applications, including SMS marketing platforms, user login verification services, and customer relationship management systems. Based on this end-to-end flow context, even if the column's name and annotations are ambiguous, the semantic rule engine can confidently infer its business meaning as "user mobile phone numbers used for authentication and marketing outreach."

[0053] The compliance rules dimension maps the technical identification results to laws, regulations, or internal corporate policies. Predefined rules include: "If data is identified as a mobile phone number, ID card number, or bank card number, its sensitivity level is automatically classified as L3, i.e., highly sensitive." "If the data is L3 sensitive data, a column-level data anonymization strategy must be enabled by default." Since the previous dimensions all point to the column being a mobile phone number, the compliance rules dimension will output the conclusion: "According to compliance requirements, it should be classified as L3 sensitive data and requires anonymization."

[0054] It is understood that, in the embodiments of this application, configurable weight coefficients may also be set for each dimension in the five-dimensional data asset identification rule model. The weight coefficients are used to perform weighted calculations on the evaluation results of each dimension in the absence of conflict, and generate a comprehensive confidence score to quantify the reliability of the identification results.

[0055] S06: Launch the lineage-driven multi-dimensional rule engine, which takes the fused data lineage map as input to provide decision-making basis for statistical rules and semantic rules in the five-dimensional data asset identification rule model. The multi-dimensional rule engine includes rule engines with multiple dimensions such as structural rules, statistical rules, semantic rules and compliance rules.

[0056] Specifically, this step includes using the dynamic statistical information of nodes in the fused data lineage graph as input parameters for the statistical rules, wherein the dynamic statistical information includes access frequency, update cycle, and data volume changes; and using the parent-child relationship, influence relationship, and business domain affiliation of neighboring nodes in the fused data lineage graph as the basis for semantic rules to perform context inference.

[0057] S07: Execute the multi-dimensional rule engine to evaluate and characterize the data to be analyzed. During the evaluation process, a conflict resolution mechanism is activated. When different dimensions of the five-dimensional data asset identification rule model produce contradictory identification conclusions, the dynamic operation tracking information recorded in the fused data lineage map is used as the arbitration basis to make automated decisions and adjudicate, and finally output the data asset identification results.

[0058] Understandably, the conflict resolution mechanism includes: monitoring whether the five-dimensional data asset identification rule model outputs two or more inconsistent or contradictory classification labels or attribute conclusions when evaluating the same data object; if so, prioritizing the retrieval of dynamic operation tracking information directly related to the data object from the fused data lineage map; using the real usage scenarios, processing logic, and flow paths reflected in the dynamic operation tracking information as the highest priority evidence to arbitrate the two or more inconsistent or contradictory classification labels or attribute conclusions, and selecting the conclusion that best matches the highest priority evidence as the final identification result.

[0059] Specifically, in certain complex situations, rules from different dimensions may conflict. For example, suppose a column in a data table contains content that perfectly matches the structure of an ID card number, but its metadata rule's column annotation is "internal QA test account," and semantic rules, through lineage analysis, reveal that its upstream and downstream connections are only to the test environment's database and automated test scripts. In this case, the conflict resolution unit is automatically triggered, executing an arbitration process.

[0060] Specifically, the system detects a serious conflict between the conclusion of "L3-level sensitive data" from the structural rules and the conclusion of "non-sensitive test data" from the metadata and semantic rules. At this time, according to the preset adjudication logic: "dynamic operation facts take precedence over static design and content patterns", the conflict resolution unit will prioritize retrieving dynamic operation tracking information directly related to the column from the fused data lineage graph.

[0061] Dynamic runtime tracking information clearly shows that over the past few months, all access requests to this column have originated from the internal test server IP address range, the user accounts performing these operations are all test accounts, and the output of all computational tasks that processed this data has never flowed into any production system. This irrefutable runtime evidence proves that the real-world use case for this data is a closed test environment.

[0062] Based on dynamic operation tracking information, which is the highest priority evidence, the conflict resolution unit will determine that the conclusions of metadata and semantic rules are more consistent with the facts, thereby rejecting the conclusions of structural rules based solely on content pattern matching.

[0063] Finally, the asset identification result output unit will output a high-confidence final result after arbitration: the business category of this column is the quality assurance domain, the asset type is test data, the sensitivity level is L0, i.e., non-sensitive, and the confidence score is 99%.

[0064] S08: Feedback Enhancement Step: The label information of the data asset identification results is reverse-labeled onto the corresponding data nodes in the fused data lineage graph, thereby enriching the business semantics of the fused data lineage graph and providing more accurate contextual information for subsequent tasks. The label information of the data asset identification results includes asset type, business affiliation, and sensitivity level.

[0065] Specifically, once the above-mentioned high-confidence data asset identification results are generated, the system will automatically perform a reverse annotation operation, permanently annotating the label information such as "the business belongs to the quality assurance domain" and "the sensitivity level is L0" as new attributes onto the corresponding data nodes in the fused data lineage map.

[0066] By using feedback enhancement steps, a closed loop of knowledge accumulation is created. In future recognition tasks, when the semantic rule engine or other modules analyze this node or its neighboring nodes again, they can directly use this verified and accurate label information, making subsequent recognition faster and more accurate. This mechanism enables the knowledge base of the entire data asset recognition system to be continuously enriched and evolved over time.

[0067] In summary, this application enhances the breadth and depth of identification through a combination of static and dynamic lineage analysis and a multi-dimensional rule engine. More importantly, its conflict resolution mechanism based on dynamic lineage addresses the shortcomings of existing technologies in handling ambiguous and contradictory scenarios, significantly improving the automation level of data asset identification and the accuracy and reliability of the final results. This provides a strong technical guarantee for enterprises to achieve refined and intelligent data governance in complex data environments.

[0068] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is used as an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the system, device, and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0069] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0070] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0071] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0072] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0073] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A data asset identification method based on lineage analysis using a multi-dimensional rule engine, characterized in that, include: Obtain static code information and dynamic runtime tracing information of the data to be analyzed; Based on the static code information, a theoretical data lineage map is constructed through abstract syntax tree parsing; Based on the dynamic operation tracking information, the actual flow path of data during operation is captured through the observability framework, and a real-time data lineage map is constructed. A time window alignment strategy is adopted to fuse the theoretical data lineage map with the real-time data lineage map to generate a fused data lineage map; A five-dimensional data asset identification rule model is constructed, which includes structural rules based on regular expressions, metadata rules based on data dictionaries, statistical rules based on data distribution and access patterns, semantic rules based on upstream and downstream data relationships, and compliance rules based on industry standards and security policies. The lineage-driven multi-dimensional rule engine is activated, and the fused data lineage map is used as input to provide decision-making basis for the statistical and semantic rules in the five-dimensional data asset identification rule model; The multi-dimensional rule engine is executed to evaluate and characterize the data to be analyzed; During the evaluation process, a conflict resolution mechanism is activated. When different dimensions of the five-dimensional data asset identification rule model produce contradictory identification conclusions, the dynamic operation tracking information recorded in the fused data lineage map is used as the arbitration basis to make automated decisions and adjudications, and finally output the data asset identification results.

2. The data asset identification method based on multi-dimensional rule engine lineage analysis according to claim 1, characterized in that, The generation of the fused data lineage map includes: Set a scheduled time window; Within the predetermined time window, the theoretical links representing data relationships during design in the theoretical data lineage graph are associated, matched, and aligned with the tracking trajectories representing data flow during actual operation in the real-time data lineage graph to ensure consistency between static structure and dynamic instances, thereby constructing the fused data lineage graph.

3. The data asset identification method based on multi-dimensional rule engine lineage analysis according to claim 1, characterized in that, The static code information includes structured query language scripts, job configuration files, and business logic processing code; The abstract syntax tree parsing is used to extract the dependencies between the data source, data transformation logic, and data destination from the static code information.

4. The data asset identification method based on multi-dimensional rule engine lineage analysis according to claim 1, characterized in that, The dynamic runtime tracking information is captured by deploying and integrating an agent or software development kit of the observability framework. The dynamic runtime tracking information includes data access requests, call chains of data processing functions, message queue transmission records, and data migration paths between different computing nodes.

5. The data asset identification method based on multi-dimensional rule engine lineage analysis according to claim 1, characterized in that, The five-dimensional data asset identification rule model is a unified and quantitative decision formula, wherein: The structural rules use regular expressions to match the format and pattern of the data content; The metadata rules verify data definitions by comparing predefined data dictionaries, technical metadata, and business metadata. The statistical rules analyze the statistical characteristics of the data, including numerical distribution, cardinality, null value rate, and access frequency. The semantic rules infer their business meaning based on the upstream sources and downstream consumption of data in the fused data lineage graph; The compliance rules determine the sensitivity level and compliance risks of data based on built-in laws and regulations, industry standards, or the company's internal data security policies.

6. The data asset identification method based on multi-dimensional rule engine lineage analysis according to claim 1, characterized in that, The aforementioned kinship-driven multi-dimensional rule engine takes the fused data kinship map as input and provides decision-making basis for the statistical and semantic rules in the five-dimensional data asset identification rule model, including: The dynamic statistical information of nodes in the fused data lineage graph is used as the input parameter of the statistical rule, wherein the dynamic statistical information includes access frequency, update cycle, and data volume change; The parent-child relationships, influence relationships, and business domain affiliations of neighboring nodes in the fused data lineage graph are used as the basis for context inference in the semantic rules.

7. The data asset identification method based on multi-dimensional rule engine lineage analysis according to claim 1, characterized in that, The conflict resolution mechanism includes: The monitoring system checks whether the five-dimensional data asset identification rule model outputs two or more inconsistent or contradictory classification labels or attribute conclusions when evaluating the same data object. If so, prioritize retrieving dynamic operation tracking information directly related to the data object from the fused data lineage map; The real-world usage scenarios, processing logic, and flow paths reflected in the dynamic operation tracking information are taken as the highest priority evidence. Arbitration is conducted on two or more inconsistent or contradictory classification labels or attribute conclusions, and the conclusion that best matches the highest priority evidence is selected as the final identification result.

8. The data asset identification method based on multi-dimensional rule engine lineage analysis according to claim 1, characterized in that, It also includes feedback enhancement steps: The label information of the data asset identification results is reverse-labeled onto the corresponding data nodes in the fused data lineage graph, thereby enriching the business semantics of the fused data lineage graph and providing more accurate contextual information for subsequent tasks. The label information of the data asset identification results includes asset type, business affiliation, and sensitivity level.

9. The data asset identification method based on multi-dimensional rule engine lineage analysis according to any one of claims 1 to 8, characterized in that, It also includes setting configurable weight coefficients for each dimension in the five-dimensional data asset identification rule model. These weight coefficients are used to perform weighted calculations on the evaluation results of each dimension in the absence of conflict, generating a comprehensive confidence score to quantify the reliability of the identification results.

10. A data asset identification system based on multi-dimensional rule engine lineage analysis, characterized in that, include: The data information acquisition unit is used to acquire static code information and dynamic runtime tracing information of the data to be analyzed. The data lineage analysis unit is used to construct and fuse a fused data lineage graph based on the static code information and dynamic runtime tracking information. The data lineage analysis unit integrates a static abstract syntax tree parsing module, a dynamic observability tracking module, and a time window alignment fusion module. The rule modeling and management unit is used to build and maintain a five-dimensional data asset identification rule model that includes structural rules, metadata rules, statistical rules, semantic rules, and compliance rules. The core decision engine unit is configured to receive the fused data lineage map as input, provide decision basis for the statistical and semantic rules in the five-dimensional data asset identification rule model, and evaluate and characterize the data to be analyzed. The conflict resolution unit, in conjunction with the core decision engine unit, is used to call the dynamic operation tracking information in the fused data lineage map as the arbitration basis when contradictory conclusions are generated in each dimension of the five-dimensional data asset identification rule model, and to make automated decisions and rulings. The asset identification result output unit is used to output the data asset identification result after being evaluated by the core decision engine unit and adjudicated by the conflict resolution unit.

Images