Data management method and device based on blood relationship and rule propagation and electronic equipment

Abstract

The invention provides a data management method and device based on blood relationship and rule propagation and electronic equipment, and the method comprises the steps: analyzing an upstream and downstream dependency relationship between to-be-analyzed scripts, and extracting corresponding metadata and metadata check rules; according to the upstream and downstream dependency relationship, reversely analyzing the script to be analyzed, and extracting a plurality of target fields; performing blood relationship traceability on each target field by adopting statement-level traceability and node-level traceability to obtain corresponding original tables and original fields; metadata checking rules corresponding to the original tables and the original fields are determined according to the metadata, multiple first target rules are obtained, metadata checking rules corresponding to the target fields are determined according to the first target rules, and second target rules are obtained; and determining a quality score of each target field according to a second target rule, and giving an alarm when each corresponding quality score is smaller than a first threshold value. The method solves the problem of inaccurate data quality supervision in the prior art.

Description

Technical Field

[0001] This invention relates to the field of data management technology, and more specifically, to a data management method, apparatus, computer-readable storage medium, and electronic device based on lineage and rule propagation. Background Technology

[0002] Traditional data inspection rule generation simply generates default rules based on field types (such as screening fields and measurement fields). However, when field combinations and processing result in derived fields, cross-level rule transfer is not possible. Without a transfer mechanism, more refined business inspection rules require extensive configuration. Furthermore, if rule transfer cannot be achieved through lineage relationships, data inspection rules can only apply simple default rules in scenarios with frequent derived fields, leading to inaccurate data quality supervision of derived fields. Summary of the Invention

[0003] The main objective of this application is to provide a data management method, apparatus, computer-readable storage medium, and electronic device based on lineage and rule propagation, so as to at least solve the problem of inaccurate data quality supervision in the prior art.

[0004] To achieve the above objectives, according to one aspect of this application, a data management method based on lineage and rule propagation is provided, comprising: extracting scripts to be analyzed from a database and analyzing the upstream and downstream dependencies between the scripts to be analyzed; extracting metadata and metadata checking rules corresponding to the scripts to be analyzed from the database, wherein the metadata includes field types and primary key constraints; parsing the scripts to be analyzed in reverse order according to the upstream and downstream dependencies; extracting feature fields according to the syntax characteristics of SQL statements to obtain multiple target fields; performing lineage tracing on each target field using statement-level tracing and node-level tracing to determine the original table and original field corresponding to each target field; determining the metadata checking rules corresponding to each original table and original field according to the metadata to obtain multiple first target rules; determining the metadata checking rules corresponding to each target field according to the first target rules to obtain second target rules; checking the target fields according to the second target rules to determine the quality score of each target field; and issuing an alarm for each target field when the corresponding quality score is less than a first threshold.

[0005] Optionally, extracting the scripts to be analyzed from the database and analyzing the upstream and downstream dependencies between the scripts to be analyzed includes: extracting the node sequence number and node script name corresponding to each script to be analyzed to obtain a first sequence number and a first script name; determining the parent node sequence number and parent node script name corresponding to each script to be analyzed based on the first sequence number and the first script name to obtain a second sequence number and a second script name; and determining the upstream and downstream dependencies between the scripts to be analyzed based on the first sequence number, the second sequence number, the first script name, and the second script name.

[0006] Optionally, the script to be analyzed is parsed in reverse order according to the upstream and downstream dependencies, and feature fields are extracted according to the SQL statement syntax characteristics to obtain multiple target fields, including: determining multiple feature fields according to the SQL statement syntax characteristics, parsing each script to be analyzed, and matching the feature fields with the scripts to be analyzed to obtain the target fields; if the feature field does not have a corresponding target field in the corresponding script to be analyzed, the upstream script of the script to be analyzed is matched according to the upstream and downstream dependencies to obtain the target fields.

[0007] Optionally, statement-level tracing is used to perform lineage tracing on each of the target fields to determine the original table and original field corresponding to each target field. This includes: when there are nested queries in the script to be analyzed to which the target field belongs, tracing inwards through the table alias prefix to determine the corresponding original table and original field through statement-level tracing.

[0008] Optionally, a lineage tracing method is used to trace the origin of each target field to determine the original table and original field corresponding to each target field. This includes: taking the script to be analyzed corresponding to the target field as the current node, checking the upstream scripts in sequence according to the upstream and downstream dependencies, until the original table and the original field are obtained.

[0009] Optionally, the second target rule is obtained by determining the metadata inspection rule corresponding to each target field according to the first target rule, including: obtaining the data logical relationship between the target field and the corresponding original field from the script to be analyzed corresponding to the target field; converting the first target rule and the data logical relationship into a natural language description to obtain input data; and inputting the input data into a pre-trained large language model for processing to obtain the second target rule.

[0010] Optionally, the target fields are checked according to the second target rule to determine the quality score of each target field, including: traversing the target fields in the script to be analyzed, determining the ratio of the number of non-empty fields in the target fields to the total number of fields, and obtaining a completeness score; matching the second target rule with each target field in the script to be analyzed, and determining the ratio of the number of fields that pass the match to the number of fields that fail the match, and obtaining an accuracy score; calculating the ratio of the number of fields that pass the match to the number of fields that fail the match within a preset time period before the current time, and obtaining a reliability score; and performing a weighted calculation on the completeness score, the accuracy score, and the reliability score to obtain the quality score.

[0011] According to another aspect of this application, a data management device based on lineage and rule propagation is provided. The device includes: an acquisition unit, configured to extract scripts to be analyzed from a database and analyze the upstream and downstream dependencies between the scripts to be analyzed, and extract metadata and metadata checking rules corresponding to the scripts to be analyzed from the database, wherein the metadata includes field types and primary key constraints; a first processing unit, configured to parse the scripts to be analyzed in reverse order according to the upstream and downstream dependencies, extract feature fields according to the syntax characteristics of SQL statements, and obtain multiple target fields; a second processing unit, configured to perform lineage tracing on each target field using statement-level tracing and node-level tracing to determine the original table and original field corresponding to each target field; a third processing unit, configured to determine the metadata checking rules corresponding to each original table and original field according to the metadata, obtain multiple first target rules, determine the metadata checking rules corresponding to each target field according to the first target rules, and obtain second target rules; and a calculation unit, configured to check the target fields according to the second target rules to determine the quality score of each target field, and issue an alarm for the target field when the corresponding quality score is less than a first threshold.

[0012] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform any of the methods described.

[0013] According to another aspect of this application, an electronic device is provided, comprising: one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including methods for any one of the methods.

[0014] Applying the technical solution of this application, in the aforementioned data management method based on lineage and rule propagation, firstly, the scripts to be analyzed are extracted from the database, and the upstream and downstream dependencies between each script are analyzed. Metadata and metadata checking rules corresponding to the scripts to be analyzed are extracted from the database. The metadata includes field types and primary key constraints. Then, the scripts to be analyzed are parsed in reverse order according to the upstream and downstream dependencies, and feature fields are extracted according to the syntax characteristics of SQL statements to obtain multiple target fields. Next, statement-level tracing and node-level tracing are used to perform lineage tracing on each target field to determine the original table and original field corresponding to each target field. Then, the metadata checking rules corresponding to each original table and original field are determined according to the metadata to obtain multiple first target rules. The metadata checking rules corresponding to each target field are determined according to the first target rules to obtain second target rules. Finally, the target fields are checked according to the second target rules to determine the quality score of each target field. If the corresponding quality score is less than a first threshold, an alarm is issued for the target field. This application introduces statement-level tracing to trace the source of data through hierarchical nesting, and combines it with node-level tracing to trace the source of data through process dependencies, realizing the cross-layer propagation of inspection rules. Compared with the existing technology that intelligently generates default rules according to field type, and the generated data inspection rules cannot be applied to derived fields after field processing, this application improves the complexity of inspection rules for derived fields through lineage tracing and rule propagation, and improves the accuracy of data quality monitoring and management through detailed inspection, thereby solving the problem of inaccurate data quality supervision in the existing technology. Attached Figure Description

[0015] Figure 1 A hardware structure block diagram of a mobile terminal for a data management method based on lineage and rule propagation provided in an embodiment of this application is shown.

[0016] Figure 2 A flowchart illustrating a data management method based on lineage and rule propagation according to an embodiment of this application is shown.

[0017] Figure 3 A structural block diagram of a data management device based on lineage and rule propagation provided according to an embodiment of this application is shown. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0020] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate for the embodiments of this application described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0021] As described in the background section, existing technologies cannot pass rules through lineage relationships. In scenarios where derived fields frequently appear, data inspection rules can only apply simple default rules, resulting in inaccurate data quality supervision of derived fields. To solve the problem of inaccurate data quality supervision in existing technologies, embodiments of this application provide a data management method, apparatus, computer-readable storage medium, and electronic device based on lineage and rule propagation.

[0022] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention.

[0023] The methods and embodiments provided in this application can be executed on a mobile terminal, computer terminal, or similar computing device. Taking running on a mobile terminal as an example, Figure 1 This is a hardware structure block diagram of a mobile terminal for a data management method based on lineage and rule propagation, according to an embodiment of the present invention. Figure 1 As shown, a mobile terminal may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 (which may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.) and a memory 104 for storing data are also shown. The mobile terminal may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1The structure shown is for illustrative purposes only and does not limit the structure of the mobile terminal described above. For example, the mobile terminal may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0024] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data management method based on lineage and rule propagation in this embodiment of the invention. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thereby implementing the above-described method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the mobile terminal via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof. The transmission device 106 is used to receive or send data via a network. Specific examples of the aforementioned networks may include wireless networks provided by the mobile terminal's communication provider. In one example, the transmission device 106 includes a network interface controller (NIC), which can be connected to other network devices via a base station to communicate with the Internet. In one example, the transmission device 106 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0025] This embodiment provides a data management method based on lineage and rule propagation that runs on a mobile terminal, computer terminal, or similar computing device. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0026] Figure 2 This is a flowchart of a data management method based on lineage and rule propagation according to an embodiment of this application. Figure 2 As shown, the method includes the following steps:

[0027] Step S201: Extract the scripts to be analyzed from the database and analyze the upstream and downstream dependencies between the scripts to be analyzed. Extract the metadata and metadata checking rules corresponding to the scripts to be analyzed from the database. The metadata includes field types and primary key constraints.

[0028] Understandably, the core of the data preprocessing stage lies in accurately understanding and capturing the structure and relationships of the data. This involves extracting relevant scripts from the database for data processing or analysis, while simultaneously analyzing the dependencies between them. Furthermore, metadata from the tables involved in the scripts is extracted concurrently, including field types, primary keys, and other information, as well as defined metadata checking rules.

[0029] In practice, database connection tools or APIs (Application Programming Interfaces) are used to extract all relevant SQL scripts (Structured Query Language) from the database, and these scripts are initially cleaned, such as removing comments, whitespace characters, and standardizing syntax. By analyzing table names, subqueries, and JOIN (Java Object Notation) operations within the scripts, a context dependency graph is established, with each script marked as a node in the processing flow. The table structure (field types, primary key constraints, etc.) is read from the target database, and all predefined metadata quality check rules are collected.

[0030] The above operations accurately recorded the dependencies between scripts, providing guidance for subsequent lineage analysis. Cleaning the scripts improved the efficiency and accuracy of subsequent parsing. The collected and integrated metadata and rules facilitated the generation and application of intelligent checking rules in later steps.

[0031] Step S202: Parse the script to be analyzed in reverse order according to the upstream and downstream dependencies, extract feature fields according to the syntax characteristics of the SQL statement, and obtain multiple target fields;

[0032] Understandably, reverse parsing scripts can effectively trace the data flow path from downstream to source, helping to identify which fields are the result of complex processing and their original state. Identifying and tagging characteristic fields using SQL statement keywords (such as GROUP BY and JOIN ON) can be a key focus of quality checks.

[0033] In practice, the script dependency graph is parsed in reverse order, starting from the last script and tracing back step by step. For each script, a parser is used to identify fields following keywords such as GROUP BY and JOIN ON, and these fields are marked as dimension fields, related fields, etc.

[0034] Through the above operations, and by parsing the script in reverse order, the integrity and traceability of data lineage were ensured. Accurate identification of feature fields provided crucial information for subsequent quality rule generation.

[0035] Step S203: Use statement-level tracing and node-level tracing to perform lineage tracing for each target field in order to determine the original table and original field corresponding to each target field;

[0036] Understandably, lineage tracing refers to tracking the entire path of data from its original source to its current state. This step is performed in two ways: at the statement level and at the node level. The former focuses on the source of fields within a single SQL statement, while the latter considers the data flow more broadly throughout the entire processing flow.

[0037] In practice, within SQL statements containing subqueries, the initial source table of a field is determined by analyzing table aliases and their associated subqueries. Based on dependency analysis, the process is traced sequentially from node to node until the original source table of the field is found.

[0038] The above operations accurately determined the original source of the target field, providing a basis for rule matching.

[0039] Step S204: Determine the metadata inspection rules corresponding to each original table and original field based on the metadata to obtain multiple first target rules; determine the metadata inspection rules corresponding to each target field based on the first target rules to obtain second target rules.

[0040] Understandably, corresponding quality inspection rules are generated based on the field type and lineage information. By utilizing field types and primary key constraints in metadata, combined with the identification results of feature fields, rules can be customized, and more complex business rules can be generated further through the reasoning capabilities of large models.

[0041] In practice, basic rules are generated based on field types, such as NOT NULL and numeric range. The field relationship description is converted into natural language, input into a large model, and business logic rules are generated, such as if C=AB, then C>=0 needs to be checked.

[0042] Through the above operations, the quality rules were accurately generated, covering not only traditional field type rules but also rules based on business logic, thereby improving the effectiveness and relevance of the rules.

[0043] Step S205: Check the target fields according to the second target rule to determine the quality score of each target field. If the corresponding quality score is less than the first threshold, issue an alarm for the target field.

[0044] Understandably, data quality is assessed in real-time or periodically by executing the generated inspection rules. The scoring mechanism, based on the execution results, reflects the completeness, accuracy, and reliability of the data.

[0045] In practice, corresponding quality rules are executed at each data processing node, and the results are recorded. A weighted average method is used, combining indicators such as completeness, accuracy, and reliability, to calculate a comprehensive quality score.

[0046] Through the above operations, data quality was quantitatively evaluated, allowing business personnel to intuitively understand the health status of the data. Timely alerts were issued for fields falling below the threshold, promoting immediate data governance.

[0047] In this embodiment, firstly, the scripts to be analyzed are extracted from the database, and the upstream and downstream dependencies between the scripts are analyzed. Metadata and metadata checking rules corresponding to the scripts to be analyzed are extracted from the database. The metadata includes field types and primary key constraints. Then, the scripts to be analyzed are parsed in reverse order according to the upstream and downstream dependencies. Feature fields are extracted according to the syntax characteristics of SQL statements to obtain multiple target fields. Next, statement-level tracing and node-level tracing are used to trace the lineage of each target field to determine the original table and original field corresponding to each target field. Then, the metadata checking rules corresponding to each original table and original field are determined according to the metadata to obtain multiple first target rules. The metadata checking rules corresponding to each target field are determined according to the first target rules to obtain second target rules. Finally, the target fields are checked according to the second target rules to determine the quality score of each target field. If the corresponding quality score is less than a first threshold, an alarm is issued for the target field. This application introduces statement-level tracing to trace the source of data through hierarchical nesting, and combines it with node-level tracing to trace the source of data through process dependencies, realizing the cross-layer propagation of inspection rules. Compared with the existing technology that intelligently generates default rules according to field type, and the generated data inspection rules cannot be applied to derived fields after field processing, this application improves the complexity of inspection rules for derived fields through lineage tracing and rule propagation, and improves the accuracy of data quality monitoring and management through detailed inspection, thereby solving the problem of inaccurate data quality supervision in the existing technology.

[0048] In order to determine the upstream and downstream relationships between the scripts to be analyzed, in one optional implementation, step S201 above includes:

[0049] Step S2011: Extract the node sequence number and node script name corresponding to each script to be analyzed to obtain the first sequence number and the first script name;

[0050] Understandably, in a data processing chain, each script or data processing node needs a clear location identifier for management and tracking. Node sequence numbers and script names serve as such identifiers, used to locate the role and position of each script in the dependency graph.

[0051] In practice, during the script extraction phase, each script is assigned a unique node sequence number, and the script name is recorded. This information can typically be obtained through database logs, metadata, or manual configuration.

[0052] By using the node sequence numbers as described above, we can ensure that the system executes scripts in the correct order, preventing data from being out of order or missing. The script name provides a brief description of the script's function, making it easier for data engineers to understand its purpose.

[0053] Step S2012: Determine the parent node sequence number and parent node script name of the corresponding script to be analyzed based on each first sequence number and first script name, and obtain the second sequence number and second script name;

[0054] Understandably, data flows are typically not linear but rather form a mesh structure, where some nodes may have multiple parent nodes or a single parent node may send data to multiple downstream nodes. Determining the parent node's sequence number and script name is crucial for clarifying the data's origin and how it is correctly transmitted between multiple nodes.

[0055] In practice, by analyzing references in the script, such as table names in SELECT FROM statements, we can determine which elements are the script's input sources, i.e., its "parent nodes."

[0056] Through the above operations, this step makes the dependency graph more complex and accurate, reflecting the true branches and merging of data flows. Explicit identification of parent nodes facilitates deeper data analysis, such as data impact analysis and reverse lookups.

[0057] Step S2013: Determine the upstream and downstream dependencies between each script to be analyzed based on the first sequence number, the second sequence number, the first script name, and the second script name.

[0058] Understandably, data processing workflows typically involve multiple stages, each extracting or processing data from a data source using SQL scripts or other scripting languages. These scripts have a strict execution order and data flow dependencies, where the results generated by upstream scripts may serve as input for downstream scripts.

[0059] In practice, the extracted first sequence number and second sequence number, as well as the first script name and second script name, are associated to form a dependency graph. The graph clearly shows the data flow path and the order in which the scripts are executed.

[0060] Through the above operations, the construction of a dependency graph allows data engineers to easily trace the complete path of data from its source to its destination. With clearly defined upstream and downstream relationships, more granular quality control rules can be developed based on the data flow, ensuring appropriate monitoring measures are in place at each stage of data processing.

[0061] Through the above embodiments, the construction of dependency graphs simplifies the data lineage analysis process, enabling data engineers to quickly pinpoint the root causes of data quality issues. Understanding the dependencies between scripts allows for more precise formulation of data quality inspection rules, ensuring that the rules are both comprehensive and practical. Establishing parent node information facilitates higher-level data analysis. The use of node sequence numbers and script names, along with the visualization of dependencies, improves the overall visibility and controllability of the data processing workflow, which is beneficial for team collaboration and project management.

[0062] In order to obtain key data features from each script to be analyzed, in one optional implementation, step S202 above includes:

[0063] Step S2021: Determine multiple feature fields based on the syntax characteristics of the SQL statement, parse each script to be analyzed, and match the feature fields with the script to be analyzed to obtain the target fields;

[0064] Understandably, during the data preprocessing stage, the data flow follows the order in which the scripts are executed, with data generated by upstream scripts serving as input for downstream scripts. Reverse-order script parsing allows for more effective identification of the final processed form of the data and the characteristic fields of the data when it reaches a specific script.

[0065] In practice, based on the upstream and downstream dependency graph constructed in the above steps, parsing begins with the last executed script, i.e., the very end of the data processing chain. The data at the end of the processing flow will present its final form, providing more direct reference value for formulating data quality inspection rules. Specifically, an SQL parser is used to identify key field types, such as filter fields in GROUP BY, JOIN ON, and WHERE conditions.

[0066] Through the above operations and reverse parsing, the system can prioritize identifying fields that have undergone complex processing. These fields are usually the focus of data quality checks. For example, the last script might contain a field called "total transaction amount." Reverse parsing can identify that this is a metric field aggregated from multiple "transaction amount" fields, thus generating more accurate non-empty or numerical range rules in subsequent steps.

[0067] As we can understand, feature fields refer to fields that play a specific role in SQL statements. For example, fields involved in GROUPBY operations might be considered dimension fields, while fields under the JOIN ON condition are related to data consistency and integrity. By matching these feature fields with the script, the characteristics of the data can be further refined, providing key information for the generation of quality rules.

[0068] In practice, SQL syntax analysis techniques are used in each parsed script to identify fields such as GROUP BY, JOIN ON, and WHERE conditions. These identified characteristic fields are then marked in their corresponding scripts, forming a script-field mapping relationship that facilitates subsequent lineage analysis and rule generation.

[0069] Through the above operations, the system is able to accurately capture key fields in each script that affect data quality. For example, "transaction date" may appear in multiple WHERE conditions to represent the time range of the data, while "customer ID" may be used as a JOIN ON condition, affecting the integrity of data association. This process provides a solid foundation for subsequent lineage analysis and rule generation.

[0070] Step S2022: If the feature field does not have a corresponding target field in the corresponding script to be analyzed, match the upstream script of the script to be analyzed according to the upstream and downstream dependencies to obtain the target field.

[0071] Understandably, in complex data processing workflows, a feature field in a script might not be directly generated within the current script but inherited from an upstream script. Therefore, when the processing logic for a feature field cannot be found in the current script, it's necessary to trace back to the upstream script to fully understand the data flow and characteristics. If the direct processing logic for a feature field cannot be found in the current script, the system will check the preceding SQL scripts or data processing nodes in reverse order based on the upstream and downstream dependency graph to find the generation process of that field. Once the processing logic for a feature field is found in the upstream script, it is marked as the target field of the current script, even if the field only appears as input or a parameter in the current script.

[0072] Through the above operations, the system can ensure that all key fields involved in the data flow are accurately identified and marked, even if their generation logic is distributed across multiple scripts. For example, if "total transaction amount" only appears as input in the final script, by tracing the upstream scripts, the system can identify that it is aggregated from multiple "transaction amount" fields through a SUM operation, and use it as the target field for subsequent quality rule generation.

[0073] Through the above embodiments, starting from the end of data processing, the system can quickly locate the fields requiring quality inspection, avoiding indiscriminate analysis of all fields and greatly improving the targeting and efficiency of rule generation. By identifying feature fields and tracing upstream scripts, the system can generate detection rules covering all key fields, avoiding rule omissions due to field omissions, while ensuring consistency between rules and data processing logic. Lineage analysis is not limited to a single script but runs through the entire data processing flow, enabling the system to deeply understand the data flow process, which is particularly important for complex data processing workflows.

[0074] In order to determine the data source for each target field, in one optional implementation, step S203 above includes:

[0075] Step S2031: If there are nested queries in the script to be analyzed to which the target field belongs, trace back inwards through the table alias prefix to determine the corresponding original table and original field through statement-level tracing.

[0076] Understandably, in data processing, data is often processed and refined through nested SQL queries and operations. Each level of nested query can potentially alter the structure and content of the data. Therefore, to accurately determine the original source of a target field (i.e., the field requiring data quality checks), statement-level tracing is necessary. This involves understanding and parsing the structure of SQL statements, using table alias prefixes to pinpoint the specific source of the field, thereby achieving precise tracing of the data's lineage.

[0077] In practice, the first step is to parse the SQL script and identify nested query structures. Nested queries typically consist of a SELECT statement containing one or more SELECT statements to generate or filter data. For identified nested queries, the system focuses on the innermost SELECT statement. By analyzing the FROM clause in the inner query and the use of table aliases, the system can trace back inwards until it finds the original definition of the target field. Table alias prefixes (such as t1.col1) are used to explicitly indicate the specific source table of a field in a nested query. Once the original source of the target field is found, the system can determine the original table and the original field. For example, if the target field "Total Sales" appears as an aggregation result in the top-level query, parsing the nested query can confirm that it was originally generated from the "Sales" field in the "Orders" table using a SUM operation.

[0078] Through the above embodiments, by using statement-level tracing, the system can accurately identify the generation logic and data source of target fields, thereby generating more accurate data quality inspection rules and avoiding misjudgments of field sources. Traditional data lineage analysis requires manual inspection of each SQL statement, while using statement-level tracing technology, the system can automatically parse nested queries, significantly improving analysis efficiency and accuracy. For data processing flows containing multi-level nested queries, statement-level tracing can delve into every detail of the data flow, ensuring that even fields that have undergone multiple layers of processing can find their original definitions.

[0079] To determine the data source for each target field, in an optional implementation, step S203 further includes:

[0080] Step S2032: Using the script to be analyzed corresponding to the target field as the current node, check the upstream scripts in sequence according to the upstream and downstream dependencies until the original table and original field are obtained.

[0081] Understandably, node-level tracing is a method of tracing data origins upstream along its lineage. By analyzing the dependencies between nodes in the data processing flow, it identifies which table (origin table) and which field (original field) a specific field (target field) originally originated from. This helps to systematically understand the data generation process and is of great significance for ensuring data quality, compliance, and audit trails.

[0082] In practice, firstly, based on the results of the reverse-order parsing script, the system identifies the key fields requiring data quality checks, i.e., the target fields. These fields may be aggregated, filtered, or otherwise processed in the current script, but their original definitions may reside in any of the multiple upstream scripts. Then, the script containing the target field is considered the current node. Using the upstream-downstream dependency graph constructed in step one, the system can clearly identify the direct upstream nodes of the current node. Next, starting from the current node, the system reverse-engineers the scripts of its direct upstream nodes, repeating the previous step. This process continues until the initial definition of the target field is found, i.e., its corresponding top-most upstream script. Finally, once the target field is located in the top-most upstream script, the system analyzes the FROM clause and field definitions in that script to determine the original table and original field of the target field. For independently synchronized source tables, it is necessary to determine whether they are the final source table based on the quality control boundaries.

[0083] Through the above embodiments, node-level tracing is not limited to a single script or query statement, but spans multiple scripts and data processing nodes, ensuring the comprehensiveness of data lineage analysis. By accurately identifying the original source of the target field, the system can generate more precise data quality inspection rules based on the nature and context of the original field, avoiding rule bias caused by unclear field sources. For scenarios involving multi-step data processing and cleaning, node-level tracing can provide a clear audit trail, helping to meet the needs of data governance, compliance review, and internal audit.

[0084] In order to determine the inspection rules for each target field through dynamic rule propagation, in an optional implementation, step S204 above includes:

[0085] Step S2041: Obtain the data logical relationship between the target field and the corresponding original field from the script to be analyzed corresponding to the target field;

[0086] Understandably, the core of this step lies in understanding and parsing the data processing script, especially the field generation logic in the SQL script. This involves understanding SQL syntax and recognizing the calculation relationships between fields.

[0087] In practice, syntax analysis tools are used to parse the SQL script, identifying the target fields and the calculation processes they participate in, such as field references in operations like SELECT, GROUP BY, and JOIN. Further analysis of the calculation expressions determines which original fields were generated from the target field and through what operations. For example, it was identified that the target field "Total Sales" was generated by summing all records in the "Sales" field using the SUM function.

[0088] Step S2042: Convert the first target rule and data logic relationship into a natural language description to obtain the input data;

[0089] Understandably, expressing technical specifications and logical relationships in human-readable natural language facilitates understanding and processing by large language models. This step involves natural language generation and conversion technologies to ensure seamless communication between machines and humans.

[0090] In practice, the data logical relationships identified in the first step and their primary target rules (such as non-empty requirements, numerical validity checks, etc.) are transformed into clear and complete sentences or paragraphs.

[0091] Step S2043: Input the input data into the pre-trained large language model for processing to obtain the second target rule.

[0092] Understandably, large language models, such as the GPT series, are capable of generating corresponding metadata checking rules based on the input natural language description due to their extensive training data and powerful language understanding capabilities. This involves the model's generalization ability and the application of domain knowledge.

[0093] In practice, the converted natural language description is used as input to invoke a pre-trained large-scale language model, requesting the model to generate or recommend specific metadata inspection rules based on the description. Based on the model's output, the details and expression of the rules are further optimized to ensure that the rules both meet business requirements and are easy to execute in the data processing workflow.

[0094] Through the above embodiments, the application of large-scale models significantly enhances the intelligence level of rule generation, ensuring a high degree of consistency between rules and data logic, and reducing errors that may be caused by manual operations. The automated process significantly improves the efficiency of rule generation, especially when facing large-scale data processing and complex logic, avoiding repetitive manual labor. By flexibly utilizing large-scale models, the system can quickly respond to changes in business logic, adaptively generating new quality inspection rules, improving the flexibility and responsiveness of data quality control. The generated secondary target rules are directly related to business requirements, effectively preventing data quality problems and ensuring data availability and the accuracy of business decisions.

[0095] In order to quantify the data quality of each target field and improve the accuracy of quality supervision, in an optional implementation, step S205 above includes:

[0096] Step S2051: Traverse the target fields in the script to be analyzed, determine the ratio of the number of non-empty fields in the target fields to the total number of fields, and obtain the integrity score;

[0097] It is understandable that the integrity assessment determines whether the data is complete, i.e. whether the fields are not empty, as empty fields will affect the availability of the data.

[0098] In practice, all target fields in the script to be analyzed are traversed, and the ratio of the number of non-empty records in each field to the total number of records is calculated.

[0099] Step S2052: Match the second target rule with each target field in the script to be analyzed to determine the ratio of the number of fields that pass the match to the number of fields that fail the match, and obtain the accuracy score;

[0100] It is understandable that the accuracy assessment determines whether the data meets the preset business rules, such as numerical range and format requirements.

[0101] In practice, the second target rule is matched with the actual value of the target field in the script, and the ratio of the number of fields that pass the rule verification to the number of fields that fail is calculated.

[0102] Step S2053: Calculate the ratio of the number of fields that passed the matching to the number of fields that failed the matching within the preset time period before the current time to obtain the credibility score;

[0103] It is understandable that credibility is the ability of data to maintain accuracy and consistency over a certain period of time, reflecting the stability of data quality over time.

[0104] In practice, the ratio of the number of fields that meet the second objective rule to the total number of fields is calculated within a preset time period (such as the last 3 days) before the current time.

[0105] Step S2054: The integrity score, accuracy score, and credibility score are weighted and calculated to obtain the quality score.

[0106] Understandably, the overall score for data quality is calculated by taking into account completeness, accuracy, and reliability, using a weighted average.

[0107] In practice, the weight of each indicator is preset according to business needs, and the completeness score, accuracy score and credibility score are calculated by weighting.

[0108] Through the above embodiments, automated calculations enable rapid and accurate data quality assessment, reducing the subjectivity and inconsistencies of manual review and improving the efficiency and accuracy of data quality evaluation. Real-time updates to the credibility score allow business users to promptly understand data quality fluctuations, facilitating faster identification and response to data quality issues and enhancing the real-time nature and effectiveness of data governance. The accuracy score, by matching the second objective rule with actual data, achieves refined management of data quality requirements, ensuring data conforms to business logic and rules, and increasing the business value of the data. Weighted calculations comprehensively consider multiple data quality dimensions, providing a holistic perspective on data quality assessment, helping business decision-makers understand the data quality status from multiple angles and make more informed decisions.

[0109] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0110] This application also provides a data management device based on lineage and rule propagation. It should be noted that this data management device based on lineage and rule propagation can be used to execute the data management method based on lineage and rule propagation provided in this application. This device is used to implement the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0111] The following describes the data management device based on lineage and rule propagation provided in the embodiments of this application.

[0112] Figure 3 This is a structural block diagram of a data management device based on lineage and rule propagation according to an embodiment of this application. Figure 3 As shown, the device includes:

[0113] The acquisition unit 10 is used to extract the scripts to be analyzed from the database and analyze the upstream and downstream dependencies between the scripts to be analyzed. It also extracts the metadata and metadata checking rules corresponding to the scripts to be analyzed from the database. The metadata includes field types and primary key constraints.

[0114] The first processing unit 20 is used to parse the script to be analyzed in reverse order according to the upstream and downstream dependencies, extract feature fields according to the syntax characteristics of SQL statements, and obtain multiple target fields.

[0115] The second processing unit 30 is used to perform lineage tracing on each target field using statement-level tracing and node-level tracing to determine the original table and original field corresponding to each target field.

[0116] The third processing unit 40 is used to determine the metadata inspection rules corresponding to each original table and original field based on the metadata, to obtain multiple first target rules, and to determine the metadata inspection rules corresponding to each target field based on the first target rules, to obtain second target rules;

[0117] The calculation unit 50 is used to check the target fields according to the second target rule to determine the quality score of each target field, and to issue an alarm for the target field if the corresponding quality score is less than the first threshold.

[0118] In this embodiment, the acquisition unit extracts the scripts to be analyzed from the database and analyzes the upstream and downstream dependencies between the scripts. It also extracts the metadata and metadata checking rules corresponding to the scripts from the database. The metadata includes field types and primary key constraints. The first processing unit parses the scripts to be analyzed in reverse order according to the upstream and downstream dependencies, extracts feature fields based on the SQL statement syntax characteristics, and obtains multiple target fields. The second processing unit uses statement-level tracing and node-level tracing to perform lineage tracing on each target field to determine the original table and original field corresponding to each target field. The third processing unit determines the metadata checking rules corresponding to each original table and original field based on the metadata, obtaining multiple first target rules. Based on the first target rules, it determines the metadata checking rules corresponding to each target field, obtaining second target rules. The calculation unit checks the target fields according to the second target rules to determine the quality score of each target field. If the corresponding quality score is less than a first threshold, an alarm is issued for the target field. This application introduces statement-level tracing to trace the source of data through hierarchical nesting, and combines it with node-level tracing to trace the source of data through process dependencies, realizing the cross-layer propagation of inspection rules. Compared with the existing technology that intelligently generates default rules according to field type, and the generated data inspection rules cannot be applied to derived fields after field processing, this application improves the complexity of inspection rules for derived fields through lineage tracing and rule propagation, and improves the accuracy of data quality monitoring and management through detailed inspection, thereby solving the problem of inaccurate data quality supervision in the existing technology.

[0119] To determine the upstream and downstream relationships between the scripts to be analyzed, in one optional implementation, the acquisition unit includes:

[0120] The first acquisition module is used to retrieve the node sequence number and node script name corresponding to each script to be analyzed, and obtain the first sequence number and the first script name.

[0121] The first determining module is used to determine the parent node sequence number and parent node script name of the corresponding script to be analyzed based on each first sequence number and first script name, and to obtain the second sequence number and second script name.

[0122] The second determination module is used to determine the upstream and downstream dependencies between each script to be analyzed based on the first sequence number, the second sequence number, the first script name, and the second script name.

[0123] In order to obtain key data features from each script to be analyzed, in one optional implementation, the first processing unit includes:

[0124] The third determination module is used to determine multiple feature fields based on the syntax characteristics of SQL statements, parse each script to be analyzed, and match the feature fields with the scripts to be analyzed to obtain the target fields;

[0125] The second acquisition module is used to match the upstream script of the script to be analyzed based on the upstream and downstream dependencies to obtain the target field when the feature field does not have a corresponding target field in the corresponding script to be analyzed.

[0126] To determine the data source for each target field, in one optional implementation, the second processing unit includes:

[0127] The first processing module is used to trace back inwards by table alias prefixes when there are nested queries in the script to be analyzed to the target field, so as to determine the corresponding original table and original field through statement-level tracing.

[0128] To determine the data source for each target field, in one optional implementation, the second processing unit further includes:

[0129] The second processing module is used to take the script to be analyzed corresponding to the target field as the current node, and check the upstream scripts in sequence according to the upstream and downstream dependencies until the original table and the original field are obtained.

[0130] In order to determine the inspection rules for each target field through dynamic rule propagation, in one optional implementation, the third processing unit mentioned above includes:

[0131] The third acquisition module is used to obtain the data logic relationship between the target field and the corresponding original field from the script to be analyzed corresponding to the target field;

[0132] The third processing module is used to convert the first target rules and data logical relationships into natural language descriptions to obtain input data.

[0133] The fourth processing module is used to process the input data into a pre-trained large language model to obtain the second target rule.

[0134] To quantify the data quality of each target field and improve the accuracy of quality control, in one optional implementation, the above-mentioned calculation unit includes:

[0135] The fourth determination module is used to traverse the target fields in the script to be analyzed, determine the ratio of the number of non-empty fields in the target fields to the total number of fields, and obtain the integrity score;

[0136] The fifth determination module is used to match the second objective rule with each objective field in the script to be analyzed, in order to determine the ratio of the number of fields that pass the match to the number of fields that fail the match, and to obtain an accuracy score;

[0137] The first calculation module is used to calculate the ratio of the number of fields that passed the matching to the number of fields that failed the matching within a preset time period before the current time, and to obtain a credibility score.

[0138] The second calculation module is used to perform weighted calculations on the integrity score, accuracy score, and credibility score to obtain the quality score.

[0139] The aforementioned data management device based on lineage and rule propagation includes a processor and a memory. The acquisition unit, first processing unit, second processing unit, third processing unit, and calculation unit are all stored as program units in the memory. The processor executes these program units stored in the memory to achieve the corresponding functions. All of the above modules are located in the same processor; alternatively, the modules may be located in different processors in any combination.

[0140] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the accuracy of data monitoring can be improved by adjusting kernel parameters.

[0141] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0142] This invention provides a computer-readable storage medium including a stored program, wherein, when the program is executed, it controls the device where the computer-readable storage medium is located to perform the data management method based on lineage and rule propagation.

[0143] Specifically, data management methods based on kinship and rule propagation include:

[0144] Step S201: Extract the scripts to be analyzed from the database and analyze the upstream and downstream dependencies between the scripts to be analyzed. Extract the metadata and metadata checking rules corresponding to the scripts to be analyzed from the database. The metadata includes field types and primary key constraints.

[0145] Step S202: Parse the script to be analyzed in reverse order according to the upstream and downstream dependencies, extract feature fields according to the syntax characteristics of the SQL statement, and obtain multiple target fields;

[0146] Step S203: Use statement-level tracing and node-level tracing to perform lineage tracing for each target field in order to determine the original table and original field corresponding to each target field;

[0147] Step S204: Determine the metadata inspection rules corresponding to each original table and original field based on the metadata to obtain multiple first target rules; determine the metadata inspection rules corresponding to each target field based on the first target rules to obtain second target rules.

[0148] Step S205: Check the target fields according to the second target rule to determine the quality score of each target field. If the corresponding quality score is less than the first threshold, issue an alarm for the target field.

[0149] This invention provides a processor for running a program, wherein the program executes the aforementioned data management method based on lineage and rule propagation.

[0150] Specifically, data management methods based on kinship and rule propagation include:

[0151] Step S201: Extract the scripts to be analyzed from the database and analyze the upstream and downstream dependencies between the scripts to be analyzed. Extract the metadata and metadata checking rules corresponding to the scripts to be analyzed from the database. The metadata includes field types and primary key constraints.

[0152] Step S202: Parse the script to be analyzed in reverse order according to the upstream and downstream dependencies, extract feature fields according to the syntax characteristics of the SQL statement, and obtain multiple target fields;

[0153] Step S203: Use statement-level tracing and node-level tracing to perform lineage tracing for each target field in order to determine the original table and original field corresponding to each target field;

[0154] Step S204: Determine the metadata inspection rules corresponding to each original table and original field based on the metadata to obtain multiple first target rules; determine the metadata inspection rules corresponding to each target field based on the first target rules to obtain second target rules.

[0155] Step S205: Check the target fields according to the second target rule to determine the quality score of each target field. If the corresponding quality score is less than the first threshold, issue an alarm for the target field.

[0156] This invention provides an electronic device, which includes a processor, a memory, and a program stored in the memory and executable on the processor. When the processor executes the program, it performs at least the following steps:

[0157] Step S201: Extract the scripts to be analyzed from the database and analyze the upstream and downstream dependencies between the scripts to be analyzed. Extract the metadata and metadata checking rules corresponding to the scripts to be analyzed from the database. The metadata includes field types and primary key constraints.

[0158] Step S202: Parse the script to be analyzed in reverse order according to the upstream and downstream dependencies, extract feature fields according to the syntax characteristics of the SQL statement, and obtain multiple target fields;

[0159] Step S203: Use statement-level tracing and node-level tracing to perform lineage tracing for each target field in order to determine the original table and original field corresponding to each target field;

[0160] Step S204: Determine the metadata inspection rules corresponding to each original table and original field based on the metadata to obtain multiple first target rules; determine the metadata inspection rules corresponding to each target field based on the first target rules to obtain second target rules.

[0161] Step S205: Check the target fields according to the second target rule to determine the quality score of each target field. If the corresponding quality score is less than the first threshold, issue an alarm for the target field.

[0162] This application also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program having at least the following method steps:

[0163] Step S201: Extract the scripts to be analyzed from the database and analyze the upstream and downstream dependencies between the scripts to be analyzed. Extract the metadata and metadata checking rules corresponding to the scripts to be analyzed from the database. The metadata includes field types and primary key constraints.

[0164] Step S202: Parse the script to be analyzed in reverse order according to the upstream and downstream dependencies, extract feature fields according to the syntax characteristics of the SQL statement, and obtain multiple target fields;

[0165] Step S203: Use statement-level tracing and node-level tracing to perform lineage tracing for each target field in order to determine the original table and original field corresponding to each target field;

[0166] Step S204: Determine the metadata inspection rules corresponding to each original table and original field based on the metadata to obtain multiple first target rules; determine the metadata inspection rules corresponding to each target field based on the first target rules to obtain second target rules.

[0167] Step S205: Check the target fields according to the second target rule to determine the quality score of each target field. If the corresponding quality score is less than the first threshold, issue an alarm for the target field.

[0168] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. They can be implemented using computer-executable program code, and thus can be stored in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those described herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0169] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0170] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0171] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0172] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0173] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0174] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0175] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information by any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0176] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0178] As can be seen from the above description, the embodiments of this application achieve the following technical effects:

[0179] 1) The data management method based on lineage and rule propagation in this application firstly extracts the scripts to be analyzed from the database and analyzes the upstream and downstream dependencies between the scripts. Metadata and metadata checking rules corresponding to the scripts to be analyzed are extracted from the database. The metadata includes field types and primary key constraints. Then, the scripts to be analyzed are parsed in reverse order according to the upstream and downstream dependencies. Feature fields are extracted based on the syntax characteristics of SQL statements to obtain multiple target fields. Next, statement-level and node-level tracing are used to perform lineage tracing on each target field to determine the original table and original field corresponding to each target field. Then, the metadata checking rules corresponding to each original table and original field are determined based on the metadata to obtain multiple first target rules. Based on the first target rules, the metadata checking rules corresponding to each target field are determined to obtain second target rules. Finally, the target fields are checked according to the second target rules to determine the quality score of each target field. If the corresponding quality score is less than a first threshold, an alarm is issued for the target field. This application introduces statement-level tracing to trace the source of data through hierarchical nesting, and combines it with node-level tracing to trace the source of data through process dependencies, realizing the cross-layer propagation of inspection rules. Compared with the existing technology that intelligently generates default rules according to field type, and the generated data inspection rules cannot be applied to derived fields after field processing, this application improves the complexity of inspection rules for derived fields through lineage tracing and rule propagation, and improves the accuracy of data quality monitoring and management through detailed inspection, thereby solving the problem of inaccurate data quality supervision in the existing technology.

[0180] 2) The data management device based on lineage and rule propagation of this application comprises: an acquisition unit extracting scripts to be analyzed from a database and analyzing the upstream and downstream dependencies between scripts; extracting metadata and metadata checking rules corresponding to the scripts from the database, including field types and primary key constraints; a first processing unit parsing the scripts to be analyzed in reverse order according to the upstream and downstream dependencies, extracting feature fields based on the syntax characteristics of SQL statements, and obtaining multiple target fields; a second processing unit performing lineage tracing on each target field using statement-level tracing and node-level tracing to determine the original table and original field corresponding to each target field; a third processing unit determining the metadata checking rules corresponding to each original table and original field based on the metadata, obtaining multiple first target rules, and determining the metadata checking rules corresponding to each target field based on the first target rules, obtaining second target rules; and a calculation unit checking the target fields according to the second target rules to determine the quality score of each target field, and issuing an alarm for each target field if the corresponding quality score is less than a first threshold. This application introduces statement-level tracing to trace the source of data through hierarchical nesting, and combines it with node-level tracing to trace the source of data through process dependencies, realizing the cross-layer propagation of inspection rules. Compared with the existing technology that intelligently generates default rules according to field type, and the generated data inspection rules cannot be applied to derived fields after field processing, this application improves the complexity of inspection rules for derived fields through lineage tracing and rule propagation, and improves the accuracy of data quality monitoring and management through detailed inspection, thereby solving the problem of inaccurate data quality supervision in the existing technology.

[0181] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A data management method based on kinship and rule propagation, characterized in that, include: Extract the scripts to be analyzed from the database and analyze the upstream and downstream dependencies between the scripts to be analyzed. Extract the metadata and metadata inspection rules corresponding to the scripts to be analyzed from the database. The metadata includes field types and primary key constraints. The script to be analyzed is parsed in reverse order based on the upstream and downstream dependencies, and feature fields are extracted based on the syntax characteristics of the SQL statements to obtain multiple target fields; Statement-level and node-level tracing are used to trace the lineage of each target field in order to determine the original table and original field corresponding to each target field. Based on the metadata, determine the metadata inspection rules corresponding to each original table and original field to obtain multiple first target rules. Based on the first target rules, determine the metadata inspection rules corresponding to each target field to obtain second target rules. The target fields are checked according to the second target rule to determine the quality score of each target field. If the corresponding quality score is less than the first threshold, an alarm is issued for the target field.

2. The method according to claim 1, characterized in that, Extracting scripts to be analyzed from the database and analyzing the upstream and downstream dependencies between these scripts, including: Extract the node sequence number and node script name corresponding to each script to be analyzed to obtain the first sequence number and the first script name; Based on each of the first sequence number and the first script name, determine the parent node sequence number and parent node script name corresponding to the script to be analyzed, and obtain the second sequence number and the second script name; The upstream and downstream dependencies between the scripts to be analyzed are determined based on the first sequence number, the second sequence number, the first script name, and the second script name.

3. The method according to claim 1, characterized in that, The script to be analyzed is parsed in reverse order based on the upstream and downstream dependencies, and feature fields are extracted based on the syntax characteristics of the SQL statements to obtain multiple target fields, including: Based on the syntax characteristics of the SQL statement, multiple feature fields are determined, each script to be analyzed is parsed, and the feature fields are matched with the scripts to be analyzed to obtain the target field; If the target field does not exist in the corresponding script to be analyzed for the feature field, the upstream script of the script to be analyzed is matched according to the upstream and downstream dependency relationship to obtain the target field.

4. The method according to claim 1, characterized in that, Statement-level tracing is used to perform lineage tracing on each of the target fields to determine the original table and original field corresponding to each target field, including: If the target field belongs to a nested query in the script to be analyzed, the original table and the original field are determined by tracing back inwards through the table alias prefix.

5. The method according to claim 1, characterized in that, The target fields are traced at the node level to determine their lineage and the corresponding original tables and fields. Using the script to be analyzed corresponding to the target field as the current node, the upstream scripts are checked sequentially according to the upstream and downstream dependencies until the original table and the original field are obtained.

6. The method according to claim 1, characterized in that, Based on the first target rule, the metadata inspection rule corresponding to each target field is determined, resulting in the second target rule, which includes: Obtain the data logic relationship between the target field and the corresponding original field from the script to be analyzed corresponding to the target field; The first target rule and the data logical relationship are converted into a natural language description to obtain the input data; The input data is fed into a pre-trained large language model for processing to obtain the second target rule.

7. The method according to claim 1, characterized in that, The target fields are examined according to the second target rule to determine the quality score of each target field, including: By iterating through the target fields in the script to be analyzed, the ratio of the number of non-empty fields in the target fields to the total number of fields is determined, and an integrity score is obtained. The second target rule is matched with each of the target fields in the script to be analyzed to determine the ratio of the number of fields that pass the match to the number of fields that fail the match, and an accuracy score is obtained. The credibility score is obtained by calculating the ratio of the number of fields that pass the matching to the number of fields that fail the matching within a preset time period before the current time. The quality score is obtained by weighting the integrity score, the accuracy score, and the credibility score.

8. A data management device based on kinship and rule propagation, characterized in that, The device includes: The acquisition unit is used to extract the scripts to be analyzed from the database and analyze the upstream and downstream dependencies between the scripts to be analyzed. It also extracts the metadata and metadata checking rules corresponding to the scripts to be analyzed from the database. The metadata includes field types and primary key constraints. The first processing unit is used to parse the script to be analyzed in reverse order according to the upstream and downstream dependencies, extract feature fields according to the syntax characteristics of SQL statements, and obtain multiple target fields. The second processing unit is used to perform lineage tracing on each of the target fields using statement-level tracing and node-level tracing, so as to determine the original table and original field corresponding to each of the target fields. The third processing unit is used to determine the metadata inspection rules corresponding to each original table and the original field based on the metadata, to obtain a plurality of first target rules, and to determine the metadata inspection rules corresponding to each target field based on the first target rules, to obtain second target rules; The calculation unit is used to check the target fields according to the second target rule to determine the quality score of each target field, and to issue an alarm for the target field if the corresponding quality score is less than a first threshold.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs comprising methods for performing any one of claims 1 to 7.

Images