Data cleaning method, device, equipment, medium and product

Abstract

The application provides a data cleaning method and device, equipment, storage medium and program product, which can be applied to the technical field of financial technology. The method comprises the following steps: performing graph structure processing on operation logs and metadata information to obtain a data blood relationship graph; generating a to-be-cleaned data list based on the cleaning priority of the data table of each node in the data blood relationship graph; performing downstream dependence backtracking on each to-be-cleaned data in the to-be-cleaned data list to obtain an influence range report of the to-be-cleaned data; in response to the influence range report meeting a preset condition, performing a simulated cleaning operation on the to-be-cleaned data in an isolated environment to obtain a simulated verification result of the to-be-cleaned data; and in response to the simulated verification result indicating that the to-be-cleaned data can be executed, adding the to-be-cleaned data to a cleaning list and implementing a data cleaning operation.

Description

Technical Field

[0001] This application relates to the field of financial technology, specifically to a data processing method, and more specifically to a data cleaning method, apparatus, device, storage medium, and program product. Background Technology

[0002] As enterprises deepen their digital transformation, big data platforms have become a critical infrastructure supporting core business operations. Enterprise data volumes are exploding, with massive amounts of redundant data accumulating, significantly increasing storage costs and governance pressures. Data cleaning, as a core component of data lifecycle management, is a necessary means for enterprises to optimize their data asset structure.

[0003] Current data cleanup solutions rely solely on single metrics such as access time to determine data value, failing to accurately reflect the data's business value. This can easily lead to the accidental deletion of high-value business data and the retention of large amounts of low-value redundant data. The risks of cleanup operations are uncontrollable, and data deletion can easily trigger downstream business disruptions. Furthermore, various types of data are isolated, making it difficult to quickly pinpoint the scope of the cleanup operation's impact, and the prioritization of cleanup lacks a scientific basis, resulting in a poor balance between data cleanup and business security. Summary of the Invention

[0004] In view of the above problems, embodiments of this application provide a data cleaning method, apparatus, device, medium, and program product.

[0005] According to a first aspect of this application, a data cleaning method is provided, comprising: performing graph structure processing on operation logs and metadata information to obtain a data lineage graph, wherein the data lineage graph indicates the dependency relationships between the operation logs and metadata information on the data tables; generating a list of data to be cleaned based on the cleaning priority of the data tables at each node in the data lineage graph; tracing the downstream dependencies of each data to be cleaned in the list of data to be cleaned to obtain an impact range report of the data to be cleaned; in response to the impact range report meeting preset conditions, performing a simulated cleaning operation on the data to be cleaned in an isolated environment to obtain a simulation verification result of the data to be cleaned; and in response to the simulation verification result indicating that the data to be cleaned can be cleaned, adding the data to be cleaned to the cleaning list and performing the data cleaning operation.

[0006] According to embodiments of this application, graph structure processing is performed on operation logs and metadata information to obtain a data lineage graph, including: collecting operation logs and metadata information of each operation job in heterogeneous data sources, where the heterogeneous data sources include at least one of data warehouse, data lake, and distributed computing engine; operation jobs include at least one of full data query, data script, distributed batch processing task, and real-time streaming task; and the metadata information indicates at least one of the data structure, data attributes, and data relationships of the metadata contained in each data table in the heterogeneous data sources. The operation logs are then parsed and semantically extracted to obtain the input and output data tables of the operation jobs, and the dependencies between the operation jobs and the input and output data tables are determined. A graph is constructed on the input and output data tables based on the metadata information and dependencies to obtain a data lineage graph with data tables as nodes and dependencies as edges. The node information includes table information and metadata information, and the edges record at least one of the upstream and downstream tables, operation job identifiers, dependency types, and most recent update times of the data tables.

[0007] According to an embodiment of this application, a list of data to be cleaned is generated based on the cleanup priority of the data tables of each node in the data lineage graph. This includes: using the data tables corresponding to each node in the data lineage graph as target tables; obtaining the downstream tables of the target tables based on the edges of the data lineage graph, and the access dependency of each downstream table on the target tables; obtaining the access popularity based on the number of accesses to the target tables indicated by the node information in the data lineage graph and the latest access interval in days; downstream tables including direct downstream tables and indirect downstream tables; determining the cleanup priority of the target tables based on the access dependency, access popularity, and the storage cost coefficient of the target tables to be obtained; and arranging the corresponding target tables in descending order of cleanup priority to obtain the list of data to be cleaned.

[0008] According to an embodiment of this application, downstream dependency tracing is performed on each piece of data to be cleaned in the data list to be cleaned to obtain an impact range report of the data to be cleaned. This includes: selecting a single target table as the data to be cleaned in the data list according to preset batch processing rules; performing a reverse depth traversal in a graph database used to store the data lineage graph to trace all downstream dependent nodes of the corresponding node of the data to be cleaned down to the leaf node, where the leaf node includes at least one of terminal business reports, application interfaces, and front-end applications; and generating an impact range report for the data to be cleaned based on the running status of each downstream dependent node, the related business parties, and the degree of business criticality to the related business parties. The impact range includes at least one of the following: a downstream operation list, an indirect downstream report list, a visual dependency link diagram, a list of affected business parties, and a cleanup risk level.

[0009] According to an embodiment of this application, a simulated cleanup operation is performed on the data to be cleaned in an isolated environment to obtain a simulated verification result of the data to be cleaned. This includes: creating a full snapshot copy of the data to be cleaned, placing the full snapshot copy in a sandbox environment that is consistent with the actual operation environment but isolated from it; determining the cleanup type corresponding to the data to be cleaned, and performing the cleanup operation corresponding to the cleanup type in the sandbox environment. The cleanup type includes at least one of full table deletion, partition cleanup, and cold data archiving; in response to the execution of the cleanup operation, running downstream jobs of the data to be cleaned in the sandbox environment, and monitoring the running status of the downstream jobs in real time, including execution status, running logs, and output data; and generating a simulated verification result of the data to be cleaned based on the running status, wherein the simulated verification result is either executable cleanup or non-executable cleanup.

[0010] According to the embodiments of this application, adding data to be cleaned to a cleanup list and performing data cleanup operations includes: marking data to be cleaned with a cleanup identifier and a cleanup prompt for data that has been verified as executable for cleanup in simulation, with the cleanup prompt corresponding to the cleanup type, to obtain cleanable objects, which are single data tables or a portion of data in a single data table; adding cleanable objects to the cleanup list corresponding to the cleanup prompt, and assigning corresponding cleanup execution methods to cleanable objects according to preset risk classification and control rules; performing cleanup operations on cleanable objects according to the execution methods, and recording a full-process audit log including operation time, cleanup objects, execution methods, and data size before cleanup; and updating metadata information and changing nodes and edges in the data lineage graph in response to the completion of the cleanup operation.

[0011] According to embodiments of this application, obtaining the access dependency of downstream tables on a target table includes: identifying the business level corresponding to each downstream table, where the business level includes at least one of core business level, important business level, general business level, and test business level; assigning a business weight coefficient corresponding to the business level to each downstream table, where the business weight coefficient indicates the importance of the business corresponding to the downstream table; calculating the sum of the business weight coefficients of all downstream tables of the target table; and using the ratio of the sum of the business weight coefficients to the maximum dependency threshold of the corresponding business domain as the access dependency coefficient of the target table, where the access dependency coefficient indicates the access dependency of all downstream tables on the target table, and the maximum dependency threshold of the corresponding business domain is either the historical maximum number of downstream dependency nodes of all data tables in the business domain where the target table is located (pre-statistically counted) or a preset standardized dependency upper limit.

[0012] According to an embodiment of this application, the cleanup priority of a target table is determined based on access dependency, access popularity, and the storage cost coefficient of the target table to be acquired. This includes: normalizing the target table's storage capacity, storage medium type, and number of replicas to obtain its storage cost coefficient; multiplying the ratio of the number of accesses to the preset maximum access threshold of the corresponding business domain by the ratio of the preset statistical period duration to the latest access interval in days to obtain the target table's access popularity; obtaining the target table's target isolation coefficient based on access popularity and access dependency, which indicates the target table's isolation level; using the product of the target isolation coefficient and the storage cost coefficient as the cleanup priority score; and mapping the cleanup priority score to obtain the cleanup priority.

[0013] According to a second aspect of this application, a data cleaning apparatus is provided, comprising: a spectrum construction module for performing graph structure processing on operation logs and metadata information to obtain a data lineage graph, wherein the data lineage graph indicates the dependency relationships between the operation logs and metadata information on the data tables; a list generation module for generating a list of data to be cleaned based on the cleaning priority of the data tables at each node in the data lineage graph; a list evaluation module for tracing the downstream dependencies of each data to be cleaned in the list of data to be cleaned to obtain an impact range report of the data to be cleaned; a simulation cleaning module for performing a simulated cleaning operation on the data to be cleaned in an isolated environment in response to the impact range report meeting preset conditions, thereby obtaining a simulation verification result of the data to be cleaned; and a cleaning implementation module for adding the data to be cleaned to the cleaning list and performing the data cleaning operation in response to the simulation verification result indicating that the data to be cleaned is ready for cleaning.

[0014] According to a third aspect of this application, an electronic device is provided, comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0015] According to a fourth aspect of this application, a computer-readable storage medium is also provided, on which a computer program or instructions are stored, wherein the computer program or instructions, when executed by a processor, implement the steps of the above-described method.

[0016] According to a fifth aspect of this application, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method. Attached Figure Description

[0017] The above-mentioned contents, other objects, features and advantages of this application will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0018] Figure 1 The illustrations depict application scenarios of data cleaning methods, apparatus, devices, media, and program products according to embodiments of this application.

[0019] Figure 2 A flowchart illustrating a data cleaning method according to an embodiment of this application is shown schematically;

[0020] Figure 3 This illustration schematically shows a data lineage map obtained in a data cleaning method according to an embodiment of this application;

[0021] Figure 4 This illustration shows the data flow diagram for generating a list of data to be cleaned in the data cleaning method according to an embodiment of this application;

[0022] Figure 5 This illustration shows the data flow diagram of obtaining the impact range report of the data to be cleaned in the data cleaning method according to an embodiment of this application;

[0023] Figure 6 The illustration shows a schematic diagram of the simulation verification results obtained in the data cleaning method according to an embodiment of this application;

[0024] Figure 7 The illustration shows a schematic diagram of the data cleaning operation performed in the data cleaning method according to an embodiment of this application;

[0025] Figure 8 This schematic diagram illustrates a structural block diagram of a data cleaning apparatus according to an embodiment of the present application;

[0026] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a data cleaning method according to an embodiment of this application. Detailed Implementation

[0027] The embodiments of this application will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of this application. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of this application for ease of explanation. However, it will be apparent that one or more embodiments may be implemented without these specific details. Furthermore, descriptions of well-known structures and technologies are omitted in the following description to avoid unnecessarily obscuring the concepts of this application.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0031] In the technical solution of this application, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entry points for users to choose to authorize or refuse.

[0032] Figure 1 The diagram illustrates an application scenario of data cleaning according to an embodiment of this application. For example... Figure 1 As shown, application scenario 100 according to an embodiment of this application may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables. For example, a user can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 through the network 104 to receive or send information, etc.

[0033] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be electronic devices such as smartphones, wearable devices, personal computers, intelligent voice interaction devices, smart home appliances, intelligent vehicles, in-vehicle terminals, aircraft, unmanned vending terminals, and extended reality devices. Extended reality devices can include virtual reality devices, augmented reality devices, and mixed reality devices. A client application for the target application can be installed and run on the terminal devices. This target application can include, but is not limited to, financial transaction applications, payment applications, shopping applications, web browser applications, search applications, instant messaging tools, email clients, and social media platform software (these are just examples). Furthermore, this application embodiment does not limit the form of the target application, and it can include, but is not limited to, applications, mini-programs, etc., installed on the terminal devices, and can also be in the form of web pages.

[0034] Server 105 can be a server providing various services, such as a backend management server supporting websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process received user requests and other data, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services such as cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks, and basic cloud computing services such as big data. The server can be the backend server of the aforementioned target application, used to provide backend services to the clients of the target application.

[0035] It should be noted that the data cleaning method provided in this application embodiment can generally be executed by server 105 and / or terminal devices 101-103. Accordingly, the data cleaning device provided in this application embodiment can generally be installed in server 105 and / or terminal devices 101-103.

[0036] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0037] Figure 2 A flowchart illustrating a data cleaning method according to an embodiment of this application is shown schematically. Figure 2 As shown, the data cleaning method 200 according to the embodiments of this application may include steps S210 to S240.

[0038] In step S210, the operation log and metadata information are processed into a graph structure to obtain a data lineage graph, which indicates the dependency relationship between the operation log and metadata information on the data tables.

[0039] As an example, operation logs refer to structured or semi-structured log files generated during the execution of various data processing tasks to record data flow and computation processes. They are the core foundation for parsing dependencies between data tables and constructing data lineage graphs. Metadata information is data that describes data; it refers to structured data in a big data platform used to describe the attributes, structure, and management information of data tables and their data. It is the core foundation for constructing data lineage graph nodes and calculating data value quantification indicators. Graph structure processing is the computer processing process of converting unstructured operation logs and metadata information into a graph data structure with nodes and edges as basic units. A data lineage graph is a directed graph constructed with data tables as nodes and dependencies between data as edges, which can completely record the entire chain of data flow relationships from generation to destruction. Dependencies refer to the upstream and downstream relationships formed between data processing operations and data tables, and between data tables in a big data platform. For example, the execution of an operation must depend on one or more input data tables, the generation of a data table must depend on the execution result of an upstream operation, and the existence of a downstream data table must depend on the data input of an upstream data table.

[0040] In step S220, a list of data to be cleaned is generated based on the cleaning priority of the data tables of each node in the data lineage graph.

[0041] As an example, cleanup priority refers to a quantitative indicator used to measure the urgency and rationality of cleaning up a data table; for instance, a higher value indicates a higher priority for cleanup. The list of data to be cleaned is a structured list of preliminary candidate cleanup objects generated based on the cleanup priority.

[0042] In step S230, downstream dependency tracing is performed on each piece of data to be cleaned in the data list to obtain a report on the scope of impact of the data to be cleaned.

[0043] As an example, downstream dependency tracing refers to an automated process that starts with the data to be cleaned and traverses the entire dependency chain in the reverse direction of the data lineage graph to locate all downstream nodes that directly or indirectly depend on the data to be cleaned. The impact report can be a document automatically generated by a big data platform, describing all downstream business and system components that may be affected by cleaning up a certain data table.

[0044] In step S240, in response to the impact range report meeting the preset conditions, a simulated cleaning operation is performed on the data to be cleaned in the isolated environment to obtain the simulation verification results of the data to be cleaned.

[0045] As an example, the preset conditions are a pre-configured set of automated rules used to determine whether the impact scope report has passed business approval. These are prerequisites for triggering subsequent simulated cleanup operations. The judgment process is automatically completed by the corresponding system of the big data platform without manual intervention. An isolated environment refers to an independent computing environment physically or logically isolated from the production environment, used to execute test operations without affecting the production system using the big data platform. The production system is the system used by the enterprise for business processing. Simulated cleanup operations refer to performing cleanup actions identical to those in the production environment on the data to be cleaned that has passed business approval in a sandbox environment completely isolated from the production environment, triggering a simulation verification process for all downstream dependent jobs. The simulation verification result is a conclusion drawn from the downstream job operation status after simulating the cleanup operation in the isolated environment, determining whether formal cleanup is possible.

[0046] In step S250, in response to the simulation verification result indicating that the data to be cleaned can be cleaned, the data to be cleaned is added to the cleanup list and the data cleanup operation is performed.

[0047] As an example, the cleanup list is a final structured list automatically generated after both business approval and simulated cleanup operation verification. It can be directly executed in the production environment and serves as the basis for formal data cleanup operations. Data cleanup operations refer to the entire process of performing final physical or logical deletion and archiving operations on verified data tables or data in the production environment based on the cleanup list, while simultaneously updating audit records and metadata information.

[0048] In one specific embodiment, taking a financial institution's big data platform as an example, the big data platform can be simply referred to as the platform. This platform carries core businesses such as lending, credit cards, and risk management. During data cleaning, the operation logs and metadata information collected from data sources such as data warehouses, computing clusters, and real-time clusters are first processed using a graph structure to construct a complete data lineage graph containing tens of thousands of data tables. Based on the lineage graph, the cleaning priority of each data table is calculated, generating a list of over three thousand candidate data tables to be cleaned. These candidate data tables are the data to be cleaned. Downstream dependencies of each data table in the cleanup list are traced, generating an impact scope report and pushing it to the relevant responsible parties. After the report is automatically approved by the system, the cleanup operation is simulated in an isolated sandbox environment, triggering downstream operations. Finally, based on the simulation verification results, over one thousand verified data tables are added to the cleanup list and formal cleanup is implemented.

[0049] Based on this, graph structure processing technology is used to integrate the scattered dynamic data flow operation logs and static attribute metadata information into a unified structured data lineage graph. This breaks down information silos between data tables, restores the full-link dependencies of data tables, and improves the efficiency and accuracy of dependency acquisition. The data lineage graph is used to determine cleanup priorities, improving the objectivity of data cleanup. Downstream dependency tracing technology is used to obtain the full-link impact range of the data to be cleaned, thereby improving the comprehensiveness and accuracy of assessing the impact range of the data to be cleaned. Compliance verification is performed using preset conditions to achieve pre-emptive prediction of cleanup impact, transforming post-event troubleshooting into pre-event risk identification, and intercepting cleanup operations that may affect core business operations from the business level. Simulated cleanup operations are first executed in an isolated environment to verify security. Formal cleanup is only implemented after successful verification, verifying the technical feasibility of cleanup operations in advance without affecting the production system, thereby reducing the risk of business interruption caused by accidental data deletion from a technical perspective. This improves the automation level of data cleanup, enhancing cleanup efficiency and accuracy.

[0050] Figure 3 The illustration shows a schematic diagram of a data lineage map obtained in a data cleaning method according to an embodiment of this application.

[0051] exist Figure 3 In the illustrated embodiment, graph structure processing is performed on the operation logs and metadata information to obtain a data lineage graph. This includes: collecting operation logs and metadata information of each operation job in heterogeneous data sources, where the heterogeneous data sources include at least one of a data warehouse, a data lake, and a distributed computing engine; and where the operation jobs include at least one of a full data query, a data script, a distributed batch processing task, and a real-time streaming task. The metadata information indicates at least one of the data structure, data attributes, and data relationships of the metadata contained in each data table in the heterogeneous data sources. The operation logs are then parsed and semantically extracted to obtain the input and output data tables of the operation jobs, and the dependencies between the operation jobs and the input and output data tables are determined. Based on the metadata information and dependencies, a graph is constructed for the input and output data tables to obtain a data lineage graph with data tables as nodes and dependencies as edges. The node information includes table information and metadata information, and the edges are used to record at least one of the upstream and downstream tables, operation job identifiers, dependency types, and the most recent update time of the data tables.

[0052] As an example, heterogeneous data sources refer to a collection of data sources using different data models, storage formats, and access interfaces. Operation jobs are a collective term for various data processing tasks executed on a big data platform. Metadata information refers to data describing a data table or its data, including information such as the structure, attributes, relationships, and storage location of the data table or its data. Operation jobs can be abbreviated as jobs. Dependencies refer to the relationship where the execution or existence of one job or data table depends on another job or data table. Node information refers to all attribute information stored in the nodes of the data lineage graph. Edges correspond to edge information, which consists of all attribute information stored in the edges of the data lineage graph, used to describe the relationships between nodes.

[0053] In a specific example, the operation logs and metadata information of each operation job in the heterogeneous data source are first collected. Since the operation log of a single operation job can involve multiple data tables, and the metadata of the data tables and their data will change accordingly after the operation job, a metadata information collection task can be triggered periodically to monitor and collect the operation logs of various operation jobs in real time. Heterogeneous data sources include data warehouses, data lakes, distributed computing engines, real-time computing engines, etc. Operation jobs include full database queries, data script execution, distributed batch processing tasks, real-time streaming tasks, etc. Metadata information includes the data structure of each data table, such as field names, field types, partitioning rules, storage formats, etc.; data attributes, such as the database to which it belongs, file size, last modification time, last access time, data ownership, etc.; and data relationships, such as foreign key associations, view mappings, etc. The collected operation logs are then parsed and semantically extracted to identify the input and output data tables of each operation job. The input data table is the upstream table of the output data table, and the output data table is the downstream table of the input data table. Finally, the dependencies between the operation job and the input and output data tables are determined. Finally, a graph is constructed based on metadata information and dependencies to obtain a data lineage graph with data tables as nodes and dependencies as edges. The node information includes table information and corresponding metadata information, while the edge information records upstream tables, downstream tables, operation job identifiers, dependency types, and the most recent update time. The dependency type can be direct or indirect.

[0054] Based on this, it covers three core heterogeneous data sources: data warehouses, data lakes, and distributed computing engines. It also supports log collection for all types of jobs, including full data queries, data scripts, distributed batch processing tasks, and real-time streaming tasks. It collects complete metadata information, including data structure, data attributes, and data relationships, avoiding dependency omissions due to incomplete data source coverage. This ensures the generated lineage graph covers the entire data flow chain of the enterprise big data platform. All collected operation logs undergo unified format parsing and semantic extraction, automatically identifying the input and output data tables for each operation job. It accurately determines the relationships between the job and the input and output tables, achieving automated dependency extraction, improving dependency extraction efficiency, reducing dependency error rates, and enhancing the consistency between dependencies and the actual data flow process. A standardized directed graph structure is constructed with data tables as nodes and dependencies as edges. The attribute information of nodes and edges reflects data flow information, improving the standardization and usability of the data lineage graph itself. It can accurately and completely reconstruct the entire link flow process between data tables, thereby improving the efficiency and accuracy of data cleaning.

[0055] Figure 4 This illustration shows the data flow diagram for generating a list of data to be cleaned in the data cleaning method according to an embodiment of this application.

[0056] exist Figure 4 In the illustrated embodiment, a list of data to be cleaned is generated based on the cleanup priority of the data tables of each node in the data lineage graph. This includes: using the data tables corresponding to each node in the data lineage graph as target tables; obtaining the downstream tables of the target tables based on the edges of the data lineage graph, as well as the access dependency of each downstream table on the target tables; obtaining the access popularity based on the number of accesses to the target tables indicated by the node information in the data lineage graph and the latest access interval in days; downstream tables include direct downstream tables and indirect downstream tables; determining the cleanup priority of the target tables based on the access dependency, access popularity, and the storage cost coefficient of the target tables to be obtained; and arranging the corresponding target tables in descending order of cleanup priority to obtain the list of data to be cleaned.

[0057] As an example, the target table is the data table node in this data lineage graph that is used for cleanup priority evaluation. Access dependency refers to a quantitative indicator measuring how closely the target table is dependent on its downstream tables. Access popularity refers to a quantitative indicator measuring how frequently the target table is accessed. Storage cost coefficient refers to a normalized quantitative indicator measuring the storage cost of the target table. Last access interval in days refers to the difference in days between the current system time and the timestamp of the last access to the target table.

[0058] In a specific example, all the data tables corresponding to the more than 20,000 nodes in the data lineage graph can be used as target tables. For each target table, all its direct and indirect downstream tables, as well as the access dependency of each downstream table on the target table, are obtained based on the edge information of the data lineage graph. Simultaneously, the access popularity of the target table is calculated based on the access frequency and the latest access interval in days recorded in the node information of the data lineage graph over the past month, half month, or quarter. Then, the cleanup priority of the target table is determined based on the access dependency, access popularity, and the storage cost coefficient of the target table to be acquired. Finally, among all target tables arranged in descending order of cleanup priority, more than 3,000 data tables with a priority higher than a preset threshold are selected, or a preset number of data tables are selected from all target tables arranged in descending order, as a standardized list of data to be cleaned.

[0059] Thus, based on the edge information of the data lineage graph, all direct and indirect downstream tables of the target table are obtained, and the access dependence of downstream tables on the target table is quantitatively calculated. This incorporates the business-related value of the data into the cleanup evaluation system, achieving a shift from focusing solely on the data's access attributes to simultaneously considering its business contribution value. This ensures that the cleanup priority assessment truly reflects the actual role of data in the business chain, fundamentally solving the technical problem of high-value business data being mistakenly cleaned up. Access popularity is calculated based on the number of accesses and the number of days since the last access, taking into account both access frequency and recent access time. This allows for a more accurate determination of the access activity of data tables, distinguishing data in different idle states, and avoiding the misjudgment of valuable data that was frequently used but is currently temporarily idle as redundant data, thus improving the accuracy of data value assessment. The storage cost coefficient is introduced as a third dimension for data value assessment, working together with access dependence and access popularity to determine cleanup priority. This achieves a comprehensive balance between business value and storage cost, prioritizing the cleanup of data tables with low data value and high storage costs, releasing storage resources in a timely manner, improving storage resource utilization efficiency, increasing data cleanup efficiency, and reducing cleanup risks.

[0060] According to an embodiment of this application, obtaining the access dependency of downstream tables on a target table includes: identifying the business level corresponding to each downstream table, where the business level includes at least one of core business level, important business level, general business level, and test business level; assigning a business weight coefficient corresponding to the business level to each downstream table, where the business weight coefficient indicates the importance of the business corresponding to the downstream table; calculating the sum of the business weight coefficients of all downstream tables of the target table; and using the ratio of the sum of the business weight coefficients to the maximum dependency threshold of the corresponding business domain as the access dependency coefficient of the target table, where the access dependency coefficient indicates the access dependency of all downstream tables on the target table, and the maximum dependency threshold of the corresponding business domain is either the historical maximum number of downstream dependency nodes of all data tables in the business domain where the target table is located (pre-statistically counted) or a preset standardized dependency upper limit.

[0061] As an example, a business level refers to a tier based on the business importance of the downstream table. A business weight coefficient is a numerical value used to quantify business importance, corresponding one-to-one with the business level. A business domain refers to a logical partitioning of the big data platform according to the enterprise's business functions; data tables involved in different business operations reside in different business domains. The maximum dependency threshold is a pre-set benchmark value used to normalize the calculation of access dependencies.

[0062] As an example, the first step is to identify the business level corresponding to each downstream table of the target table. Business levels include core business level, important business level, general business level, and testing business level. Then, a business weight coefficient corresponding to the business level is assigned to each downstream table. This weight coefficient indicates the importance of the business corresponding to that downstream table. For example, the core business level corresponds to a weight coefficient of 1.0, the important business level to 0.5, the general business level to 0.2, and the testing business level to 0.1. Finally, the sum of the business weight coefficients of all downstream tables of the target table is calculated. Finally, the ratio of the sum of business weight coefficients to the maximum dependency threshold of the corresponding business domain is used as the access dependency coefficient of the target table. The access dependency coefficient indicates the degree of access dependency of all downstream tables on the target table. Its calculation formula can be expressed as: Access dependency coefficient = (Σ downstream dependency weight) / (maximum dependency threshold × 1.0). The maximum dependency threshold is the maximum possible dependency number, that is, the maximum number of downstream tables that may depend on the target table. This number can be pre-calculated based on the historical maximum number of downstream dependency nodes, or it can be a standardized dependency upper limit preset by the staff. For example, if the maximum number of downstream dependency nodes of the target table in the history of this business domain is one thousand, then the maximum dependency threshold can be set to one thousand, or it can be the upper limit value of one thousand two thousand as perceived by the staff, etc.

[0063] Based on this, business importance is quantified into calculable weight coefficients, so that dependency calculation is no longer a simple count of nodes, but a true reflection of the actual supporting value of data to the business. Data depended on by core businesses will be calculated with a higher dependency level, thus receiving a lower cleanup priority in subsequent evaluations, fundamentally avoiding the risk of core business data being mistakenly cleaned up. A normalized calculation method based on business domains is adopted, calculating the ratio of the target table's weight to the maximum dependency threshold of the business domain to which the table belongs, resulting in a standardized access dependency level. This eliminates the impact of differences in data link complexity across different business domains on the evaluation results, making access dependency level a standardized indicator. All data tables within the same business domain are evaluated using a unified benchmark, ensuring the fairness and accuracy of the evaluation results. This enables a unified value ranking of data from different business domains across the entire platform, thereby improving the accuracy and scientific rigor of determining data cleanup priorities.

[0064] According to an embodiment of this application, the cleanup priority of a target table is determined based on access dependency, access popularity, and the storage cost coefficient of the target table to be acquired. This includes: normalizing the target table's storage capacity, storage medium type, and number of replicas to obtain its storage cost coefficient; multiplying the ratio of the number of accesses to the preset maximum access threshold of the corresponding business domain by the ratio of the preset statistical period duration to the latest access interval in days to obtain the target table's access popularity; obtaining the target table's target isolation coefficient based on access popularity and access dependency, which indicates the target table's isolation level; using the product of the target isolation coefficient and the storage cost coefficient as the cleanup priority score; and mapping the cleanup priority score to obtain the cleanup priority.

[0065] As an example, normalization refers to the process of converting values ​​with different dimensions into values ​​with a unified dimension to narrow the numerical range and facilitate the calculation and comparison of storage cost coefficients. The target isolation coefficient is a quantitative indicator that measures the degree of isolation of a target table within the business chain. The cleanup priority score is a specific value obtained through quantitative calculation to determine the cleanup priority. Mapping refers to the process of converting the cleanup priority score into the corresponding cleanup priority level. The preset maximum access threshold refers to the baseline value of the maximum allowed number of accesses to a data table within a certain business domain within a preset statistical period. The preset statistical period duration is a pre-set time period length used for counting accesses.

[0066] In a more specific example, the storage cost coefficient of the target table is first obtained by processing it according to a preset normalization rule based on the target table's storage capacity, storage medium type, and number of replicas, thereby eliminating the numerical magnitude differences between indicators of different dimensions. Then, the ratio of the number of accesses to the target table within a preset statistical period to the preset maximum access threshold of the corresponding business domain is multiplied by the ratio of the preset statistical period duration to the latest access interval in days, yielding the access popularity of the target table. Next, the target isolation coefficient of the target table is calculated based on the access popularity and access dependency. The target isolation coefficient = (1 - access dependency) × (1 - access popularity), indicating the degree of isolation of the target table in the business chain. Then, the product of the target isolation coefficient and the storage cost coefficient is used as the cleanup priority score of the target table. Finally, the cleanup priority score is mapped to a corresponding cleanup priority level, such as high, medium, or low. By mapping the cleanup priority scores, continuous numerical ranges are converted into discrete cleanup priority levels with clear business meanings. This transforms abstract quantitative scores into intuitive and executable priority instructions, enabling automatic allocation of cleanup resources and execution time based on priority levels, thereby improving the automation and efficiency of the cleanup process.

[0067] Based on this, the introduction of a target isolation coefficient amplifies the combined effect of low dependency and low access frequency. The lower the data dependency and access frequency, the higher the target isolation coefficient, and the more prioritized it should be for cleanup. This allows for accurate identification of isolated redundant data in the business chain, ensuring that zombie data with no dependencies, no access, and high storage requirements receives the highest cleanup priority. This improves the accuracy of redundant data identification and ensures that limited cleanup resources are used to clean up the least valuable data. Setting the cleanup priority score as the product of the target isolation coefficient and the storage cost coefficient achieves objectivity and standardization in cleanup priority calculation, and the evaluation process is traceable and verifiable, improving the rigor and standardization of data cleanup on the big data platform.

[0068] Figure 5 This illustration shows the data flow diagram of obtaining the impact range report of the data to be cleaned in the data cleaning method according to an embodiment of this application;

[0069] exist Figure 5In the illustrated embodiment, downstream dependency tracing is performed on each piece of data to be cleaned in the data list to obtain an impact range report for the data to be cleaned. This includes: selecting a single target table as the data to be cleaned in the data list according to preset batch processing rules; performing a reverse depth-first traversal in the graph database used to store the data lineage graph to trace all downstream dependent nodes of the corresponding node of the data to be cleaned down to the leaf node, where the leaf node includes at least one of terminal business reports, application interfaces, and front-end applications; and generating an impact range report for the data to be cleaned based on the running status of each downstream dependent node, the associated business parties, and the degree of business criticality to the associated business parties. The impact range includes at least one of the following: a downstream operation list, an indirect downstream report list, a visual dependency link diagram, a list of affected business parties, and a cleanup risk level.

[0070] As an example, batch processing rules refer to pre-defined rules used to control the number of data items to be cleaned simultaneously. Reverse depth-first traversal refers to a graph traversal algorithm that starts from the target node and performs a depth-first search along the dependency relationships in the reverse direction. A leaf node is a terminal node in the data lineage graph that has no downstream dependent nodes. Running status indicates whether a downstream dependent node is currently active. Business criticality measures the importance of downstream dependent nodes to the business. Cleanup risk level measures the degree of business risk that cleaning the target table may pose.

[0071] In a more specific example, a list of data to be cleaned is obtained. This list is essentially composed of data tables, which are also the target tables with the highest cleanup priority. Then, preset batch processing rules are determined, such as processing 200 data tables simultaneously. 200 target tables are selected from the list of data to be cleaned sequentially. A reverse depth-first traversal is performed in the graph database storing the data lineage, tracing the entire chain to all downstream dependent nodes of the node corresponding to the data to be cleaned, down to the leaf nodes. Leaf nodes include terminal business reports, application interfaces, and front-end applications. Then, the running status of each downstream dependent node is analyzed. For example, the last execution time of the data table corresponding to the downstream dependent node is used to determine whether the data table is actively running. The associated business parties, responsible persons (if applicable), and the business criticality level of the data table corresponding to the downstream dependent node are obtained from the metadata information. This business criticality level can be core, important, or general. Based on the aforementioned downstream dependency node status, a list of direct downstream jobs (including job ID, job name, and responsible person), a list of indirect downstream reports (including report name, business domain, and responsible person), a complete dependency chain path diagram with a visual structure, a list of affected business parties (including related departments and contacts), and risk assessment levels (high, medium, and low), also known as cleanup risk levels, are generated. Then, the downstream job list, indirect downstream report list, visual dependency chain diagram, list of affected business parties, and cleanup risk levels are compiled into an impact scope report. This impact scope report can be pushed to all related business parties to collect feedback and determine whether it meets preset conditions. These preset conditions can be formulated based on the company's specific operational status and can be a fully automated judgment process or a manual approval process; no restrictions are placed here. For example, if all relevant parties agree to the cleanup, it meets the preset conditions and is marked as "can be cleaned up." If any relevant party raises an objection, it is marked as "must be retained" and the reason is recorded. If the feedback is unclear, it is marked as "needs further confirmation." If both "must be retained" and "needs further confirmation" exist, it means that the preset conditions are not met, and the cleanup process for that data table is suspended.

[0072] Based on this, data within the data list to be cleaned is analyzed one by one according to preset batch processing rules, which can reasonably control the amount of data processed in a single batch and improve the stability of the entire analysis process. Utilizing a graph database storing the data lineage map to perform a reverse depth-first traversal, it is possible to complete full-level tracing along the data dependency chain, until reaching leaf nodes such as terminal business reports, application interfaces, and front-end applications, completely tracing all direct and indirect downstream related objects related to the target table. This avoids the omission of related business due to incomplete link tracing, comprehensively assesses the true impact range of the data, and improves the accuracy of data cleanup. Combining the operating status of downstream dependent nodes, their associated business parties, and the degree of business criticality, a multi-dimensional comprehensive judgment is made to generate a standardized impact range report that includes a downstream operation list, an indirect downstream report list, a visualized dependency chain, a list of affected business parties, and a cleanup risk level. This comprehensively displays the list information and logical relationships obtained from downstream dependency tracing, improving the clarity of risk classification and the efficiency of judging data dependencies.

[0073] Figure 6 The illustration shows a schematic diagram of the simulation verification results obtained in the data cleaning method according to an embodiment of this application.

[0074] exist Figure 6 In the illustrated embodiment, a simulated cleanup operation is performed on the data to be cleaned in an isolated environment to obtain a simulated verification result of the data to be cleaned. This includes: creating a full snapshot copy of the data to be cleaned, placing the full snapshot copy in a sandbox environment that is consistent with the actual operation environment but isolated from it; determining the cleanup type corresponding to the data to be cleaned, and performing the cleanup operation corresponding to the cleanup type in the sandbox environment. The cleanup type includes at least one of full table deletion, partition cleanup, and cold data archiving; in response to the execution of the cleanup operation, running downstream jobs of the data to be cleaned in the sandbox environment, and monitoring the running status of the downstream jobs in real time, including execution status, running logs, and output data; and generating a simulated verification result of the data to be cleaned based on the running status, whereby the simulated verification result indicates whether the cleanup is executable or not.

[0075] As an example, a sandbox environment can refer to a test runtime environment configured identically to the actual operating environment and logically isolated from it. A full snapshot copy can be a data file obtained by completely replicating the data to be cleaned, ensuring that the simulated data is completely identical to the original data in terms of content and structure. Cleaning type refers to the data processing method set for different data forms of data to be cleaned, mainly including three categories: full table deletion, partition cleanup, and cold data archiving. Full table deletion refers to removing the entire data table and its storage content as a whole. Partition cleanup refers to deleting only the data in a specified partition of the data table, retaining the content of other partitions of the data table. Cold data archiving can refer to migrating historical data that has been rarely accessed for a long time to a low-cost storage medium for offline retention, and then deleting the corresponding data in the original environment. Downstream jobs can be various data processing tasks that depend on the current data to be cleaned. Running status is used to characterize the overall running status of downstream jobs, specifically covering job execution status, system running logs, and final output data.

[0076] In a more specific example, the first data table to be verified is selected from the data to be cleaned. A full snapshot copy of this data table is created in a sandbox environment configured identically to and isolated from the actual operational environment. The cleanup type corresponding to the current data table to be verified is determined, and the cleanup operation is simulated in the sandbox environment. For example, if the cleanup type is full table deletion, a preset DROP TABLE operation is executed in the sandbox environment to delete the entire table. If the cleanup type is partition cleanup, a preset ALTER TABLE DROP PARTITION operation is executed in the sandbox environment to delete the data in the partitions, retaining the content of the other partitions. If the cleanup type is cold data archiving, the data is migrated in the sandbox environment and then the original data is deleted. After completing the simulated cleanup operation, all downstream jobs corresponding to this data table are started by calling the job scheduling interface or replaying historical job parameters. Simultaneously, the success of downstream job completion, the occurrence of error logs during downstream job execution, and the presence of empty results or abnormal data volumes in the output data of downstream jobs are monitored in real time. By collecting execution status and result data from downstream operations, a comprehensive assessment of the collected operational statuses is conducted. If all downstream operations run normally and produce normal data, the verification passes and is marked as "executable cleanup." If any operation fails or produces abnormal output, the verification fails, is marked as "unexecutable cleanup," and the reason for failure and error log are recorded. After obtaining the simulation verification results of the data to be cleaned, a verification report for the data table is generated, including: whether the verification result passed, a summary of the execution log, failure details and error stack, and remediation suggestions.

[0077] Based on this, a full snapshot copy of the data to be cleaned is created and deployed to an independent sandbox environment. This replicates the real data cleanup environment, allowing testing to be conducted under completely isolated conditions, thus avoiding the risk of simulation operations interfering with the formal cleanup environment. Differentiating between various cleanup types—full table deletion, partition cleanup, and cold data archiving—and matching corresponding operations, it adapts to different data processing needs, ensuring consistency between the simulation process and actual cleanup operations, and enhancing the reference value of the verification results. After cleanup, all downstream jobs are actively run, and the execution status, logs, and output data are comprehensively monitored. This fully reproduces the business chain operation after data cleanup, avoiding potential anomalies. The simulation verification results are judged based on the downstream job operation status, accurately identifying the feasibility of the cleanup operation, proactively identifying and avoiding potential risks and failures that may arise during formal cleanup, and effectively improving the security and reliability of data cleanup work.

[0078] Figure 7 The illustration shows a schematic diagram of the data cleaning operation performed in the data cleaning method according to an embodiment of this application.

[0079] exist Figure 7 In the illustrated embodiment, the data to be cleaned is added to the cleanup list, and the data cleanup operation is performed, including: marking the data to be cleaned with a cleanup identifier and a cleanup prompt for the data that the simulation verification result shows is executable for cleanup, the cleanup prompt corresponds to the cleanup type, and the cleanupable object is a single data table or part of the data in a single data table; adding the cleanupable object to the cleanup list corresponding to the cleanup prompt, and assigning a corresponding cleanup execution method to the cleanupable object according to the preset risk classification and control rules; performing the cleanup operation on the cleanupable object according to the execution method, and recording a full-process audit log including the operation time, cleanup object, execution method, and data size before cleanup; and updating the metadata information and changing the nodes and edges in the data lineage graph in response to the completion of the cleanup operation.

[0080] As an example, the cleanup identifier can be an identification label used to mark target data as eligible for formal cleanup. Cleanup prompts can be messages generated based on the cleanup type, used to determine the specific cleanup method to be used. Cleanable objects can refer to entities that are allowed to undergo cleanup after simulation verification; this can be a complete data table or a portion of the data within a table. Risk classification and control rules can be pre-defined management specifications based on business impact and risk levels, used to match different cleanup execution methods. The full-process audit log is a record file that retains key information during the data cleanup operation. Metadata information can be used to describe the structure, attributes, and other content of the data table.

[0081] In a more specific example, data on the cleanup list that has passed the simulation verification and is deemed executable for cleanup is uniformly marked with a cleanup identifier, and matching cleanup prompts are set based on its corresponding cleanup type to determine the final cleanupable objects. These objects can be a complete single data table or a portion of data within a table. For cleanupable objects, corresponding cleanup execution methods are assigned according to preset risk classification and control rules. Low-risk cleanup objects are directly physically deleted upon successful verification and without objection from business stakeholders. Medium-risk cleanup objects, even if verified, are moved to the recycle bin and retained for thirty days due to their association with core business operations. High-risk cleanup objects involve cross-departmental data or compliance requirements and require secondary approval before cleanup can proceed. For each cleanupable object on the cleanup list, a corresponding cleanup operation is executed, and a full-process audit log is simultaneously recorded during the cleanup process. The full-process audit log may include the operation time, operator, cleanup object, cleanup method, data size before cleanup, and associated verification report number. The operator can be a system or a human; the cleanup object can be a database name, table name, or partition; and the cleanup method can be physical deletion, recycle bin, or archiving. If the cleanup operation for a currently cleanable object is not completed normally, the corresponding cleanup execution method will continue to be matched and executed according to the preset risk classification and control rules, repeatedly triggering the corresponding cleanup job process until the cleanup operation for that object is completely completed. Once the cleanup task for that object is completed, a metadata information update event is triggered, changing the data lineage graph and removing the cleaned data nodes and corresponding associated edges. Furthermore, a cleanup completion notification can be pushed to the data manager and relevant business parties via email or system message. This process continues to cycle until all cleanup tasks in the cleanup list are completed. In addition, a cleanup effectiveness report is generated regularly, which includes statistics on the cumulative number of cleaned data tables and partitions, the cumulative released storage space, cost savings calculated based on storage unit price, and the execution results of the cleanup tasks.

[0082] Based on this, data to be cleaned is labeled with a cleanup identifier and a cleanup prompt matching the cleanup type, clearly defining the scope of cleanup objects and distinguishing between the entire table and partial data within a table. A corresponding cleanup list is generated according to the cleanup prompts, and differentiated execution methods are matched based on risk-based hierarchical control rules. This enables classified management and tiered handling of cleanup tasks, ensuring business security while improving data cleanup efficiency. During cleanup operations, key information is fully recorded to form an audit log, preserving a complete operational trace and ensuring traceability and problem localization. After cleanup operations are completed, metadata information and nodes and edges in the data lineage graph are updated synchronously, ensuring that data description information, dependencies, and actual data status remain consistent in real time, guaranteeing the accuracy and integrity of the entire data system.

[0083] Based on the above data cleaning method, embodiments of this application also provide a data cleaning apparatus. The following will be combined with... Figure 8 The device is described in detail.

[0084] Figure 8 A schematic block diagram of a data cleaning apparatus according to an embodiment of this application is shown.

[0085] like Figure 5 As shown, the data cleaning device 800 of this embodiment includes a spectrum construction module 810, a list generation module 820, a list evaluation module 830, a simulation cleaning module 840, and a cleaning implementation module 850.

[0086] The graph construction module 810 performs graph structure processing on the operation logs and metadata information to obtain a data lineage graph, which indicates the dependencies between the data tables targeted by the operation logs and metadata information. In one embodiment, the graph construction module 810 can be used to execute step S210 described above, which will not be repeated here.

[0087] The list generation module 820 is used to generate a list of data to be cleaned based on the cleanup priority of the data tables of each node in the data lineage graph. In one embodiment, the list generation module 820 can be used to execute step S220 described above, which will not be repeated here.

[0088] The inventory evaluation module 830 is used to trace downstream dependencies of each piece of data to be cleaned in the inventory of data to be cleaned, and to obtain a report on the scope of impact of the data to be cleaned. In one embodiment, the inventory evaluation module 830 can be used to perform step S230 described above, which will not be repeated here.

[0089] The simulation cleanup module 840 is used to perform a simulated cleanup operation on the data to be cleaned in an isolated environment in response to the impact range report meeting preset conditions, and obtain the simulation verification result of the data to be cleaned. In one embodiment, the simulation cleanup module 840 can be used to execute step S240 described above, which will not be repeated here.

[0090] The cleanup implementation module 850 is used to add the data to be cleaned to the cleanup list and perform the data cleanup operation in response to the simulation verification result indicating that the data to be cleaned can be cleaned. In one embodiment, the cleanup implementation module 850 can be used to perform step S250 described above, which will not be repeated here.

[0091] According to an embodiment of this application, the graph construction module 810 performs graph structure processing on the operation logs and metadata information to obtain a data lineage graph, including: collecting operation logs and metadata information of each operation job in heterogeneous data sources, where the heterogeneous data sources include at least one of a data warehouse, a data lake, and a distributed computing engine; the operation jobs include at least one of full data query, data script, distributed batch processing task, and real-time streaming task; the metadata information indicates at least one of the data structure, data attributes, and data relationships of the metadata contained in each data table in the heterogeneous data sources; performing format parsing and semantic extraction on the operation logs to obtain the input data table and output data table of the operation job, and determining the dependency relationship between the operation job and the input data table and the output data table; constructing a graph for the input data table and the output data table based on the metadata information and dependency relationship to obtain a data lineage graph with data tables as nodes and dependency relationships as edges; wherein, the node information of the nodes includes table information and metadata information, and the edges are used to record at least one of the upstream table, downstream table, operation job identifier, dependency relationship type, and most recent update time of the data table.

[0092] According to an embodiment of this application, the list generation module 820 generates a list of data to be cleaned based on the cleanup priority of the data tables of each node in the data lineage graph. This includes: using the data tables corresponding to each node in the data lineage graph as target tables; obtaining the downstream tables of the target tables based on the edges of the data lineage graph, and the access dependency of each downstream table on the target tables; obtaining the access popularity based on the number of accesses to the target tables indicated by the node information in the data lineage graph and the latest access interval in days; downstream tables include direct downstream tables and indirect downstream tables; determining the cleanup priority of the target tables based on the access dependency, access popularity, and the storage cost coefficient of the target tables to be obtained; arranging the corresponding target tables in descending order of cleanup priority to obtain the list of data to be cleaned; obtaining the access dependency of downstream tables on the target tables, including: identifying the business level corresponding to each downstream table, where the business level includes at least one of core business level, important business level, general business level, and test business level; assigning a business weight coefficient corresponding to the business level to the downstream tables, where the business weight coefficient indicates the importance of the business corresponding to the downstream table; calculating the access dependency of all downstream tables on the target tables. The process involves: summing the business weight coefficients of the downstream tables; using the ratio of the sum of the business weight coefficients to the maximum dependency threshold of the corresponding business domain as the access dependency coefficient of the target table, which indicates the degree of access dependency of all downstream tables on the target table; the maximum dependency threshold of the corresponding business domain being the historical maximum number of downstream dependency nodes of all data tables within the business domain of the target table (pre-statistically counted) or a preset standardized dependency upper limit; determining the cleanup priority of the target table based on the access dependency level, access popularity, and the storage cost coefficient of the target table to be acquired, including: normalizing the target table's storage capacity, storage medium type, and number of replicas to obtain the target table's storage cost coefficient; multiplying the ratio of the number of accesses to the preset maximum access threshold of the corresponding business domain by the ratio of the preset statistical period duration to the latest access interval in days to obtain the target table's access popularity; obtaining the target table's target isolation coefficient based on the access popularity and access dependency level, which indicates the degree of isolation of the target table; using the product of the target isolation coefficient and the storage cost coefficient as the cleanup priority score; and mapping the cleanup priority score to obtain the cleanup priority.

[0093] According to an embodiment of this application, the inventory evaluation module 830 performs downstream dependency tracing on each piece of data to be cleaned in the data to be cleaned inventory to obtain an impact range report of the data to be cleaned. This includes: selecting a single target table as the data to be cleaned in the data to be cleaned in sequence according to preset batch processing rules; performing a reverse depth traversal in the graph database used to store the data lineage graph to trace all downstream dependent nodes of the corresponding node of the data to be cleaned down to the leaf node, where the leaf node includes at least one of terminal business reports, application interfaces, and front-end applications; and generating an impact range report for the data to be cleaned based on the running status of each downstream dependent node, the related business parties, and the degree of business criticality to the related business parties. The impact range includes at least one of the following: downstream operation list, indirect downstream report list, visual dependency link diagram, list of affected business parties, and cleanup risk level.

[0094] According to an embodiment of this application, the simulation cleanup module 840 performs a simulated cleanup operation on the data to be cleaned in an isolated environment to obtain a simulation verification result of the data to be cleaned. This includes: creating a full snapshot copy of the data to be cleaned, placing the full snapshot copy in a sandbox environment that is consistent with the actual operation environment but isolated from it; determining the cleanup type corresponding to the data to be cleaned, and performing the cleanup operation corresponding to the cleanup type in the sandbox environment. The cleanup type includes at least one of full table deletion, partition cleanup, and cold data archiving; in response to the execution of the cleanup operation, running downstream jobs of the data to be cleaned in the sandbox environment and monitoring the running status of the downstream jobs in real time. The running status includes execution status, running logs, and output data; and generating a simulation verification result of the data to be cleaned based on the running status. The simulation verification result is either executable cleanup or non-executable cleanup.

[0095] According to an embodiment of this application, the cleanup implementation module 850 adds the data to be cleaned to the cleanup list and performs data cleanup operations, including: marking the data to be cleaned with a cleanup identifier and a cleanup prompt for the data that the simulation verification result shows is executable for cleanup, the cleanup prompt corresponding to the cleanup type, obtaining cleanable objects, which are a single data table or part of the data in a single data table; adding the cleanable objects to the cleanup list corresponding to the cleanup prompt, and assigning a corresponding cleanup execution method to the cleanable objects according to preset risk classification and control rules; performing cleanup operations on the cleanable objects according to the execution method, and recording a full-process audit log including operation time, cleanup object, execution method, and data size before cleanup; and updating metadata information and changing nodes and edges in the data lineage graph in response to the completion of the cleanup operation.

[0096] According to embodiments of this application, any multiple modules among the spectrum construction module 810, inventory generation module 820, inventory evaluation module 830, simulation cleanup module 840, and cleanup implementation module 850 can be merged into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functions of one or more of these modules can be combined with at least some of the functions of other modules and implemented in one module. According to embodiments of this application, at least one of the spectrum construction module 810, inventory generation module 820, inventory evaluation module 830, simulation cleanup module 840, and cleanup implementation module 850 can be at least partially implemented as hardware circuitry, such as field-programmable gate arrays, programmable logic arrays, systems-on-a-chip, systems-on-a-substrate, systems-on-package, application-specific integrated circuits, or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the spectrum construction module 810, inventory generation module 820, inventory evaluation module 830, simulation cleanup module 840, and cleanup implementation module 850 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0097] Figure 9 A block diagram schematically illustrates an electronic device suitable for implementing a data cleaning method according to an embodiment of this application.

[0098] like Figure 9 As shown, an electronic device 900 according to an embodiment of this application includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory 902 or a program loaded from a storage portion 908 into a random access memory 903. The processor 901 may include, for example, a general-purpose microprocessor, an instruction set processor and / or an associated chipset and / or a dedicated microprocessor. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for executing different steps of the method flow according to an embodiment of this application.

[0099] Random access memory 903 stores various programs and data required for the operation of electronic device 900. Processor 901, read-only memory 902, and random access memory 903 are interconnected via bus 904. Processor 901 executes various steps of the method flow according to embodiments of this application by executing programs stored in read-only memory 902 and / or random access memory 903. It should be noted that the programs may also be stored in one or more memories other than read-only memory 902 and random access memory 903. Processor 901 may also execute various steps of the method flow according to embodiments of this application by executing programs stored in said one or more memories.

[0100] According to embodiments of this application, the electronic device 900 may further include an input / output interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube, liquid crystal display, etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card, such as a local area network card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the input / output interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0101] Embodiments of this application also provide a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the method according to the embodiments of this application.

[0102] According to embodiments of this application, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including but not limited to: portable computer disks, hard disks, random access memory, read-only memory, erasable programmable read-only memory, portable compact disk read-only memory, optical storage devices, magnetic storage devices, or any suitable combination thereof. In embodiments of this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this application, the computer-readable storage medium may include the read-only memory 902 described above, and / or random access memory 903, and / or one or more memories other than read-only memory 902 and random access memory 903.

[0103] Embodiments of this application also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this application.

[0104] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0105] In embodiments of this application, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by processor 901, it performs the functions defined in the system of embodiments of this application. According to embodiments of this application, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0106] According to embodiments of this application, program code for executing the computer programs provided in the embodiments of this application can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. The program code can be executed entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0107] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0108] Those skilled in the art will understand that the features described in the various embodiments of this application can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this application. In particular, the features described in the various embodiments of this application can be combined and / or combined in various ways without departing from the spirit and teachings of this application. All such combinations and / or combinations fall within the scope of this application.

Claims

1. A data cleaning method, characterized in that, The method includes: The operation logs and metadata information are processed into a graph structure to obtain a data lineage graph, which indicates the dependency relationship between the operation logs and the data tables to which the metadata information is attached. Based on the cleaning priority of the data tables of each node in the data lineage graph, a list of data to be cleaned is generated; Perform downstream dependency tracing on each piece of data to be cleaned in the list of data to be cleaned to obtain a report on the scope of impact of the data to be cleaned. In response to the reported impact range meeting preset conditions, a simulated cleanup operation is performed on the data to be cleaned in an isolated environment to obtain the simulation verification results of the data to be cleaned. In response to the simulation verification result indicating that the data to be cleaned can be cleaned, the data to be cleaned is added to the cleanup list and the data cleanup operation is performed.

2. The method according to claim 1, characterized in that, Graph structure processing is performed on the operation logs and metadata information to obtain a data lineage graph, including: The operation logs and metadata information of each operation job in the heterogeneous data source are collected. The heterogeneous data source includes at least one of a data warehouse, a data lake, and a distributed computing engine. The operation job includes at least one of a full data query, a data script, a distributed batch processing task, and a real-time streaming task. The metadata information indicates at least one of the data structure, data attributes, and data relationships of the metadata contained in each data table in the heterogeneous data source. The operation log is parsed and semantically extracted to obtain the input data table and output data table of the operation job, and the dependency relationship between the operation job and the input data table and the output data table is determined. Based on the metadata information and the dependencies, a graph is constructed for the input data table and the output data table to obtain a data lineage graph with data tables as nodes and dependencies as edges. The node information of the nodes includes table information and the metadata information, and the edges are used to record at least one of the following: the upstream table, the downstream table, the operation job identifier, the dependency type, and the most recent update time.

3. The method according to claim 1, characterized in that, Based on the cleanup priority of the data tables of each node in the data lineage graph, a list of data to be cleaned is generated, including: Use the data table corresponding to each node in the data lineage graph as the target table; Based on the edges of the data lineage graph, the downstream tables of the target table are obtained, as well as the access dependence of each downstream table on the target table. The access popularity is obtained according to the number of accesses to the target table indicated by the node information of the data lineage graph and the latest access interval in days. The downstream tables include direct downstream tables and indirect downstream tables. The cleanup priority of the target table is determined based on the access dependency, the access popularity, and the storage cost coefficient of the target table to be acquired. Arrange the target tables in descending order of cleaning priority to obtain the list of data to be cleaned.

4. The method according to claim 1, characterized in that, For each piece of data to be cleaned in the list of data to be cleaned, downstream dependency tracing is performed to obtain an impact range report of the data to be cleaned, including: According to the preset batch processing rules, select a single target table from the list of data to be cleaned as the data to be cleaned in sequence; Perform a reverse depth traversal in the graph database used to store the data lineage graph to trace all downstream dependent nodes of the node corresponding to the data to be cleaned up to the leaf node. The leaf node includes at least one of terminal business reports, application interfaces and front-end applications. Based on the operating status of each downstream dependent node, the associated business parties, and the degree of business criticality to the associated business parties, an impact scope report is generated for the data to be cleaned. The impact scope includes at least one of the following: a list of downstream operations, a list of indirect downstream reports, a visual dependency chain diagram, a list of affected business parties, and a cleanup risk level.

5. The method according to claim 1, characterized in that, The data to be cleaned is simulated and cleaned in an isolated environment to obtain simulation verification results, including: Create a full snapshot copy of the data to be cleaned, and place the full snapshot copy in a sandbox environment that is consistent with the actual operation environment but isolated from the actual operation environment; Determine the cleanup type corresponding to the data to be cleaned, and perform the cleanup operation corresponding to the cleanup type in the sandbox environment. The cleanup type includes at least one of full table deletion, partition cleanup, and cold data archiving. In response to the cleanup operation, a downstream job of the data to be cleaned is run in the sandbox environment, and the running status of the downstream job is monitored in real time, including execution status, running logs and output data; Based on the operating status, a simulation verification result is generated for the data to be cleaned, and the simulation verification result is either executable cleanup or non-executable cleanup.

6. The method according to claim 5, characterized in that, Add the data to be cleaned to the cleanup list and perform data cleanup operations, including: To simulate and verify that the data to be cleaned is ready for cleanup, a cleanup identifier and a cleanup prompt are marked. The cleanup prompt corresponds to the cleanup type, thus obtaining cleanable objects. The cleanable objects are a single data table or a portion of the data in the single data table. Add the cleanable objects to the cleanup list corresponding to the cleanup prompt, and assign corresponding cleanup execution methods to the cleanup objects according to the preset risk classification and control rules; Perform a cleanup operation on the cleanable object according to the execution method, and record a full-process audit log including operation time, cleanup object, execution method, and data size before cleanup; In response to the completion of the cleanup operation, the metadata information is updated, and the nodes and edges in the data lineage graph are changed.

7. The method according to claim 3, characterized in that, Obtaining the access dependency of the downstream table on the target table includes: Identify the business level corresponding to each of the downstream tables, wherein the business level includes at least one of the following: core business level, important business level, general business level, and test business level; Assign a business weight coefficient to the downstream table that corresponds to the business level, wherein the business weight coefficient indicates the importance of the business corresponding to the downstream table; Calculate the sum of the business weight coefficients of all downstream tables of the target table; The ratio of the sum of the business weight coefficients to the maximum dependency threshold of the corresponding business domain is used as the access dependency coefficient of the target table. The access dependency coefficient indicates the degree of access dependency of all downstream tables on the target table. The maximum dependency threshold of the corresponding business domain is the historical maximum number of downstream dependency nodes of all data tables in the business domain where the target table is located, which is pre-statistically counted, or a preset standardized dependency upper limit value.

8. The method according to claim 3, characterized in that, Based on the access dependency, access popularity, and storage cost coefficient of the target table to be acquired, the cleanup priority of the target table is determined, including: Based on the storage capacity, storage medium type, and number of replicas of the target table, normalization processing is performed to obtain the storage cost coefficient of the target table. The ratio of the number of accesses to the preset maximum access threshold of the corresponding business domain is multiplied by the ratio of the preset statistical period duration to the latest access interval in days to obtain the access popularity of the target table. The target isolation coefficient of the target table is obtained based on the access popularity and the access dependence degree. The target isolation coefficient is used to indicate the isolation degree of the target table. The product of the target isolation coefficient and the storage cost coefficient is used as the cleanup priority score; The cleanup priority score is mapped to obtain the cleanup priority.

9. A data cleaning device, characterized in that, The device includes: The spectrum construction module is used to perform graph structure processing on operation logs and metadata information to obtain a data lineage graph, which indicates the dependency relationship between the operation logs and the data tables to which the metadata information is attached; The list generation module is used to generate a list of data to be cleaned based on the cleanup priority of the data tables of each node in the data lineage graph. The inventory evaluation module is used to trace the downstream dependencies of each piece of data to be cleaned in the data to be cleaned inventory and obtain a report on the scope of impact of the data to be cleaned. The simulation cleanup module is used to perform a simulated cleanup operation on the data to be cleaned in an isolated environment in response to the impact range report meeting preset conditions, and to obtain the simulation verification result of the data to be cleaned. The cleanup implementation module is used to add the data to be cleaned to the cleanup list and perform data cleanup operations in response to the simulation verification result indicating that the data to be cleaned can be cleaned.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more computer programs. The characteristic feature is that the one or more processors execute the one or more computer programs to implement the steps of the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program or instructions stored thereon, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed by a processor, they implement the steps of the method according to any one of claims 1 to 8.

Images