Business self-adapting data cleaning method and system based on multi-thread parallel

Abstract

This application provides a business-adaptive data cleaning method and system based on multi-threaded parallelism, belonging to the field of data cleaning technology. The method includes: collecting data to be cleaned, its lineage information, and historical quality score information to form a metadata graph; then, preprocessing and sharding the data to be cleaned based on the data characteristics recorded in the graph, generating a set of data shards carrying label information; next, performing dependency-aware multi-threaded task scheduling based on this set, wherein the dependency relationships of data units are analyzed based on the graph to allocate units without dependencies to different execution threads in parallel, and computing resources are allocated to each thread based on the label information, resulting in a set of data units that have been scheduled and processed; then, anomaly detection and inference are performed on this set to generate data cleaning rules; finally, business-adaptive data cleaning operations are performed based on these rules. This application improves the adaptive capability and processing efficiency of real-time cleaning of multi-source heterogeneous data.

Description

Technical Field

[0001] This application relates to the field of data cleaning technology, and in particular to a business adaptive data cleaning method and system based on multi-threaded parallelism. Background Technology

[0002] The business-adaptive data cleaning method is mainly applied to the quality governance of multi-source heterogeneous data during real-time synchronization. With the popularization of enterprise data lake warehouse architecture, this method shows broad application prospects in ensuring data consistency, integrity and timeliness.

[0003] In existing technologies, a basic data relationship graph is first established by parsing the metadata of the data source. Then, the data stream is initially classified and filtered based on preset rules. Next, a multi-threaded parallel architecture is used to perform format conversion and content verification operations on the classified data. Finally, abnormal data is marked or isolated based on the verification results.

[0004] However, under this processing mode, due to the influence of network environment fluctuations and changes in the source system during the real-time synchronization of the data stream, problems such as structural conflicts and content anomalies often occur, resulting in a deviation between the processing results and business expectations, making it difficult to meet the quality requirements of real-time data applications; therefore, there is a technical problem in the existing technology that the business adaptability of the data cleaning process needs to be improved. Summary of the Invention

[0005] This application provides a business adaptive data cleaning method and system based on multi-threaded parallelism to solve the problems of poor adaptability and low processing efficiency of real-time cleaning of multi-source heterogeneous data in the prior art.

[0006] To address the aforementioned technical problems, firstly, this application provides a business adaptive data cleaning method based on multi-threaded parallelism, comprising:

[0007] Collect the data to be cleaned, the lineage information of the data to be cleaned, and the quality score information of historical cleaning tasks;

[0008] Based on the blood relationship information and the quality score information, a metadata map is formed;

[0009] Based on the data characteristics recorded in the metadata map, the data to be cleaned is preprocessed and fragmented to generate a set of data fragments carrying tag information;

[0010] Dependency-aware multi-threaded task scheduling is performed based on the data shard set. The multi-threaded task scheduling analyzes the dependency relationship of data units in the data shard set based on the metadata graph, allocates multiple data units without dependency relationship to different execution threads in parallel, and allocates computing resources to each execution thread based on the marking information. Based on the resource allocation result, a set of data units that have been scheduled and processed is obtained.

[0011] Using a graph neural network model, anomaly detection and inference are performed on the data unit set that has been scheduled and processed, so as to generate data cleaning rules;

[0012] In the multi-threaded execution environment established by the multi-threaded task scheduling, a business-adaptive data cleaning operation is performed on the data unit set that has been scheduled and processed based on the data cleaning rules.

[0013] Secondly, this application provides a business adaptive data cleaning system based on multi-threaded parallelism, comprising:

[0014] The data acquisition module is used to acquire the data to be cleaned, the lineage information of the data to be cleaned, and the quality score information of historical cleaning tasks.

[0015] A forming module is used to form a metadata map based on the blood relationship information and the quality score information;

[0016] The sharding module is used to preprocess and shard the data to be cleaned based on the data features recorded in the metadata map, and generate a set of data shards carrying tag information.

[0017] An execution module is used to perform dependency-aware multi-threaded task scheduling based on the data shard set. The multi-threaded task scheduling analyzes the dependency relationship of data units in the data shard set based on the metadata graph, allocates multiple data units without dependency relationship to different execution threads in parallel, allocates computing resources to each execution thread based on the marking information, and obtains a set of data units after scheduling based on the resource allocation result.

[0018] The detection module is used to perform anomaly detection and inference on the set of scheduled data units using a graph neural network model, so as to generate data cleaning rules;

[0019] The cleaning module is used to perform business-adaptive data cleaning operations on the set of scheduled data units based on the data cleaning rules in the multi-threaded execution environment established by the multi-threaded task scheduling.

[0020] Thirdly, this application provides an electronic device, comprising:

[0021] Memory, used to store computer programs;

[0022] A processor, used to execute the computer program to implement the steps of the multi-threaded parallel business adaptive data cleaning method as described in the first aspect above.

[0023] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps of the multi-threaded parallel business adaptive data cleaning method described in the first aspect above.

[0024] The technical solution provided in this application has the following beneficial effects:

[0025] This application collects the data to be cleaned, its lineage information, and historical quality score information, which can establish a complete data foundation for subsequent processing. Then, based on this information, a metadata graph is formed, realizing the organic integration of data entity relationships and quality history. Next, according to the data characteristics recorded in the graph, the data to be cleaned is preprocessed and fragmented, and a set of data fragments carrying label information is generated, so that data from different sources and structures can receive differentiated preliminary processing. Then, based on this set, a dependency-aware multi-threaded task scheduling is executed. By analyzing the dependency relationship of data units, units without dependency relationships are allocated to different execution threads in parallel, and computing resources are allocated based on the label information, thereby improving the processing efficiency in a multi-threaded environment.

[0026] Then, a graph neural network model is used to perform anomaly detection and inference on the data unit set after scheduling and processing to generate data cleaning rules, realizing intelligent anomaly identification and rule generation; finally, in a multi-threaded execution environment, business-adaptive data cleaning operations are performed based on the generated rules to ensure that the cleaning process can match the current business context and data characteristics.

[0027] Furthermore, in the process of dependency-aware multi-threaded task scheduling, this application reads the metadata graph to analyze the dependency relationship of each data piece and identifies the associated data pieces with dependency relationships. Then, based on these dependency relationships, a task dependency graph containing nodes and edges is constructed. Next, the topology sorting algorithm is used to analyze the graph to determine the target data pieces to be processed in parallel that do not have dependency relationships. Finally, the required computing resources are determined according to the labeling information carried by each data piece.

[0028] Subsequently, the computing resource requirements based on the target data shards are allocated to multiple execution threads through a load balancing algorithm, and each thread is allocated a corresponding memory space and processor time quota. The processing results data are generated through parallel processing by multiple threads, and finally the processing results are combined according to the dependency relationship to form a set of data units that have been scheduled and processed. This process achieves dynamic balanced allocation of computing resources while ensuring the correctness of data dependency logic, effectively improving the overall efficiency of parallel processing.

[0029] These or other aspects of this application will become more apparent in the following description of the embodiments. Attached Figure Description

[0030] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0031] Figure 1 A flowchart illustrating a business adaptive data cleaning method based on multi-threaded parallelism provided in this application embodiment;

[0032] Figure 2 This application provides a schematic diagram illustrating a specific implementation of a business adaptive data cleaning method based on multi-threaded parallelism.

[0033] Figure 3 This is a schematic diagram of the structure of a business adaptive data cleaning system based on multi-threaded parallelism, provided in an embodiment of this application. Detailed Implementation

[0034] To address the problems of existing technologies, this application proposes a business-adaptive data cleaning method based on multi-threaded parallelism. Its core lies in constructing a metadata graph by collecting data lineage and historical quality information, which serves as the basis for data feature perception. This method abandons the fixed processing mode of traditional static rules. Through the coordinated efforts of data lineage perception, dynamic scheduling of multi-threaded resources, and intelligent rule generation mechanisms, the cleaning strategy can adaptively adjust to changes in data characteristics and business context. This avoids misjudgment and discarding of time-sensitive data in network fluctuation scenarios and eliminates information omissions caused by rule rigidity when the source structure changes. It fundamentally solves the problem of data cleaning failure caused by dynamic environmental changes in existing technologies, improving the quality assurance capability and business adaptability of real-time data synchronization.

[0035] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0036] The core of this application is to provide a business-adaptive data cleaning method based on multi-threaded parallelism, and a flowchart of one specific implementation is shown below. Figure 1 As shown, the method includes:

[0037] Step 101: Collect the data to be cleaned, the lineage information of the data to be cleaned, and the quality score information of historical cleaning tasks.

[0038] In step 101, the data to be cleaned refers to the raw data units in the real-time synchronization process from database tables, API data streams, and log files to the data lake warehouse;

[0039] Lineage information refers to data that records the characteristics of these original data units in terms of source, transformation and dependency, specifically including table entities, field entities and dependency flow information between table entities and field entities;

[0040] The quality score information for historical data cleaning tasks refers to a set of scores stored in a structured format, generated after evaluating the data quality of each field entity during past cleaning operations. This quality score information is generated by the data quality assessment system after each cleaning task is completed and stored in the historical task log.

[0041] Each quality score record must contain at least the following fields: data field identifier (field_id), cleaning task execution timestamp (task_timestamp), and quality score (quality_score). The quality score is a continuous numerical value between 0 and 100, used to quantify the overall quality performance of that field after the cleaning task; a higher score indicates better quality. This score is calculated by weighting multiple dimensions such as data completeness, accuracy, consistency, and timeliness. The specific calculation method is a mature technology in this field and will not be elaborated here.

[0042] In this embodiment of the application, the data to be cleaned is obtained from the data source through the data acquisition interface, the lineage information is read from the metadata storage component, and the quality score information is extracted from the historical task log. These three parts of data are used as the input basis for subsequent processing.

[0043] Step 102: Based on the blood relationship information and the quality score information, form a metadata map.

[0044] Metadata graphs refer to graph-structured data that integrates data entities, inter-entity dependencies, and historical quality information of entities. They are used to characterize the complete contextual features of the data to be cleaned in terms of its source, transformation, dependencies, and quality changes.

[0045] In this embodiment, step 102 includes the following process:

[0046] Step 1021: Parse the bloodline relationship information to obtain table information, field information, and dependency information between the table information and field information.

[0047] In step 1021, table information refers to the descriptive data about the data table in the blood relationship information, which specifically includes the table identifier and table name; field information refers to the descriptive data about the data field in the blood relationship information, which specifically includes the field identifier, field name and field type.

[0048] Dependency information refers to the data flow relationship between tables and fields recorded in the lineage information. Specifically, this dependency information includes the correspondence between upstream table identifiers and downstream table identifiers, as well as the correspondence between upstream field identifiers and downstream field identifiers.

[0049] In this embodiment, during the data preparation phase after the multi-threaded parallel cleaning process is started, the collected lineage information is first read by the lineage parser. All table identifiers and table names involved are extracted from this information as table information, and all field identifiers, field names, and field types involved are extracted as field information. Then, the upstream and downstream correspondences between tables and between fields are parsed from the lineage information. These correspondences are used as dependency information, which will be used to build a graph structure that supports dependency-aware scheduling in the future.

[0050] Step 1022: Based on the table information, field information, and dependency information, construct a hierarchical structure diagram containing a first node and a second node, wherein the first node corresponds to the table information and the second node corresponds to the field information.

[0051] In step 1022, the hierarchical structure graph refers to a two-layer graph structure with tables as upper-level nodes and fields as lower-level nodes. The first node represents table information and serves as the upper-level node, while the second node represents field information and serves as the lower-level node. The edges connecting the first node represent table-level dependency information, and the edges connecting the second node represent field-level dependency information. This hierarchical structure graph is the basis for analyzing data unit dependencies in multi-threaded task scheduling.

[0052] In this embodiment, each table identifier in the table information is first used as a first node, and each field identifier in the field information is used as a second node. Then, according to the table-level dependencies in the dependency information, connection edges are established between the corresponding first nodes, and according to the field-level dependencies in the dependency information, connection edges are established between the corresponding second nodes. Finally, a subordinate connection is established between the second node corresponding to each field and the first node corresponding to its table, forming a hierarchical structure diagram containing two layers of nodes. This diagram will be used for subsequent analysis of the dependencies between data units in the data shard set.

[0053] Step 1023: Associate the quality score information with the corresponding field information to form a historical quality record corresponding to the field information, and add the historical quality record as dynamic information to the corresponding second node in the hierarchical structure diagram.

[0054] In step 1023, historical quality records refer to a set of quality scores arranged in chronological order that are associated with specific field information, used to describe the changes in the quality performance of the field in historical cleaning tasks; dynamic information refers to data attributes attached to graph nodes that change over time, which will be used in multi-threaded task scheduling to help determine the business priority and processing level of data units.

[0055] In this embodiment, the quality score information is first grouped according to field identifiers. For each field identifier, all quality scores corresponding to that field identifier are sorted in chronological order to form a historical quality record in time series form. Then, this historical quality record is added as dynamic information to the second node corresponding to the field identifier in the hierarchical structure diagram, so that the second node carries both the static information of the field itself and the dynamic information of historical quality changes. This information will be used in subsequent multi-threaded task scheduling to dynamically allocate computing resources in combination with the tag information.

[0056] Step 1024: Using the community detection algorithm, traverse the hierarchical structure graph to identify multiple associated fields with the same upstream dependency, divide the multiple associated fields into the same group, and add grouping information to each group in the hierarchical structure graph.

[0057] In step 1024, the associated field refers to multiple fields in the dependency information that have a common upstream field; the grouping information refers to a common tag that identifies all fields in the same group, which will be used in multi-threaded task scheduling to quickly identify data units with dependencies.

[0058] In this embodiment, the specific steps for field clustering using a community detection algorithm (e.g., Louvain's algorithm) are as follows: First, each second node (field node) in the hierarchical graph is initialized as an independent community. Then, the algorithm iterates based on modularity optimization. In each iteration, for each node i, the modularity gain ΔQ resulting from moving it to the community of its neighbor node j is calculated. If the maximum gain max(ΔQ) > 0, then node i is moved to the community of its corresponding neighbor.

[0059] Otherwise, node i remains in its original community. This process iterates through all nodes until the community structure no longer changes. Next, the algorithm aggregates the current community into new "super nodes," constructs a new network, and repeats the node movement process until the modularity can no longer be improved. Finally, the algorithm converges, yielding the community partitioning results for the field nodes.

[0060] In the scenario of this application, since the edges between fields are established based on lineage dependency, field nodes that are assigned to the same community mean that they are closely connected in the dependency graph, usually sharing the same upstream data source or going through similar transformation paths, i.e., having "the same upstream dependency".

[0061] Finally, a unique group identifier is assigned to each community, and this identifier is added as group information to the attribute of the corresponding second node in the hierarchical structure graph. This allows for quick identification of data units with close dependencies during multi-threaded task scheduling by directly using the group identifier.

[0062] Step 1025: Based on the grouping information and the dependency information, determine the dependency distance between any two second nodes in the hierarchical graph using a graph embedding model.

[0063] In step 1025, dependency distance refers to a numerical value that characterizes the degree of dependency between two second nodes in the vector space. The smaller the value, the tighter the dependency. In multi-threaded task scheduling, it is used to determine the strength of dependency between data units to determine the feasibility of parallel processing.

[0064] In this embodiment, the hierarchical structure graph and its carried grouping information, dependency information, and historical quality records are first used as input data, and a pre-trained graph embedding model is loaded. Each second node in the hierarchical structure graph is vectorized using the graph embedding model to generate an embedding vector for each second node. Then, for any two second nodes, their Euclidean distance in the vector space is calculated as the dependency distance. This dependency distance comprehensively reflects the similarity of the two fields in terms of dependency, grouping, and historical quality. In subsequent multi-threaded task scheduling, the dependency distance will be used to determine which data units can be processed in parallel.

[0065] Step 1026: Generate a metadata graph based on the dependency distance, the grouping information, and the second node carrying historical quality records.

[0066] In this embodiment, the dependency distance between each second node calculated in step 1025 is first added as an edge weight to the corresponding connection edge in the hierarchical structure graph; then, the grouping information added in step 1024 is retained as a node attribute on each second node; finally, the second nodes carrying historical quality records, the updated edge weights, and the grouping information are integrated and encapsulated to form a metadata graph containing complete dependency quantification information. This graph will be used in subsequent steps to analyze the dependency relationship of each data unit in the data shard set to perform dependency-aware multi-threaded task scheduling, and will also be used for anomaly detection and inference in the graph neural network model.

[0067] This application constructs a metadata graph that includes field nodes, dependencies, historical quality records, grouping information, and dependency distances, achieving deep integration and structured expression of data lineage and quality history information. This provides a complete data context for dependency-aware scheduling and intelligent anomaly detection in multi-threaded parallel cleaning processes.

[0068] Step 103: Based on the data features recorded in the metadata map, preprocess and fragment the data to be cleaned to generate a set of data fragments carrying tag information.

[0069] Among them, data characteristics refer to the information stored in the metadata graph about the data source, structure type, and historical quality of fields; tagging information refers to the descriptive data attached to the data block, which is used to indicate the source, processing level, and sharding type of the data block. This information will be used to allocate computing resources and determine the processing order in subsequent multi-threaded task scheduling; the data shard set refers to the set of multiple data blocks generated after the data to be cleaned has undergone preprocessing and sharding operations. Each data block in this set carries tagging information, and the data blocks in the set are arranged in the order of their generation time, which is used as the input data unit for subsequent multi-threaded task scheduling.

[0070] In this embodiment, step 103 includes the following process:

[0071] Step 1031: Based on the historical quality records of the second node corresponding to the data source in the metadata map, determine the initial processing level of data units from different sources in the data to be cleaned.

[0072] In step 1031, the initial processing level refers to the priority value assigned to data units from different sources based on the field's historical quality records. This value reflects the stability of the source data's quality in historical cleaning tasks. Sources with smaller score fluctuations in the quality records will receive a higher processing level.

[0073] In this embodiment, historical quality records for each second node corresponding to the data source to be cleaned are first read from the metadata graph. For each data source, it is determined whether historical quality records exist.

[0074] If there is no historical quality record, such as when cleaning the data source for the first time, a preset default initial processing level is assigned to the data source, such as level 5. Assuming the level range is 1-10, the higher the level, the higher the priority.

[0075] If historical quality records exist, the fluctuation of the quality score in the corresponding second node's historical quality record within a preset time window (e.g., the most recent 10 cleaning tasks) is analyzed. The fluctuation can be quantified by calculating the standard deviation of the quality score sequence. Data sources with smaller standard deviations in historical quality records are assigned higher initial processing levels (e.g., levels 8-10), while sources with larger standard deviations are assigned lower initial processing levels (e.g., levels 1-3).

[0076] Step 1032: Based on the data characteristics recorded in the metadata graph and in combination with the initial processing level, the data to be cleaned is divided into processing data streams of different priorities.

[0077] In step 1032, the data characteristics include the data source type and data structure type recorded in the metadata graph, the processed data stream refers to the data stream classification result after being divided according to the initial processing level and data characteristics, and the priority refers to the order in which the processed data stream is assigned in subsequent sharding and thread scheduling.

[0078] In this embodiment of the application, the data characteristics of each source of the data to be cleaned are first obtained from the metadata map, including the data source type and data structure type; then the initial processing level determined in step 1031 is comprehensively evaluated with these data characteristics; then, the data to be cleaned is divided into a first priority processing data stream and a second priority processing data stream according to the evaluation results, wherein the data source with a higher initial processing level and a relatively simple data structure is assigned to the first priority processing data stream, and the data source with a lower initial processing level or a complex data structure is assigned to the second priority processing data stream.

[0079] Step 1033: For the first priority processing data stream, a preset first sharding strategy is used to cut it into multiple first-class data blocks, and a first-class tag containing the data source, the initial processing level and the first sharding type is attached to each first-class data block.

[0080] In step 1033, the first sharding strategy refers to the preset cutting rules for high-priority data streams. These rules divide the data streams according to natural boundaries with the goal of maintaining the integrity of the data records. The first type of data block refers to the data unit generated after being cut by the first sharding strategy. The first type of tag refers to the identification information attached to the first type of data block. This identification information specifically includes the source identifier, initial processing level, and first sharding type identifier of the data block.

[0081] In this embodiment, the first priority data stream identified in step 1032 is first used as the processing object; then, the data stream is cut according to a preset first fragmentation strategy, which divides the data records based on their natural boundaries, such as newline characters or record separators, to generate multiple complete first-class data blocks; next, a first-class tag is attached to each generated first-class data block, which records the data source, initial processing level, and first fragmentation type identifier of the data block; finally, these first-class data blocks carrying the first-class tag are temporarily stored in preparation for the subsequent merging step.

[0082] Step 1034: For the second priority processing data stream, a preset second sharding strategy is used to cut and reconstruct it, generating multiple second type data blocks, and a second type tag containing the data source, the initial processing level, and the second sharding type is attached to each second type data block. The second priority is lower than the first priority.

[0083] In step 1034, the second sharding strategy refers to the preset cutting rules for low-priority data streams, which require structural parsing and division into logical units; the second type of data block refers to the data unit generated after being cut and reconstructed by the second sharding strategy; the second type of tag refers to the identification information attached to the second type of data block, which specifically includes the source identifier, initial processing level, and second sharding type identifier of the data block.

[0084] In this embodiment, the second priority processing data stream identified in step 1032 is first used as the processing object; then, the data stream is processed according to a preset second fragmentation strategy. This strategy first parses the nested structure of the data stream, identifies the boundaries of logical units, and then cuts and reconstructs the data based on the identified boundaries to generate multiple relatively complete second-type data blocks; next, a second-type tag is attached to each generated second-type data block, which records the data source, initial processing level, and second fragmentation type identifier of the data block; finally, these second-type data blocks carrying the second-type tag are temporarily stored in preparation for the subsequent merging step.

[0085] Step 1035: Merge the first type of data block with the first type of tag and the second type of data block with the second type of tag to form a data fragment set carrying tag information.

[0086] In this embodiment, firstly, all first-class data blocks carrying the first-class tag generated in step 1033 are collected; simultaneously, all second-class data blocks carrying the second-class tag generated in step 1034 are collected; then, these data blocks are merged and arranged according to the time order of data block generation to form a unified data fragment set; finally, this set is used as the output result of step 103 and passed to the subsequent dependency-aware multi-threaded task scheduling step, wherein the tag information carried by each data block will be used for computing resource allocation decisions during the scheduling process.

[0087] This application achieves refined preprocessing of the data to be cleaned by adopting differentiated sharding strategies for data streams of different priorities and determining the initial processing level by combining historical quality records in the metadata graph. This provides structured data units carrying rich contextual information for subsequent multi-threaded task scheduling.

[0088] Step 104: Perform dependency-aware multi-threaded task scheduling based on the data shard set. The multi-threaded task scheduling analyzes the dependency relationship of data units in the data shard set based on the metadata graph, allocates multiple data units without dependency relationship to different execution threads in parallel, and allocates computing resources to each execution thread based on the marking information. Based on the resource allocation result, a set of data units that have been scheduled and processed is obtained.

[0089] Among them, dependency-aware multithreaded task scheduling refers to a multithreaded scheduling mechanism that can identify the dependencies between data units and allocate them in parallel according to the dependencies.

[0090] Dependency refers to the upstream and downstream relationships between data entities corresponding to different data shards in a data shard set within the metadata graph. Specifically, the processing result of one data shard serves as a prerequisite for the processing of another data shard. This relationship is represented by the connecting edges between second-level nodes in the metadata graph. For example, in an order data synchronization scenario, there is a dependency relationship between the data shard corresponding to the order details field and the data shard corresponding to the order summary field, because the calculation of the order summary field requires the parsing and processing of the order details field to be completed first.

[0091] Computing resources refer to the estimated size of memory space and processor time required to process a single data fragment. This value is estimated based on the fragment type and data source in the tag information carried by the data fragment. The set of data units processed by scheduling refers to the aggregated result of data units generated after multi-threaded task scheduling.

[0092] In this embodiment, step 104 includes the following process, such as... Figure 2 As shown:

[0093] Step 1041: Read the metadata graph to analyze the dependency relationship of each data shard in the data shard set in the metadata graph, and identify multiple related data shards with dependency relationships.

[0094] In step 1041, associated data fragments refer to two or more data fragments that have direct or indirect dependencies in the metadata graph, and these data fragments need to be processed in a specific order.

[0095] In this embodiment, a pre-built metadata graph is first read from the storage system. Then, each data shard in the data shard set is traversed. For each data shard, its corresponding second node in the metadata graph is found. Next, the connection edges of the second node are traversed to find other second nodes that have a dependency relationship with the second node. Finally, the data shards corresponding to these second nodes are identified as related data shards with a dependency relationship.

[0096] In practical applications, suppose the data shard set contains data shard P1 corresponding to the order amount field, data shard P2 corresponding to the payment amount field, and data shard P3 corresponding to the user name field. By reading the metadata graph, it is found that there is a connection edge between the second node corresponding to the order amount field and the second node corresponding to the payment amount field. Therefore, data shard P1 and data shard P2 are identified as related data shards with a dependency relationship, while data shard P3 has no connection edge with other shards and therefore has no dependency relationship.

[0097] Step 1042: Based on the dependency relationship, construct a task dependency graph, which includes multiple nodes and edges connecting the nodes, with each node corresponding to a data shard.

[0098] In step 1042, the task dependency graph is a directed graph structure used to intuitively express the dependency relationships between data fragments, where nodes represent data fragments to be processed and edges represent the dependency directions between fragments.

[0099] In this embodiment of the application, each data shard in the data shard set is first used as a node in the task dependency graph. Then, according to the dependency relationship identified in step 1041, a directed edge is added between two nodes with a dependency relationship. The direction of the directed edge is from the upstream dependent shard to the downstream dependent shard. Finally, after all nodes and edges are added, a complete task dependency graph is formed.

[0100] In practical applications, based on the identification results of step 1041, data fragment P1, data fragment P2, and data fragment P3 are taken as three nodes. Since data fragment P1 and data fragment P2 have a dependency relationship and P1 is upstream and P2 is downstream, a directed edge from P1 to P2 is added between node P1 and node P2. Node P3 has no connection edge, forming a task dependency graph containing three nodes and one directed edge.

[0101] Step 1043: Analyze the task dependency graph using the topological sorting algorithm to determine multiple target data fragments to be processed in parallel, wherein there is no dependency relationship between the multiple target data fragments.

[0102] In step 1043, the target data shard refers to a set of data shards that are independent and can be processed in parallel, as determined by topological sorting analysis.

[0103] In this embodiment of the application, the task dependency graph constructed in step 1042 is first used as the input of the topology sorting algorithm. The topology sorting algorithm traverses all nodes in the graph. During the traversal, nodes with an in-degree of zero are identified, that is, nodes without any predecessor dependencies. These nodes with an in-degree of zero are used as target data shards that can be processed in parallel in the current batch. Then, these nodes and their associated edges are removed from the graph, and the next round of traversal is continued to identify new nodes with an in-degree of zero. This process is repeated until all nodes have been processed.

[0104] In practical applications, the task dependency graph containing nodes P1, P2, P3 and their dependencies is input into the topology sorting algorithm. In the first round of traversal, it is found that the in-degree of nodes P1 and P3 is 0 and there is no predecessor dependency. Therefore, data shards P1 and P3 are determined as the first batch of target data shards. There is no dependency between these two data shards, so they can be processed in parallel.

[0105] Step 1044: Determine the computing resources required to process each data fragment based on the tag information carried by each data fragment.

[0106] In step 1044, the data shard refers to the multiple target data shards to be processed in parallel as determined in step 1043. This is because in step 1044, it is necessary to determine the computing resources required to process each target data shard based on the tagging information carried by the target data shards, so as to perform load balancing allocation in the future. The tagging information refers to the descriptive data attached to the data shards, which specifically includes the data source identifier, initial processing level, and shard type of the data shard.

[0107] In this embodiment of the application, firstly, for each target data fragment determined in step 1043, the tag information carried by the data fragment is read, and the type of the data fragment, including the first type of data block or the second type of data block and the data source identifier, is extracted from the tag information. Then, according to the preset computing resource mapping relationship, the fragment type and data source are converted into the corresponding memory space requirement value and processor time requirement value. Finally, these two requirement values ​​are recorded as the computing resources required by the data fragment.

[0108] In practical applications, for target data shard P1, reading its first-class marker reveals that it is a first-class data block originating from the order table. According to preset rules, each first-class data block requires 16 megabytes of memory and 0.2 seconds of processor time. Therefore, the computational resources required for data shard P1 are determined to be 16 megabytes of memory and 0.2 seconds of processor time. For target data shard P3, reading its first-class marker reveals that it is a first-class data block originating from the user table. Similarly, the computational resources required for it are determined to be 16 megabytes of memory and 0.2 seconds of processor time.

[0109] Step 1045: Based on the computing resources required by the multiple target data shards, the multiple target data shards are allocated to multiple execution threads using a load balancing algorithm, and each execution thread is allocated a corresponding memory space size and processor time quota. The multiple execution threads process the multiple target data shards to generate processing result data, which is the resource allocation result.

[0110] Here, memory space size refers to the memory capacity allocated to each execution thread for processing data slices, processor time quota refers to the time slice length allocated to each execution thread for processing data slices, and processing result data refers to the data content output by each execution thread after completing data slice processing.

[0111] Step 1045 may specifically include the following steps:

[0112] A1: Get resource information of multiple currently executing threads.

[0113] In step A1, resource information refers to the available status data of each execution thread at the current moment. Specifically, the resource information includes the remaining memory capacity and current processor load of each execution thread.

[0114] In this embodiment of the application, the running status of all execution threads in the current multi-threaded execution environment is first queried through the thread management interface. For each execution thread, its total allocated memory and the amount of memory used are collected to calculate the remaining memory capacity. At the same time, its processor utilization is collected to calculate the available processor time. Then, this information is summarized to form a resource information list of the current multiple execution threads.

[0115] In practical applications, assuming there are two execution threads in the current multi-threaded execution environment, we can find that thread T1 has a remaining memory capacity of 32 megabytes and a processor utilization rate of 50%, which means the available processor time is 50%, while thread T2 has a remaining memory capacity of 48 megabytes and a processor utilization rate of 25%, which means the available processor time is 75%, thus forming a resource information list.

[0116] A2: Combining the resource information with the computing resources required for the multiple target data shards, a resource allocation scheme is determined through a load balancing algorithm.

[0117] In step A2, the resource allocation scheme refers to the mapping relationship between the target data fragments and the execution threads, as well as the specific values ​​of the memory space size and processor time quota allocated to each execution thread.

[0118] In this embodiment of the application, the resource information list obtained in step A1 and the computing resources required for each target data shard determined in step 1044 are first used as inputs to the load balancing algorithm. Then, the load balancing algorithm is used for optimization calculation. The algorithm aims to minimize the difference in processing time between threads, allocates the target data shards one by one to the execution threads that can meet their computing resource requirements, determines the memory space size and processor time quota allocated to each execution thread, and finally outputs a complete resource allocation scheme.

[0119] In practical applications, the resource information of threads T1 and T2, along with the computational resource requirements of target data shards P1 and P3, are input into the load balancing algorithm. After calculation, the algorithm determines that data shard P1 is allocated to thread T1 and configured with 16 megabytes of memory space and 0.2 seconds of processor time, and data shard P3 is allocated to thread T2 and configured with 16 megabytes of memory space and 0.2 seconds of processor time, thus forming a resource allocation scheme.

[0120] A3: According to the resource allocation scheme, the multiple target data fragments are allocated to multiple execution threads respectively, and a corresponding memory space size and processor time quota are configured for each execution thread.

[0121] In this embodiment of the application, the resource allocation scheme determined in step A2 is first parsed to extract the mapping relationship between each target data slice and the execution thread, as well as the corresponding memory space size and processor time quota. Then, each target data slice is sent to the corresponding execution thread through the thread scheduling interface. At the same time, a memory usage limit and a processor time limit are set for each execution thread to ensure that the thread does not exceed the allocated resource limit when processing the data slice.

[0122] In practical applications, according to the resource allocation scheme, data fragment P1 is sent to thread T1 through the thread scheduling interface, and thread T1 is set with a memory space limit of 16 megabytes and a processor time limit of 0.2 seconds; at the same time, data fragment P3 is sent to thread T2, and thread T2 is set with a memory space limit of 16 megabytes and a processor time limit of 0.2 seconds.

[0123] A4: In each execution thread, the allocated target data slices are parsed and structured according to the configured memory space size and processor time quota. After each execution thread completes its processing, the corresponding processing result data is output.

[0124] In this embodiment, a data processing routine is first started in each execution thread. The routine performs processing operations according to the configured memory space size and processor time quota. The thread reads the target data fragments in the allocated memory space, parses them according to the preset data format specifications, and reassembles the parsed field content according to a unified structure. During the processing, memory usage and processor time consumption are monitored in real time to ensure that the allocated quota is not exceeded. Finally, after each thread completes processing, the generated processing result data is output to the specified storage location.

[0125] In practical applications, thread T1 parses data fragment P1 using a 16-megabyte memory space and a 0.2-second processor time quota, extracts the order amount field content, converts it into a standardized format, and outputs the processing result data R1 after processing. Similarly, thread T2 parses data fragment P3 using a 16-megabyte memory space and a 0.2-second processor time quota, extracts the username field content, converts it into a standardized format, and outputs the processing result data R3 after processing.

[0126] Step 1046: Based on the dependency relationship, combine the resource allocation results to form a set of data units that have been scheduled and processed.

[0127] In this embodiment, the processing result data output by all execution threads in step 1045 is first collected. Then, the dependency relationships recorded in the task dependency graph constructed in step 1042 are read. The processing result data is sorted and combined according to the logical order defined by the dependency relationships. For data units with dependency relationships, they are arranged in the order of upstream first and downstream last. For data units without dependency relationships, they are arranged in the order of processing completion time. Finally, all processing result data are integrated into a complete data set, which is output as a set of data units that have been scheduled and processed.

[0128] In practical applications, the processing result data R1 and R3 are collected. According to the information in the task dependency graph that data fragments P1 and P2 have a dependency relationship while P3 has no dependency relationship, since the current batch has only processed P1 and P3 and P2 has not yet been processed, R1 and R3 are temporarily stored and their dependency status is marked, and they will be finally combined after the subsequent batches have processed P2.

[0129] This application achieves data dependency identification and dynamic resource allocation in a multi-threaded environment by using a dependency-aware task scheduling mechanism, combined with dependency analysis in metadata graphs and tagging information carried by data shards. This effectively avoids data conflict issues in parallel processing and improves multi-threaded processing efficiency.

[0130] Step 105: Using a graph neural network model, perform anomaly detection and inference on the set of data units that have been scheduled and processed, so as to generate data cleaning rules.

[0131] The graph neural network model adopts an encoder-decoder architecture, specifically including a feature extraction layer composed of three layers of graph convolutional networks, a temporal fusion layer composed of gated recurrent units, a matching calculation layer implemented by an attention mechanism, and an inference layer composed of two fully connected networks. The first layer of the feature extraction layer maps the input features of each second node to a 64-dimensional hidden representation. The second layer of the graph convolutional network aggregates the information of neighboring nodes to generate 128-dimensional graph features. The third layer of the graph convolutional network further refines and generates 256-dimensional final graph features.

[0132] The temporal fusion layer adopts a bidirectional gated recurrent unit structure, taking the historical quality record of each second node as the time step input. It captures the time dependency pattern of quality changes through forward and backward propagation, generating 128-dimensional temporal features, which are then concatenated and fused with the map features to obtain 384-dimensional associated features. The matching calculation layer adopts a scaled dot product attention mechanism, converting each data unit in the data unit set that has been scheduled and processed into a query vector. It performs attention calculation with the associated features as key-value pairs to obtain the matching score between each data unit and the historical pattern as bias information. The inference layer consists of a two-layer fully connected network with a 384-dimensional input layer, a 128-dimensional hidden layer, and a 6-dimensional output layer. The output layer corresponds to the probability distribution of normal type and 5 preset abnormal types.

[0133] The training process of this model includes a pre-training phase and a fine-tuning phase. In the pre-training phase, unlabeled historical data is used to perform self-supervised learning through a mask reconstruction task. The attribute features of some nodes are randomly masked to allow the model to reconstruct the original features in order to learn the intrinsic representation of the graph structure. In the fine-tuning phase, historical datasets with anomaly labels are used for supervised learning. The cross-entropy loss function is calculated by comparing the probability distribution of anomaly types output by the model with the true labels. The model parameters are updated through the backpropagation algorithm. The training is completed when the classification accuracy on the validation set reaches more than 95% and the loss function value converges.

[0134] It should be noted that the above structure is exemplary. This application does not impose specific limitations on the internal structure design of the graph neural network model, and corresponding settings can be made according to the actual situation.

[0135] Data cleaning rules are processing instructions that include operation conditions and operation content, generated for specific anomaly types.

[0136] In this embodiment, step 105 includes the following process:

[0137] Step 1051: Input the data unit set that has been scheduled and processed and the metadata graph into the graph neural network model. Process the metadata graph through the feature extraction layer of the graph neural network model to obtain graph features. The graph features are used to characterize the dependency structure between data entities.

[0138] In step 1051, the feature extraction layer is the first processing layer in the graph neural network model. This layer is responsible for extracting structured feature information from the input metadata graph. Graph features refer to the vectorized representation generated after processing by the feature extraction layer. This representation encodes the connection relationships and dependency structures between nodes in the metadata graph.

[0139] In this embodiment, the metadata graph generated in step 102 and the data unit set processed by scheduling obtained in step 104 are first used as two inputs to the graph neural network model. The metadata graph is input to the feature extraction layer of the model. The feature extraction layer aggregates the neighbor node information of each second node in the metadata graph through graph convolution operation, extracts the attribute information of the node itself and the connection relationship information between the nodes layer by layer, and generates the embedding vector of each second node after multi-layer aggregation. The embedding vectors of all nodes together constitute the graph feature. The graph feature completely preserves the dependency relationship structure information between data entities in the metadata graph.

[0140] In practical applications, the metadata graph containing order amount field nodes, payment amount field nodes, and user name field nodes is input into the feature extraction layer. This layer extracts the features of each node through graph convolution operations, generating embedding vectors of [0.25, 0.60, 0.15] for the order amount field node, [0.30, 0.55, 0.15] for the payment amount field node, and [0.80, 0.10, 0.10] for the user name field node. These embedding vectors together constitute the graph features.

[0141] Step 1052: Through the temporal fusion layer of the graph neural network model, combined with the graph features and historical quality records in the metadata graph, a correlation feature is generated. The correlation feature is used to characterize the correlation pattern of data quality and dependency within a specific business period.

[0142] In step 1052, the temporal fusion layer is the second processing layer in the graph neural network model. This layer is responsible for fusing graph features with time series data. The associated features refer to the comprehensive feature representation generated after processing by the temporal fusion layer. This representation encodes the association pattern between data quality changes and dependencies within a specific business period.

[0143] A specific business period refers to the time interval corresponding to historical quality records. This time interval is divided into multiple consecutive time periods according to the business operation cycle, such as peak business periods or off-peak business periods. The association pattern refers to the regular information learned by the graph neural network model from graph features and historical quality records through the time series fusion layer. This information reflects the correspondence between data quality changes and dependencies between data entities within a specific business period.

[0144] In this embodiment, the graph features generated in step 1051 are first used as the main input of the temporal fusion layer. At the same time, the historical quality records carried by each second node are read from the metadata graph. These historical quality records are a set of quality scores stored in time series form. Then, the temporal fusion layer extracts the temporal features of the historical quality records of each second node through a recurrent neural network structure to generate a temporal feature vector for each node. Next, the temporal feature vector of each node is concatenated and fused with the graph feature vector of that node. Finally, the association features of each node are generated by mapping through a fully connected layer. These association features comprehensively reflect the association pattern between the data quality change pattern of the node and its position in the dependency graph during a specific business period.

[0145] In practical applications, the historical quality records of the order amount field node read from the metadata graph are [85, 88, 92], the historical quality records of the payment amount field node are [82, 86, 89], and the historical quality records of the user name field node are [95, 94, 96]. The temporal fusion layer first extracts the temporal features of each node through a recurrent neural network, generating the temporal feature vector of the order amount field node as [0.40, 0.30, 0.30], and the temporal feature vector of the payment amount field node as [0.40, 0.30, 0.30]. The vector is [0.35, 0.35, 0.30]. The time feature vector of the user name field node is [0.45, 0.25, 0.30]. Then, the time feature vector of each node is concatenated with the graph feature vector generated in step 1051. After concatenation, the order amount field node is [0.25, 0.60, 0.15, 0.40, 0.30, 0.30]. After mapping through a fully connected layer, the associated feature vector of this node is generated as [0.28, 0.52, 0.20].

[0146] Step 1053: By comparing the set of data units processed by the scheduling and the associated features through the matching calculation layer of the graph neural network model, deviation information is obtained. The deviation information is used to indicate the degree of deviation between the current data state and the historical association pattern.

[0147] In step 1053, the matching calculation layer is the third processing layer in the graph neural network model. This layer is responsible for comparing the current data unit with the historical association pattern. The deviation information refers to the quantitative value output by the matching calculation layer, which reflects the degree of difference between the actual state of the current data unit and the expected state learned based on historical data.

[0148] In this embodiment, the data unit set obtained in step 104 after scheduling is first used as one of the inputs to the matching calculation layer, and the association features of each node generated in step 1052 are used as another input to the matching calculation layer. For each data unit in the data unit set after scheduling, the matching calculation layer extracts the field identifier corresponding to the data unit, and then finds the feature vector corresponding to the field in the association features based on the field identifier. Then, the actual content of the data unit is converted into the current state vector through the embedding layer. Finally, the cosine similarity between the current state vector and the association feature vector is calculated, and the similarity value is converted into a deviation score. The lower the deviation score, the greater the deviation between the current data state and the historical association pattern.

[0149] In practical applications, assuming that the data unit set processed by the schedule contains data units with an order amount field and an actual content value of 95, the matching calculation layer first converts the data unit into a current state vector of [0.30, 0.55, 0.15], then finds the associated feature vector of the order amount field node as [0.28, 0.52, 0.20], calculates the cosine similarity between the two vectors, and obtains a similarity value of 0.98, which is converted into a deviation score of 0.02. This deviation score indicates that the current data state deviates relatively little from the historical association pattern.

[0150] Step 1054: Determine the anomaly type based on the deviation information and the graph features through the inference layer of the graph neural network model.

[0151] In step 1054, the inference layer is the fourth processing layer in the graph neural network model. This layer is responsible for comprehensively judging the anomaly type based on the deviation information and graph features. The anomaly type refers to the result of classifying the abnormal situations existing in the data unit. The anomaly type specifically includes categories such as format anomaly, numerical drift anomaly, and null value anomaly.

[0152] In this embodiment, the deviation information generated in step 1053 and the map features generated in step 1051 are first input into the inference layer. The inference layer performs joint processing on the input deviation information and map features through a multilayer perceptron structure. For each data unit whose deviation information exceeds a preset threshold, the position information of the data unit in the map features and the features of other nodes associated with it are further analyzed. The abnormal type of the data unit is output through a multi-classifier. For data units whose deviation information does not exceed the threshold, the inference layer outputs a normal type identifier.

[0153] In practical applications, if the deviation score of the order amount field data unit is 0.02, which does not exceed the preset threshold of 0.10, the inference layer outputs a normal type. If the deviation score of another field data unit is 0.25, which exceeds the threshold of 0.10, the inference layer analyzes the position information of this field in the spectral features and determines that its abnormal type is numerical drift anomaly.

[0154] Step 1055: Generate data cleaning rules based on the anomaly type and the node information in the metadata graph.

[0155] In this embodiment of the application, the anomaly types of each data unit determined in step 1054 are first collected. For each data unit that is determined to be abnormal, the node information of the second node corresponding to the data unit is read from the metadata graph. The node information includes field name, field type and dependency information. Then, the corresponding rule template is matched from the preset rule template library according to the anomaly type and node information. The specific parameters in the node information are filled into the template to generate complete operation conditions and operation content. Finally, the generated data cleaning rules are stored in the rule library for subsequent cleaning operation steps to call.

[0156] In practical applications, for a field identified as having numerical drift anomalies, its field name ("Order Amount" and field type ("Number") are retrieved from the metadata graph. The corresponding rule template for numerical drift anomalies is then matched from the rule template library. The template content states that if a field value exceeds the range of the historical mean plus or minus three standard deviations, it should be replaced with the historical mean. After filling the field name and field type into the template, a complete data cleaning rule is generated. The rule content states that if the value of the "Order Amount" field exceeds the range of the historical quality record mean plus or minus three standard deviations, the field value should be replaced with the historical quality record mean.

[0157] This application uses a hierarchical processing model of graph neural network to extract dependency features from metadata graphs and integrate historical quality records, thereby realizing intelligent anomaly detection and rule generation for the data unit set that has been scheduled and processed, enabling the cleaning rules to adaptively match the business context of the current data.

[0158] Step 106: In the multi-threaded execution environment established by the multi-threaded task scheduling, perform a business-adaptive data cleaning operation on the data unit set that has been scheduled and processed based on the data cleaning rules.

[0159] In step 106, the business-adaptive data cleaning operation refers to the process of performing differentiated processing on the set of scheduled data units in an established multi-threaded execution environment according to the data cleaning rules generated in step 105. This process can automatically match the corresponding cleaning rules and execute cleaning actions based on the business context and quality characteristics of the data units.

[0160] In this embodiment, the status information of the multi-threaded execution environment established in step 104 is first obtained. This multi-threaded execution environment includes multiple execution threads that have completed resource configuration. Then, the data cleaning rules for the current batch of data unit sets that have been scheduled and processed are read from the rule generation result in step 105. Next, the data cleaning rules are distributed to each execution thread in the multi-threaded execution environment. Then, each execution thread applies the corresponding data cleaning rules to each data unit in the allocated data unit set in parallel, performing cleaning operations including outlier correction, format normalization, and null value filling. Finally, the data units after the cleaning operations are completed are recombined and output as the final business-usable data set.

[0161] In this embodiment, after step 106, the method further includes the following steps:

[0162] B1: Obtain quality evaluation information for the cleaned data unit set.

[0163] In step B1, the cleaned data unit set refers to the data unit set output after the cleaning operation in step 106; the quality evaluation information refers to the quantitative indicators generated after evaluating the quality of each data unit in the cleaned data unit set. The quality evaluation information specifically includes completeness indicators, accuracy indicators, and timeliness indicators.

[0164] In this embodiment of the application, the cleaned data unit set output in step 106 is first received. Then, for each data unit in the set, its integrity index (i.e., the proportion of non-empty fields), accuracy index (i.e., the degree to which the data format and content conform to the specifications), and timeliness index (i.e., the difference between the data arrival time and the business requirement time) are calculated according to the preset quality assessment rules. Finally, the quality assessment results of all data units are summarized to form the quality evaluation information of the cleaned data unit set.

[0165] B2: Compare the quality evaluation information with the corresponding historical quality records in the metadata map to obtain quality difference information.

[0166] In step B2, the quality difference information refers to the comparison between the quality evaluation information of the cleaned data unit set and the historical quality records stored in the metadata map. This result reflects the quality changes brought about by this cleaning operation.

[0167] In this embodiment, the historical quality records corresponding to each data unit in the cleaned data unit set are first read from the metadata graph. These historical quality records are the quality score sequences of the field at historical time points before cleaning. Then, each indicator in the quality evaluation information of the cleaned data unit set is compared with the latest record in the corresponding historical quality records, and the difference between each indicator is calculated. Finally, these differences are summarized to form quality difference information, where a positive difference indicates a quality improvement and a negative difference indicates a quality decline.

[0168] B3: Based on the decision tree model, analyze the correlation between the quality difference information and the data cleaning rules, and generate rule evaluation results.

[0169] In step B3, the rule evaluation result refers to the evaluation result of the decision tree model on the effectiveness of the data cleaning rules.

[0170] In this embodiment, the quality difference information generated in step B2 and the data cleaning rules generated in step 105 are first used as inputs to the decision tree model. This decision tree model is a pre-trained classification or regression model, and its input features include: the type of cleaning rule applied, such as numerical drift correction or null value imputation; the field features to which the rule is applied, such as field type and historical quality volatility; and the quality difference information generated after applying the rule, such as changes in completeness and accuracy indicators.

[0171] The model outputs a score for the effectiveness of the rule, such as a value between 0 and 1, where 1 represents very effective. The decision tree model uses its tree structure to evaluate input features layer by layer. For example, it first determines whether the "quality difference is positive"; if so, it proceeds to the left subtree, then determines whether the "rule type is numerical drift correction," and so on, finally outputting a effectiveness score at the leaf node. In this way, the model can analyze which cleaning rule is most effective in improving data quality under what data features. The final output is a rule evaluation result containing the rule identifier and its corresponding effectiveness score.

[0172] B4: Update the historical quality records in the metadata graph based on the evaluation results of the rules.

[0173] In this embodiment of the application, the rule evaluation results generated in step B3 are first parsed to extract the rule identification information related to this cleaning. Then, for each data unit in the cleaned data unit set, the quality evaluation information corresponding to the data unit is combined with the current timestamp to form a new quality record. Finally, the second node corresponding to the data unit is found in the metadata graph, and the new quality record is added to the end of the historical quality record sequence of the second node to complete the dynamic update of the metadata graph.

[0174] B5: Based on the reinforcement learning algorithm, combined with the updated historical quality records and the rule evaluation results, adjust the weights of the feature extraction layer in the graph neural network model, and based on the adjusted weights, perform feature extraction operations on the metadata graph in the next round of data cleaning.

[0175] In this embodiment, the updated historical quality records from step B4 and the rule evaluation results generated in step B3 are first used as inputs to the reinforcement learning algorithm. This reinforcement learning algorithm can be implemented based on the policy gradient method, where: Environment state: a snapshot of the current data environment consisting of updated historical quality records and data features from the metadata graph; Action: the specific operation that adjusts the weight parameters of the feature extraction layer in the graph neural network model; Reward: the effectiveness score in the rule evaluation results generated in step B3. A higher score indicates that the "action" taken in the previous round resulted in a better cleaning effect, thus a higher reward is given.

[0176] Reinforcement learning algorithms predict and execute an action based on the current state through a policy network. The environment then provides a new state and reward. Based on the received reward signal, the algorithm updates its policy network using policy gradient ascent, aiming to maximize the expected value of future cumulative rewards. Through this closed-loop optimization process of "trial-evaluation-adjustment," the weights of the model's feature extraction layer are continuously optimized, enabling it to better extract features from the metadata graph that contribute to generating efficient cleaning rules. Finally, the adjusted weights are applied to the graph neural network model to perform feature extraction operations on the metadata graph in the next round of data cleaning, achieving continuous adaptive evolution of the model.

[0177] This application introduces a feedback mechanism to evaluate data cleaning rules by comparing quality assessment information with historical quality records, and dynamically optimizes the weight parameters of the graph neural network model through reinforcement learning algorithms, forming a closed-loop optimization of the data cleaning process, so that the cleaning effect can be continuously improved as data accumulates.

[0178] Figure 3 A schematic diagram of a business adaptive data cleaning system based on multi-threaded parallelism provided in this application embodiment is shown below. Figure 3 As shown, the system includes:

[0179] The acquisition module 31 is used to acquire the data to be cleaned, the lineage information of the data to be cleaned, and the quality score information of historical cleaning tasks.

[0180] The forming module 32 is used to form a metadata map based on the blood relationship information and the quality score information.

[0181] The sharding module 33 is used to preprocess and shard the data to be cleaned based on the data characteristics recorded in the metadata map, and generate a set of data shards carrying tag information.

[0182] The execution module 34 is used to perform dependency-aware multi-threaded task scheduling based on the data shard set. The multi-threaded task scheduling analyzes the dependency relationship of data units in the data shard set based on the metadata graph, allocates multiple data units without dependency relationship to different execution threads in parallel, allocates computing resources to each execution thread based on the marking information, and obtains the data unit set after scheduling based on the resource allocation result.

[0183] The detection module 35 is used to perform anomaly detection and inference on the set of scheduled data units using a graph neural network model, so as to generate data cleaning rules.

[0184] The cleaning module 36 is used to perform business-adaptive data cleaning operations on the data unit set that has been scheduled and processed, based on the data cleaning rules, in the multi-threaded execution environment established by the multi-threaded task scheduling.

[0185] The business adaptive data cleaning system based on multi-threaded parallelism in this application is used to implement the aforementioned business adaptive data cleaning method based on multi-threaded parallelism. Therefore, the specific implementation of the business adaptive data cleaning system based on multi-threaded parallelism can be found in the embodiment section of the business adaptive data cleaning method based on multi-threaded parallelism mentioned above. The specific implementation can be referred to the description of the corresponding embodiments, which will not be repeated here.

[0186] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for executing the computer program to implement the steps of any of the above-described multi-threaded parallel business adaptive data cleaning methods.

[0187] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-described multi-threaded parallel business adaptive data cleaning methods.

[0188] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as USB flash drives, read-only memory, random access memory, portable hard drives, magnetic disks, or optical disks.

[0189] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above embodiments of the business adaptive data cleaning method based on multi-threaded parallelism.

[0190] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0191] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0192] The above provides a detailed description of a multi-threaded parallel adaptive data cleaning method and system provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A business-adaptive data cleaning method based on multi-threaded parallelism, characterized in that, include: Collect the data to be cleaned, the lineage information of the data to be cleaned, and the quality score information of historical cleaning tasks; Based on the blood relationship information and the quality score information, a metadata map is formed; Based on the data characteristics recorded in the metadata map, the data to be cleaned is preprocessed and fragmented to generate a set of data fragments carrying tag information; Dependency-aware multi-threaded task scheduling is performed based on the data shard set. The multi-threaded task scheduling analyzes the dependency relationship of data units in the data shard set based on the metadata graph, allocates multiple data units without dependency relationship to different execution threads in parallel, and allocates computing resources to each execution thread based on the marking information. Based on the resource allocation result, a set of data units that have been scheduled and processed is obtained. Using a graph neural network model, anomaly detection and inference are performed on the data unit set that has been scheduled and processed, so as to generate data cleaning rules; In the multi-threaded execution environment established by the multi-threaded task scheduling, a business-adaptive data cleaning operation is performed on the data unit set that has been scheduled and processed based on the data cleaning rules.

2. The method according to claim 1, characterized in that, The process involves performing dependency-aware multi-threaded task scheduling based on the data shard set. This multi-threaded task scheduling analyzes the dependencies between data units in the data shard set based on the metadata graph, allocates multiple data units without dependencies to different execution threads in parallel, and allocates computing resources to each execution thread based on the tagging information. Based on the resource allocation results, a set of data units that has undergone scheduling processing is obtained, including: Read the metadata graph to analyze the dependency relationship of each data shard in the data shard set in the metadata graph, and identify multiple related data shards with dependency relationships; Based on the aforementioned dependencies, a task dependency graph is constructed, which includes multiple nodes and edges connecting the nodes, with each node corresponding to a data shard. Using a topological sorting algorithm, the task dependency graph is analyzed to determine multiple target data fragments to be processed in parallel, wherein there are no dependencies between the multiple target data fragments; Based on the tagging information carried by each data fragment, determine the computing resources required to process each data fragment; Based on the computing resources required for the multiple target data shards, a load balancing algorithm is used to allocate the multiple target data shards to multiple execution threads, and each execution thread is allocated a corresponding memory space size and processor time quota. The multiple execution threads process the multiple target data shards and generate processing result data, which is the resource allocation result. Based on the dependencies, the resource allocation results are combined to form a set of data units that have undergone scheduling processing.

3. The method according to claim 2, characterized in that, The computational resources required for the multiple target data shards are allocated to multiple execution threads using a load balancing algorithm. Each execution thread is allocated a corresponding memory space and processor time quota. The multiple execution threads process the multiple target data shards to generate processing result data, including: Get resource information for multiple currently executing threads; By combining the resource information with the computing resources required for the multiple target data shards, a resource allocation scheme is determined through a load balancing algorithm; According to the resource allocation scheme, the multiple target data fragments are allocated to multiple execution threads respectively, and each execution thread is configured with a corresponding memory space size and processor time quota; In each execution thread, the allocated target data slices are parsed and structured according to the configured memory space size and processor time quota. After each execution thread completes its processing, the corresponding processing result data is output.

4. The method according to claim 1, characterized in that, The method of using a graph neural network model to perform anomaly detection and inference on the scheduled data unit set to generate data cleaning rules includes: The set of data units that have been scheduled and processed and the metadata graph are input into the graph neural network model. The feature extraction layer of the graph neural network model is used to process the metadata graph to obtain graph features. The graph features are used to characterize the dependency structure between data entities. By combining the graph features and historical quality records in the metadata graph through the temporal fusion layer of the graph neural network model, association features are generated. These association features are used to characterize the association patterns of data quality and dependencies within a specific business period. By comparing the set of data units processed by the scheduling and the associated features through the matching calculation layer of the graph neural network model, deviation information is obtained. The deviation information is used to indicate the degree of deviation between the current data state and the historical association pattern. The anomaly type is determined by the inference layer of the graph neural network model based on the deviation information and the graph features. Data cleaning rules are generated based on the anomaly type and the node information in the metadata graph.

5. The method according to claim 1, characterized in that, The process of forming a metadata map based on the kinship information and the quality score information includes: The bloodline relationship information is parsed to obtain table information, field information, and dependency information between the table information and field information; Based on the table information, field information, and dependency information, a hierarchical structure diagram containing a first node and a second node is constructed, wherein the first node corresponds to the table information and the second node corresponds to the field information; The quality score information is associated with the corresponding field information to form a historical quality record corresponding to the field information, and the historical quality record is added as dynamic information to the corresponding second node in the hierarchical structure diagram. Using a community detection algorithm, the hierarchical structure graph is traversed to identify multiple associated fields with the same upstream dependency. These multiple associated fields are then grouped into the same group, and grouping information is added to each group in the hierarchical structure graph. Based on the grouping information and the dependency information, the dependency distance between any two second nodes in the hierarchical graph is determined by a graph embedding model. Based on the dependency distance, the grouping information, and the second node carrying historical quality records, a metadata graph is generated.

6. The method according to claim 1, characterized in that, The process of preprocessing and fragmenting the data to be cleaned based on the data features recorded in the metadata map, generating a set of data fragments carrying tag information, includes: Based on the historical quality records of the second node corresponding to the data source in the metadata map, the initial processing level of data units from different sources in the data to be cleaned is determined; Based on the data characteristics recorded in the metadata graph and in combination with the initial processing level, the data to be cleaned is divided into processing data streams of different priorities; For the first priority processing data stream, a preset first sharding strategy is used to cut it into multiple first-class data blocks, and a first-class tag containing the data source, the initial processing level and the first sharding type is attached to each first-class data block; For the second priority data stream, a preset second sharding strategy is used to cut and reconstruct it, generating multiple second type data blocks. Each second type data block is attached with a second type tag containing the data source, the initial processing level, and the second sharding type. The second priority is lower than the first priority. The first type of data block with the first type of tag attached and the second type of data block with the second type of tag attached are merged to form a data fragment set carrying tag information.

7. The method according to claim 1, characterized in that, After performing a business-adaptive data cleaning operation on the scheduled data unit set based on the data cleaning rules, the process further includes: Obtain quality evaluation information for the cleaned data unit set; By comparing the quality evaluation information with the corresponding historical quality records in the metadata map, quality difference information is obtained; Based on the decision tree model, the correlation between the quality difference information and the data cleaning rules is analyzed, and rule evaluation results are generated. Based on the evaluation results of the rules, update the historical quality records in the metadata graph; Based on the reinforcement learning algorithm, and combining the updated historical quality records and the rule evaluation results, the weights of the feature extraction layer in the graph neural network model are adjusted. Based on the adjusted weights, the feature extraction operation of the metadata graph is performed in the next round of data cleaning.

8. A business adaptive data cleaning system based on multi-threaded parallelism, characterized in that, include: The data acquisition module is used to acquire the data to be cleaned, the lineage information of the data to be cleaned, and the quality score information of historical cleaning tasks. A forming module is used to form a metadata map based on the blood relationship information and the quality score information; The sharding module is used to preprocess and shard the data to be cleaned based on the data features recorded in the metadata map, and generate a set of data shards carrying tag information. An execution module is used to perform dependency-aware multi-threaded task scheduling based on the data shard set. The multi-threaded task scheduling analyzes the dependency relationship of data units in the data shard set based on the metadata graph, allocates multiple data units without dependency relationship to different execution threads in parallel, allocates computing resources to each execution thread based on the marking information, and obtains a set of data units after scheduling based on the resource allocation result. The detection module is used to perform anomaly detection and inference on the set of scheduled data units using a graph neural network model, so as to generate data cleaning rules. The cleaning module is used to perform business-adaptive data cleaning operations on the set of scheduled data units based on the data cleaning rules in the multi-threaded execution environment established by the multi-threaded task scheduling.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor, configured to implement the steps of the multi-threaded parallel business adaptive data cleaning method as described in any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, enables the implementation of the business adaptive data cleaning method based on multi-threaded parallelism as described in any one of claims 1 to 7.

Images