Distributed system mining data cleaning method

Abstract

The invention discloses a cleaning method for mining data of a distributed system, and belongs to the technical field of distributed data cleaning, which comprises the following steps: the system is connected with multiple data sources by adopting distributed nodes, extracts meta-information of the data sources, matches with a data field semantic knowledge base, generates a semantic mapping table, analyzes the meta-information of the data sources by utilizing a graph neural network, and cleans the data sources. Mining an association relationship in the metadata information, performing semantic analysis on the association relationship according to a semantic mapping table to generate a business rule, determining a target data field according to a semantic analysis result, generating a cleaning strategy, converting the business rule into a cleaning rule corresponding to each data field based on the cleaning strategy, and storing the cleaning rule in the target data field. An improved genetic algorithm is adopted to establish a dual-mode collaborative cleaning model, distributed nodes are divided into edge nodes and center nodes, and data cleaning is performed on different types of data, so that the accuracy and dynamics of cleaning rule generation can be improved, the real-time performance and the global consistency can be effectively balanced, and the system load is reduced.

Description

Technical Field

[0001] This application belongs to the field of distributed data cleaning technology, specifically, it relates to a method for cleaning data mining data in a distributed system. Background Technology

[0002] With the rapid development of big data and IoT technologies, the scale of data is growing explosively. Distributed systems, due to their massive data storage, parallel processing, and cross-node collaboration capabilities, have become the mainstream architecture in the field of data mining. Data cleaning, as a crucial pre-processing step in data mining, aims to remove noise, outliers, and redundant information from the data, and to achieve data format standardization and quality optimization. This directly determines the accuracy and reliability of subsequent data analysis and modeling. In current application scenarios, data sources exhibit significant heterogeneity, and some data needs to meet real-time processing requirements while others require cross-node collaborative verification. Traditional centralized cleaning architectures are no longer able to cope with the pressure of cleaning massive amounts of heterogeneous data, leading to the emergence of distributed data cleaning architectures.

[0003] Although distributed data cleaning architectures have been applied to some extent, problems such as semantic inconsistencies in heterogeneous data sources, insufficient mining of metadata relationships, poor adaptability of cleaning strategies, and low efficiency of distributed node collaboration still seriously restrict the quality and efficiency of data cleaning, becoming key technical bottlenecks that urgently need to be solved in the field of distributed system data mining. Summary of the Invention

[0004] To address the aforementioned problems and technical deficiencies, this application adopts the following technical solution: a method for cleaning data from a distributed system, comprising the following steps: The system uses distributed nodes to connect with multiple data sources, extracts the metadata of the data sources, matches it with the semantic knowledge base of the data domain, and generates a semantic mapping table. By using graph neural networks to analyze the metadata of data sources, we can mine the relationships in the metadata information and generate business rules by semantic parsing of the relationships based on the semantic mapping table. The target data domain is determined based on the semantic parsing results and a cleaning strategy is generated. Based on the cleaning strategy, the business rules are transformed into cleaning rules corresponding to each data domain. An improved genetic algorithm is used to establish a dual-mode collaborative cleaning model, which divides distributed nodes into edge nodes and central nodes to clean different types of data.

[0005] Preferably, the system is deployed on a distributed architecture, including multiple edge nodes and at least one central node. The edge nodes are connected to heterogeneous data sources, support multiple data source protocols, and the extracted metadata is stored in a distributed metadata repository in a standardized format.

[0006] Furthermore, the metadata includes: structural metadata, attribute metadata, and business metadata.

[0007] Furthermore, the generation of the semantic mapping table includes: A domain-specific semantic knowledge base is built by generating annotations from domain experts or by extending from public ontology, and the data domain concept is represented using the OWL language. A BERT-based semantic similarity algorithm is used to perform similarity matching calculations between the attributes and descriptions in the metadata and the data domain concepts in the knowledge base, and the calculation results are obtained. A preset matching threshold is set, and the calculation result is compared with the matching threshold. Based on the comparison result, a semantic mapping table is generated using the concepts of meta-information attributes, semantic tags, and data domains. The semantic tags include domain-specific business tags.

[0008] Preferably, the semantic parsing includes: A graph attention network model is used to transform metadata into a two-layer graph structure that can be dynamically updated. The lower layer is the attribute association layer, and the upper layer is the business association layer. Metadata attributes are used as nodes, and the relationships between attributes are used as edges. Edge feature weights are calculated by weighting the co-occurrence frequency of metadata with semantic similarity, and hidden relationships in metadata are mined based on edge feature weights. The relationships output by the graph attention network are mapped to a semantic mapping table, and the semantic labels in the semantic mapping table are used to parse them into business rules.

[0009] Furthermore, based on the semantic parsing results, a clustering algorithm is used to divide the data into three-dimensional data domains according to data type, business scenario, and quality requirements, and a dedicated dynamic cleaning strategy is generated for each data domain. Based on a dynamic cleaning strategy and combined with the tags in the semantic mapping table associated with each business rule, the business rules are transformed into cleaning rules that can be executed for each data domain. The cleaning rules are represented in a specific language for each data domain.

[0010] Furthermore, the edge nodes deploy a lightweight cleaning engine that processes data with high real-time requirements based on basic rules generated by an improved genetic algorithm, performing format standardization, simple anomaly removal, and local data caching. The central node deploys a full-version cleaning engine that processes data requiring cross-node collaboration based on globally optimal rules, performing semantic deduplication, complex anomaly correction, and cross-domain data consistency verification.

[0011] Furthermore, the dual-mode collaborative cleaning model incorporates a collaborative mechanism to achieve a closed-loop process of edge preprocessing, center optimization, and edge feedback. The central node monitors the status of the edge nodes through heartbeat packets. When the processing pressure on the edge nodes exceeds the threshold, it dynamically allocates some rule calculation tasks to the central node.

[0012] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the content of the distributed system data cleaning method described above.

[0013] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the content of the distributed system data cleaning method as described above.

[0014] Compared to existing technologies, the beneficial effects of this application are as follows: (1) This application improves the semantic alignment of heterogeneous data sources: by constructing a domain-specific semantic knowledge base, the three-dimensional meta-information is matched with the knowledge base concepts to generate a semantic mapping table containing domain-specific business tags, which provides a semantic basis for subsequent business rule parsing and cleansing rule transformation; (2) This application adopts graph attention network, calculates edge feature weights by weighting the co-occurrence frequency of metadata and semantic similarity, mines the hidden business associations behind metadata, and parses them into business rules by combining semantic mapping table. Then, by dividing the data domain, it generates exclusive dynamic cleaning strategies, transforms the business rules into cleaning rules with specific language representations of each data domain, and improves the accuracy and dynamism of cleaning rule generation. (3) This application constructs a dual-mode collaborative cleaning model by improving the genetic algorithm and combining it with a closed-loop collaborative mechanism. At the same time, the central node can dynamically allocate tasks according to the pressure threshold of the edge nodes, forming a closed-loop process of edge preprocessing, central optimization, and edge feedback, which effectively balances real-time performance and global consistency and reduces system load. Attached Figure Description

[0015] In the attached diagram: Figure 1 This is a schematic diagram of the method steps in an embodiment of this application. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments. Generally, the components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Example

[0017] like Figure 1As shown, a method for cleaning data from a distributed system includes the following steps: The system uses distributed nodes to connect with multiple data sources, extracts the metadata of the data sources, matches it with the semantic knowledge base of the data domain, and generates a semantic mapping table. The system is deployed on a distributed architecture, including multiple edge nodes and at least one central node. The edge nodes are connected to heterogeneous data sources and support multiple data source protocols. After the metadata is extracted, the metadata information is stored in a distributed metadata repository in a standardized format.

[0018] Meta-information includes: structural meta-information, attribute meta-information, and business meta-information.

[0019] The generation of the semantic mapping table includes: A domain-specific semantic knowledge base is built by generating annotations from domain experts or by extending from public ontology, and the data domain concept is represented using the OWL language. A BERT-based semantic similarity algorithm is used to perform similarity matching calculations between the attributes and descriptions in the metadata and the data domain concepts in the knowledge base, and the calculation results are obtained. A preset matching threshold is set, and the calculation result is compared with the matching threshold. Based on the comparison result, a semantic mapping table is generated using the concepts of meta-information attributes, semantic tags, and data domains. The semantic tags include domain-specific business tags.

[0020] By using graph neural networks to analyze the metadata of data sources, we can mine the relationships in the metadata information and generate business rules by semantic parsing of the relationships based on the semantic mapping table. Semantic parsing includes: A graph attention network model is used to transform metadata into a two-layer graph structure that can be dynamically updated. The lower layer is the attribute association layer, and the upper layer is the business association layer. Metadata attributes are used as nodes, and the relationships between attributes are used as edges. Edge feature weights are calculated by weighting the co-occurrence frequency of metadata with semantic similarity, and hidden relationships in metadata are mined based on edge feature weights. The relationships output by the graph attention network are mapped to a semantic mapping table, and the semantic labels in the semantic mapping table are used to parse them into business rules.

[0021] Based on the parsing results, the target data domain is determined and a cleaning strategy is generated. Based on the cleaning strategy, the business rules are transformed into cleaning rules corresponding to each data domain. Based on the semantic parsing results, a clustering algorithm is used to divide the data into three-dimensional data domains according to data type, business scenario and quality requirements, and a dedicated dynamic cleaning strategy is generated for each data domain. Based on a dynamic cleaning strategy and combined with the tags in the semantic mapping table associated with each business rule, the business rules are transformed into cleaning rules that can be executed for each data domain. The cleaning rules are represented in a specific language for each data domain.

[0022] An improved genetic algorithm is used to establish a dual-mode collaborative cleaning model, which divides distributed nodes into edge nodes and central nodes to clean different types of data.

[0023] A lightweight cleaning engine is deployed on edge nodes. Based on the basic rules generated by the improved genetic algorithm, it processes data with high real-time requirements, performs format standardization, simple anomaly removal, and local data caching. The central node deploys a full-version cleaning engine that processes data requiring cross-node collaboration based on globally optimal rules, performing semantic deduplication, complex anomaly correction, and cross-domain data consistency verification.

[0024] The dual-mode collaborative cleaning model has a collaborative mechanism to realize a closed-loop process of edge preprocessing, center optimization, and edge feedback; The central node monitors the status of the edge nodes through heartbeat packets. When the processing pressure on the edge nodes exceeds the threshold, it dynamically allocates some rule calculation tasks to the central node.

[0025] The dual-mode collaborative cleaning model incorporates a dual-track mechanism of risk assessment and compliance review, including: A privacy risk assessment should be conducted before the cleaning process is carried out; The cleaning process integrates dynamic desensitization functionality; After the cleaning is completed, a quality traceability report containing data relationships is generated. Example

[0026] When this application is applied to the multi-source data collaborative management system of smart parks, edge nodes establish stable connections with heterogeneous data sources through preset protocol adaptation modules, video surveillance equipment transmits video frame metadata through the RTSP protocol, environmental sensors periodically upload collected data through the MQTT protocol, access control systems push access records in real time through the TCP / IP protocol, and the business management platform provides structured data through the HTTP interface. Each edge node has a built-in metadata extraction engine that extracts three types of metadata for different data source types. Component information: Video surveillance equipment outputs "Device ID - Acquisition Time - Frame Number - Resolution - Bitrate"; The environmental sensor outputs "Sensor ID - Acquisition Time - Parameter Type - Value - Unit"; The access control system outputs "Access ID - Access Time - Personnel ID - Card Type - Access Status"; The business platform outputs "Business Type - Data ID - Creation Time - Field Name - Field Type - Field Length"; Attribute metadata: The attribute metadata for the temperature sensor is "Value: Data type float, Value range -20~85℃, Accuracy 0.1℃, Update frequency 10s / time"; The attribute metadata for the access control record is "Personnel ID: Data type string, ID number, Not null constraint"; Business metadata: The business metadata of temperature sensor data is related to smart park environmental monitoring and air conditioning control, which is used to determine whether the indoor and outdoor temperature difference meets the energy-saving threshold. The business metadata of access control records is related to personnel access permission management and attendance statistics, which is used to filter valid access records. Edge nodes upload the extracted metadata to the distributed metadata repository in a standardized format, while simultaneously retaining a cached copy locally.

[0027] For the four core business domains of environmental monitoring, personnel management, equipment operation and maintenance, and energy consumption statistics, multiple core concepts and relationships are labeled. For example, the environmental monitoring domain is labeled with "temperature data - normal threshold - abnormal threshold - associated humidity data". Based on the public ontology OWL-S, it extends and adds common concepts. It uses the OWL language to encode all concepts and relationships into knowledge base files, which are deployed on the knowledge base server of the central node and support edge nodes to call them through RPC interfaces.

[0028] Edge nodes access the semantic knowledge base of the central node, employing a BERT-based semantic similarity algorithm to match and calculate the attribute names and descriptions in the metadata with the concepts in the knowledge base. Input the "Value (float, -20~85℃)" and "Business Description: Environmental Monitoring - Air Conditioning Control Related Data" from the temperature sensor metadata. The input text is converted into a 768-dimensional vector, and cosine similarity is calculated with concept vectors such as "Environmental Monitoring Domain - Temperature Data - Numerical Attributes (float, -20~85℃)" and "Air Conditioning Control Related Data - Environmental Parameters" in the knowledge base to obtain similarity values; The preset matching threshold is 0.8. The calculation results are compared with the threshold to filter out concepts with a similarity of ≥0.8. Generate a semantic mapping table, where each record contains three elements: "meta-information attributes - semantic tags - data domain concepts".

[0029] The central node deploys a graph attention network (GAT) model, reads metadata uploaded by all edge nodes from a distributed metadata repository, and constructs a dynamically updated two-layer graph structure. The graph structure supports dynamic updates. When a new data source is added, attribute nodes such as charging pile ID and charging power, as well as charging management business nodes, are automatically added, and related edges are established.

[0030] Calculate the edge feature weights, count the number of times two attributes appear simultaneously in the past 24 hours as the co-occurrence frequency, and call the BERT algorithm to calculate the similarity of attribute descriptions as semantic similarity. Based on edge feature weights, a hidden correlation was discovered that "energy consumption values ​​are positively correlated with air conditioner operating time"; the edge feature weights of "personnel ID" and "office area temperature" revealed a hidden correlation that "temperature in densely populated areas is higher than in open areas".

[0031] The relationships (including explicit and implicit relationships) output by the graph attention network are mapped to a semantic mapping table and then parsed using semantic labels: The explicit association of "temperature value - humidity value" with the semantic tag "environmental monitoring - real-time parameters" is parsed into a business rule: in environmental monitoring data, temperature value and humidity value must be collected simultaneously, and if either parameter is missing, it is marked as incomplete data; The hidden association "Energy Consumption Value - Air Conditioner Operating Status" is mapped to the semantic tag "Energy Consumption Management - Device Association", which is parsed as a business rule: In the energy consumption data, when the air conditioner operating status is 'on', the energy consumption value should be ≥ the energy consumption threshold; otherwise, it is judged as abnormal data. The hidden association "Personnel ID - Office Area Temperature" is mapped to the semantic tag "Personnel Management - Environment Association", which is parsed as a business rule: In the same office area, when the number of personnel IDs passing through is greater than or equal to the preset number of people, the temperature value should be less than or equal to the air conditioning set threshold. If it exceeds the threshold, it is marked as data that needs to be adjusted. The generated business rules include "rule ID - associated attributes - semantic tags - rule description - priority".

[0032] The K-Means clustering algorithm is used to divide the data domain based on the three dimensions of "data type-business scenario-quality requirements".

[0033] Generate a dedicated dynamic cleaning strategy for each data domain, for example: Numerical environmental monitoring: Real-time performance is prioritized, and an "edge preprocessing + central verification" mode is adopted. The cleaning priority is followed by format standardization, anomaly removal, and integrity completion. Temporary caching is allowed, and abnormal data retains the original record and is marked. Personnel access character type: legality takes priority, adopts "real-time edge verification + central deduplication" mode, cleaning priority, format standardization → legality verification → semantic deduplication, personnel ID needs to be compared with the park personnel database, illegal IDs are directly removed; Energy consumption statistics (numerical type): Consistency is prioritized, adopting a "centralized processing + edge feedback" mode, with cleaning priority, cross-domain consistency verification → accuracy correction → integrity completion, and comparison with equipment operating status data in the same time period; The dynamic cleaning strategy is stored in the strategy management module of the central node, and its priority is automatically adjusted based on data quality feedback.

[0034] Based on the dynamic cleaning strategy of each data domain, combined with the generated business rules, and the labels in the semantic mapping table, the business rules are transformed into cleaning rules that can be executed in the data domain. The rules are represented in the specific language of each data domain. For example, energy consumption statistics numerical data are represented by Python script fragments (not code, only logical description). The transformed cleaning rules are stored according to data domains. Edge nodes only synchronize lightweight rules (each rule ≤ 10KB) for the data domain they are responsible for, while the central nodes store all rules.

[0035] An improved genetic algorithm is deployed at the central node to optimize the cleaning rules. The encoding method uses binary encoding, with each bit corresponding to the activation status of a cleaning rule. The chromosome length is 120, corresponding to 120 cleaning rules. A fitness function is designed, and the improved strategy crossover operator adopts adaptive crossover probability. The mutation operator introduces rule priority weights to avoid the core rules being destroyed. Finally, the globally optimal rule set and the basic rule set of the edge nodes are output.

[0036] A lightweight cleaning engine is deployed on the edge nodes, which loads the basic rule set, processes the real-time data from the corresponding data source, performs format standardization, simple anomaly removal, caches the pre-processed valid data according to data domain, and uploads the data summary to the central node in real time. For example, an edge node connects to 10 temperature sensors and receives 10 data points every 10 seconds. After preprocessing, one abnormal data point with a temperature of 90℃ is removed, nine valid data points are cached, and a data summary is uploaded to the central node.

[0037] The central node deploys a full-version cleaning engine, loads the globally optimal rule set, receives preprocessed data and data summaries from all edge nodes, and performs deep cleaning, including semantic deduplication, complex anomaly correction, and cross-domain data consistency verification.

[0038] The central node stores the final, deeply cleaned data in a distributed data warehouse and generates a cleaning report.

[0039] Edge nodes upload preprocessed data and data summaries to the central node every 5 minutes to ensure real-time performance. After receiving the data, the central node completes deep cleaning within 1 minute and analyzes the trend of anomaly rate in the data summary. If the anomaly rate of a certain type of data exceeds 15% for 3 consecutive times, the corresponding cleaning rules are optimized and distributed to the corresponding edge nodes within 5 minutes. Edge nodes receive the optimized rules, replace the corresponding rules in the original basic rule set, execute them in the next preprocessing, and feed back the rule execution effect to the central node, forming a closed loop.

[0040] The central node monitors the status of edge nodes through heartbeat packets, including information such as edge node CPU utilization, memory utilization, current data processing volume, and queue length. When the CPU utilization of edge node A reaches 85%, after receiving the heartbeat packet, the central node identifies its overload state within 2 seconds, migrates the calculation task of the "video frame metadata format standardization" part of the rules of the edge node to the central node, and returns the result to the edge node after the central node completes the processing. The CPU utilization of the edge node drops to 62% and returns to normal processing state. When an edge node sends three consecutive heartbeat packets showing that its metrics are below the threshold, the overload condition is lifted, and the central node will relocate the migrated tasks back to the edge node within 10 seconds to ensure the processing autonomy of the edge node. Example

[0041] From a hardware perspective, this application provides an embodiment of an electronic device containing all or part of a method for cleaning data from distributed system mining. The electronic device includes a service processor and a distributed memory. The service processor is connected to the memory, and the distributed memory stores a service self-management program configured to store machine-readable instructions. The service processor executes the service self-management program, and the instructions, when executed by the processor, implement the method for cleaning data from distributed system mining as described above. Example

[0042] Embodiments of this application also provide a computer-readable storage medium capable of implementing the data cleaning method for distributed system mining with a server or client as the execution subject in the above embodiments. The computer-readable storage medium stores a computer program that, when executed by a processor, implements all the contents of the data cleaning method for distributed system mining with a server or client as the execution subject in the above embodiments.

[0043] The embodiments described above are merely preferred embodiments of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications, improvements, and substitutions without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A method for cleaning data from a distributed system, characterized in that, Includes the following steps: The system uses distributed nodes to connect with multiple data sources, extracts the metadata of the data sources, matches it with the semantic knowledge base of the data domain, and generates a semantic mapping table. By using graph neural networks to analyze the metadata of data sources, we can mine the relationships in the metadata information and generate business rules by semantic parsing of the relationships based on the semantic mapping table. The target data domain is determined based on the semantic parsing results and a cleaning strategy is generated. Based on the cleaning strategy, the business rules are transformed into cleaning rules corresponding to each data domain. An improved genetic algorithm is used to establish a dual-mode collaborative cleaning model, which divides distributed nodes into edge nodes and central nodes to clean different types of data.

2. The method for cleaning data from a distributed system according to claim 1, characterized in that, The system is deployed on a distributed architecture, including multiple edge nodes and at least one central node. The edge nodes are connected to heterogeneous data sources, support multiple data source protocols, and the extracted metadata is stored in a distributed metadata repository in a standardized format.

3. The method for cleaning data from a distributed system according to claim 2, characterized in that, The metadata includes: structural metadata, attribute metadata, and business metadata.

4. The method for cleaning data from a distributed system according to claim 2, characterized in that, The generation of the semantic mapping table includes: A domain-specific semantic knowledge base is built by generating annotations from domain experts or by extending from public ontology, and the data domain concept is represented using the OWL language. A BERT-based semantic similarity algorithm is used to perform similarity matching calculations between the attributes and descriptions in the metadata and the data domain concepts in the knowledge base, and the calculation results are obtained. A preset matching threshold is set, and the calculation result is compared with the matching threshold. Based on the comparison result, a semantic mapping table is generated using the concepts of meta-information attributes, semantic tags, and data domains. The semantic tags include domain-specific business tags.

5. The method for cleaning data from a distributed system according to claim 1, characterized in that, The semantic parsing includes: A graph attention network model is used to transform metadata into a two-layer graph structure that can be dynamically updated. The lower layer is the attribute association layer, and the upper layer is the business association layer. Metadata attributes are used as nodes, and the relationships between attributes are used as edges. Edge feature weights are calculated by weighting the co-occurrence frequency of metadata with semantic similarity, and hidden relationships in metadata are mined based on edge feature weights. The relationships output by the graph attention network are mapped to a semantic mapping table, and the semantic labels in the semantic mapping table are used to parse them into business rules.

6. The method for cleaning data from a distributed system according to claim 5, characterized in that, Based on the semantic parsing results, a clustering algorithm is used to divide the data into three-dimensional data domains according to data type, business scenario and quality requirements, and a dedicated dynamic cleaning strategy is generated for each data domain. Based on a dynamic cleaning strategy and combined with the tags in the semantic mapping table associated with each business rule, the business rules are transformed into cleaning rules that can be executed for each data domain. The cleaning rules are represented in a specific language for each data domain.

7. The method for cleaning data from a distributed system according to claim 5, characterized in that, The edge nodes deploy a lightweight cleaning engine that processes data with high real-time requirements based on basic rules generated by an improved genetic algorithm, performing format standardization, simple anomaly removal, and local data caching. The central node deploys a full-version cleaning engine that processes data requiring cross-node collaboration based on globally optimal rules, performing semantic deduplication, complex anomaly correction, and cross-domain data consistency verification.

8. The method for cleaning data from a distributed system according to claim 7, characterized in that, The dual-mode collaborative cleaning model has a collaborative mechanism to realize a closed-loop process of edge preprocessing, center optimization, and edge feedback. The central node monitors the status of the edge nodes through heartbeat packets. When the processing pressure on the edge nodes exceeds the threshold, it dynamically allocates some rule calculation tasks to the central node.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the content of the data cleaning method for mining distributed systems as described in claim 1.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the content of the data cleaning method for mining distributed systems as described in claim 1.

Images