A dynamic configurable data cleaning method and system based on big data

Abstract

The present invention belongs to the field of data processing technology. The present invention discloses a dynamic and configurable data cleaning method based on big data and a system thereof. The method comprises collecting heterogeneous data sources and accessing the heterogeneous data sources in combination with a data source adapter interface; utilizing a simulated antigen recognition memory mechanism to identify abnormal patterns in the heterogeneous data sources and encode them into antigen fingerprints; constructing a data value density scoring function based on the antigen fingerprint to obtain a data value density score; constructing a three-dimensional storage topological space according to the data value density score, adopting a lattice migration algorithm to automatically identify the critical points of hot and cold data, triggering hierarchical migration when the heterogeneous data source is cooled to a preset temperature threshold, introducing a thermodynamic diffusion mechanism, and simulating the migration process of the heterogeneous data source in the three-dimensional storage topological space as a heat diffusion path; and significantly improving the operating efficiency of the cleaning task.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and more specifically, to a big data-based dynamically configurable data cleaning method and system thereof. Background Art

[0002] Patent publication number CN105930523A discloses a data cleansing framework based on dynamically configurable rules for use in big data environments. This inventive method is a cross-domain, reusable, and configurable approach that integrates data conversion, data inspection, and data repair, thereby improving the descriptive capabilities and execution efficiency of the cleansing process. Experimental results on multiple real-world datasets demonstrate that the system can seamlessly integrate dynamically configurable rules into multiple data sources and diverse application areas. This system has been implemented in multiple projects, further validating the method's effectiveness in real-world scenarios.

[0003] Existing configurable data cleaning methods and systems have the following main problems:

[0004] In the dynamic and configurable data cleaning process based on big data, the number of heterogeneous data sources is huge, the structure is complex, and the data access patterns are changeable. Traditional hot and cold data management methods mainly rely on static thresholds to divide data into hot and cold states, and perform data migration and hierarchical management through manual or periodic triggering. Multi-source heterogeneous data exhibits highly complex and dynamic behavior patterns in actual cleaning applications. Traditional threshold-based hot and cold division is difficult to accurately capture the critical changes in the hot and cold states of data, resulting in confusion between hot and cold data, affecting the priority scheduling and resource allocation of cleaning tasks.

[0005] The intrinsic value of data constantly changes during the big data cleaning process. Traditional methods fail to fully consider the phase-change characteristics of value density over time, making it difficult to intelligently adjust cleaning strategies, impacting cleaning effectiveness and efficiency. In large-scale data storage, data access hotspots exhibit distinct spatial local aggregation characteristics. However, traditional hot and cold data migration mechanisms ignore this spatial correlation, resulting in slow resource scheduling and migration responses, impacting the overall performance of the data cleaning system. Existing data migration strategies are mostly periodic or manually triggered, lacking dynamic adaptability and inability to respond to data state changes in real time. This leads to inefficient utilization of storage and computing resources, impacting the timeliness and accuracy of dynamic data cleaning.

[0006] In view of this, the present invention proposes a dynamic and configurable data cleaning method and system based on big data to solve the above problems. Summary of the Invention

[0007] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: a dynamic and configurable data cleaning method based on big data, comprising:

[0008] S1. Collect heterogeneous data sources and integrate them with data source adapter interfaces. Utilize a simulated antigen recognition memory mechanism to identify abnormal patterns in heterogeneous data sources and encode them into antigen fingerprints. Construct a data value density scoring function based on the antigen fingerprints to obtain a data value density score.

[0009] S2. Construct a three-dimensional storage topology space based on data value density scores. A lattice migration algorithm is used to automatically identify the critical points of hot and cold data. When the temperature of heterogeneous data sources drops to a preset threshold, layered migration is triggered. A thermodynamic diffusion mechanism is introduced to simulate the migration process of heterogeneous data sources in the three-dimensional storage topology space as a heat diffusion path.

[0010] S3. Build a data cleaning flowchart by dragging and dropping, determine the execution order of the cleaning operators in the data cleaning flowchart based on topological sorting, and automatically schedule cleaning tasks;

[0011] S4. Construct a Darwinian evolution system for cleaning rules, perform periodic evolution on the cleaning rules, define the fitness function of the cleaning rules, simulate the natural selection mechanism to dynamically optimize the cleaning strategy, and output the evolved rule instruction set;

[0012] S5: Build a heterogeneous processing architecture and customize the VHDL kernel to clean heterogeneous data sources. Collaborate with Spark's batch reading mechanism based on the heat diffusion path and call the evolved rule instruction set to complete the preset data cleaning task.

[0013] S6. Construct a dual-chain structure with an operation evidence chain and a quality certification chain to audit and verify the heterogeneous data sources after cleaning; use a verifiable summary function to record the cleaning credibility of the heterogeneous data sources after cleaning; when the cleaning credibility is less than the preset cleaning credibility threshold, trigger the re-cleaning mechanism and update the cleaning rules.

[0014] Preferably, the method for identifying abnormal patterns in heterogeneous data sources and encoding them into antigen fingerprints includes:

[0015] Heterogeneous data sources include structured data, semi-structured data, unstructured data, streaming data, and system operation and maintenance data. Standard deviation normalization is performed on heterogeneous data sources to convert them into standardized data formats.

[0016] Constructing a simulated antibody library, defining each antibody in the simulated antibody library as an adaptable and variable antibody template; the antibody template is any one of a K-nearest neighbor model-based template, a cluster center vector-based template, or a small sample neural network-based template;

[0017] The matching degree between the heterogeneous data source and the antibody template is calculated using a similarity detection mechanism; the similarity detection mechanism can be any of cosine similarity, Euclidean distance, or kernel function similarity. If the matching degree is less than a preset matching threshold, it is identified as an abnormal pattern and marked as a simulated antigen.

[0018] The simulated antigen is fingerprinted, and the fingerprint encoding process includes feature dimensionality reduction, fingerprint hash mapping and feature vector normalization to generate an antigen fingerprint containing data identification, anomaly category, timestamp and feature hash value.

[0019] Preferably, the method for obtaining the data value density score includes:

[0020] Extract multi-dimensional information from antigen fingerprints and construct data feature vectors. The multi-dimensional information includes antigen occurrence frequency, anomaly level, fingerprint similarity distribution, historical trigger records, and contextual information entropy.

[0021] A data value density scoring function is constructed based on the data feature vector. The data value density scoring function is a nonlinear weighted function. Each antigen fingerprint is input into the corresponding data value density scoring function to calculate the corresponding data value density score.

[0022] Preferably, the method for constructing the three-dimensional storage topological space includes:

[0023] Design a three-dimensional coordinate system for the three-dimensional storage topology space. Combine X, Y, and Z as the three-dimensional coordinates of the three-dimensional storage topology space. Organize all three-dimensional coordinates into an integer grid to form a logical three-dimensional storage topology space. Each dimension is defined as the X-axis, Y-axis, and Z-axis. The X-axis represents the value density dimension, the Y-axis represents the data structure dimension, and the Z-axis represents the time life cycle dimension.

[0024] Establish grading and coding rules for each dimension. For the X-axis, normalize all data value density scores and linearly map them to the [0,1] scoring interval. Define stratification levels based on the scoring interval, and preset the first, second, and third thresholds for data value density scores.

[0025] If the data value density score is greater than or equal to the second threshold of the data value density score and less than or equal to the third threshold of the data value density score, it is classified as the high value tier; if the data value density score is greater than or equal to the first threshold of the data value density score and less than the second threshold of the data value density score, it is classified as the medium value tier; if it is less than the first threshold of the data value density score, it is classified as the low value tier;

[0026] For the Y axis, set static mapping codes for the data structure types included in the data structure dimension. The data structure types include structured data, semi-structured data, and unstructured data. For structured data, let Y = 0; for semi-structured data, let Y = 1; for unstructured data, let Y = 2. The data structure dimension is extensible and supports subclasses. When Y = 3, it indicates mixed structure data.

[0027] For the Z axis, the time life cycle stage is defined according to the most recent access time, and the mapping rules are set; the first threshold of the most recent access time, the second threshold of the most recent access time, and the third threshold of the most recent access time are preset; when the most recent access time is less than the first threshold of the most recent access time, Z=0; when the most recent access time is greater than or equal to the first threshold of the most recent access time and less than the second threshold of the most recent access time, Z=1; when the most recent access time is greater than the second threshold of the most recent access time, Z=2.

[0028] Preferably, the method for obtaining the heat diffusion path includes:

[0029] In the three-dimensional storage topology space, each coordinate point is regarded as a lattice node, which contains m data units. For each data unit , define its current heat value as ; Treat the three-dimensional coordinate system as a spatial lattice grid and define each lattice node The average heat is ;

[0030] Taking into account the spatial correlation of thermal diffusion, the local thermal diffusion coefficient is introduced , establish a lattice heat diffusion coupling model and preset a temperature threshold. When the temperature of the heterogeneous data source drops to the preset temperature threshold, the hierarchical migration judgment is triggered; the heterogeneous data source with a temperature less than or equal to the preset temperature threshold is identified as cold data, and the heterogeneous data source with a temperature greater than the preset temperature threshold is identified as hot data;

[0031] A data migration trigger mechanism is defined, and hierarchical migration operations are performed when the trigger conditions of the data migration trigger mechanism are met. The heat diffusion path of heterogeneous data sources is migrated from high-temperature lattice nodes to low-temperature lattice nodes to simulate the gradual release of heat and the migration of data access hotspots. The heat diffusion path is the spatial trajectory of lattice node temperature changes and migration operations.

[0032] Preferably, the method for automatically scheduling cleaning tasks includes:

[0033] Preset data cleaning operator library. Operator nodes can be selected from the library by dragging and dropping to build a data cleaning flowchart. Directed edges of the data cleaning flowchart are generated based on the connection relationship between operator nodes. Each directed edge is automatically labeled with data dependency.

[0034] The data cleaning flow chart is formalized as a directed acyclic graph, and the operator nodes in the directed acyclic graph are sorted based on the topological sorting algorithm to generate an operator execution sequence that satisfies the data dependencies, where the operator nodes without dependencies are marked as executable in parallel; executable task units are generated according to the operator execution sequence, and the cleaning tasks are automatically scheduled for execution in the heterogeneous processing architecture.

[0035] Preferably, the method for obtaining the evolved rule instruction set includes:

[0036] Preset data cleaning rules and initialize the data cleaning rule set as a rule population. The cleaning rules are represented by encoding, and the initial rule population is generated from the historical cleaning rule library. Define the rule fitness function, which comprehensively considers cleaning accuracy, cleaning coverage, and execution efficiency, and comprehensively evaluates rule performance through weighted summation.

[0037] In each evolution cycle, the rule with the highest fitness is selected and retained according to the rule fitness function; a crossover operation is performed on the selected cleaning rules, and new cleaning rules are generated by combining the characteristics of the cleaning rules; a mutation operation is performed on the new cleaning rules generated by the crossover;

[0038] A fitness threshold is preset, and the cleaning rules with fitness less than the preset fitness threshold are replaced with the newly evolved cleaning rules; after L evolution cycles, the cleaning rules with fitness reaching the preset fitness threshold are selected to form the final rule instruction set.

[0039] Preferably, the method of calling the evolved rule instruction set to complete the preset data cleaning task includes:

[0040] Build a heterogeneous processing architecture consisting of an FPGA acceleration unit, a Spark distributed processing engine, and a data scheduling manager. Divide the data cleaning process into three stages: preprocessing, core cleaning, and output verification. In the preprocessing stage, the Spark distributed processing engine is used to batch divide, pre-cache, and bind cleaning rules to heterogeneous data.

[0041] In the core cleaning phase, a customized VHDL kernel is used to clean heterogeneous data. In the output verification phase, the Spark distributed processing engine receives the cleaning results from the FPGA acceleration unit and integrates and outputs them, thus completing the preset data cleaning task.

[0042] A VHDL cleaning pipeline consisting of N configurable logic modules is constructed based on a stream processing architecture. The evolved rule instruction set is converted into an instruction format recognizable by the FPGA and mapped to the corresponding configurable logic modules in the VHDL cleaning pipeline based on the instruction content. The configurable logic modules include a field recognition module, a missing value repair module, an outlier replacement module, and a format standardization module. Each module can be enabled on demand to achieve dynamic modular cleaning.

[0043] By connecting to heterogeneous data sources through the Spark distributed processing engine, a data heat-aware scheduling mechanism is built based on the heat diffusion path. The data scheduling manager pushes hot data to the FPGA acceleration unit for cleaning and processing, while cold data is cached and awaits scheduling.

[0044] For each batch of heterogeneous data sources pushed, the corresponding rule instruction set identifier is bound to drive the FPGA acceleration unit to load the corresponding configurable logic module and perform cleaning operations; the cleaned heterogeneous data source is returned from the FPGA acceleration unit to the Spark distributed processing engine for formatting and packaging, and the cleaned heterogeneous data source is output.

[0045] Preferably, the method for auditing and verifying the cleansed heterogeneous data source includes:

[0046] A dual-chain structure with an operation evidence chain and a quality certification chain is constructed. For each data cleaning operation process, the data cleaning operation timestamp, the called cleaning rule ID, the data cleaning flow chart, the unique identifier of the processed heterogeneous data source, the operator and the algorithm subject ID are recorded in real time. The summary hash value of the data cleaning operation event is calculated using the verifiable summary function Merkle Tree to form the operation fingerprint.

[0047] The operation fingerprint is written into the operation evidence chain as a transaction and linked with the hash of the previous block to form a blockchain structure. The operation evidence chain supports smart contracts. When the data cleaning operation is completed, the smart contract automatically triggers the audit verification of the heterogeneous data source after cleaning based on the operation fingerprint.

[0048] A dynamic and configurable data cleaning system based on big data, comprising:

[0049] The adaptive detection module is used to collect heterogeneous data sources and integrate them with the data source adapter interface. It uses a simulated antigen recognition memory mechanism to identify abnormal patterns in heterogeneous data sources and encode them into antigen fingerprints. Based on the antigen fingerprints, it constructs a data value density scoring function to obtain a data value density score.

[0050] The intelligent tiered storage module constructs a three-dimensional storage topology space based on data value density scores. It uses a lattice migration algorithm to automatically identify the critical points of hot and cold data, triggering tiered migration when the temperature of heterogeneous data sources drops to a preset threshold. It also introduces a thermodynamic diffusion mechanism to simulate the migration process of heterogeneous data sources in the three-dimensional storage topology space as a heat diffusion path.

[0051] The data cleaning orchestration module builds a data cleaning flowchart by dragging and dropping, determines the execution order of cleaning operators in the data cleaning flowchart based on topological sorting, and automatically schedules cleaning tasks;

[0052] The ecological rule evolution module builds a Darwinian evolution system for cleaning rules, periodically evolves the cleaning rules, defines the fitness function of the cleaning rules, simulates the natural selection mechanism to dynamically optimize the cleaning strategy, and outputs the evolved rule instruction set;

[0053] The hardware-accelerated cleaning module builds a heterogeneous processing architecture and customizes the VHDL kernel to clean heterogeneous data sources. It collaborates with Spark's batch reading mechanism based on the heat diffusion path and calls the evolved rule instruction set to complete the preset data cleaning tasks.

[0054] The trusted cleaning evidence module builds a dual-chain structure with an operation evidence chain and a quality certification chain to audit and verify heterogeneous data sources after cleaning. It uses a verifiable summary function to record the cleaning credibility of the heterogeneous data sources after cleaning. When the cleaning credibility is less than the preset cleaning credibility threshold, the re-cleaning mechanism is triggered and the cleaning rules are updated.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] This invention constructs a comprehensive heat metric by comprehensively considering access frequency, update frequency, and changes in data value density. Based on a local heat diffusion coupling model, it can capture the hot and cold critical points of multi-source heterogeneous data in a big data environment in real time, significantly improving the accuracy of hot and cold data classification, thereby optimizing data cleaning priorities and strategy selection. Dynamically reflecting the temporal changes and phase transitions in data value density enables the data cleaning system to automatically adjust cleaning rules and task scheduling based on real-time changes in data value, enhancing the adaptability and intelligence of the cleaning process.

[0057] By simulating the spatial diffusion of heat in thermodynamics, it accurately reflects the migration trajectory of data access hotspots in three-dimensional storage space, supports dynamic tiered migration strategies based on heat diffusion paths, effectively avoids migration delays, and improves storage resource utilization and the efficiency of data cleaning tasks. Utilizing preset temperature thresholds and migration trigger mechanisms, it can automatically sense data temperature changes and trigger migration operations, enabling real-time dynamic management of hot and cold data, ensuring that data cleaning tasks are efficiently executed at the appropriate storage level, and enhancing the robustness and responsiveness of the system. The dynamic migration mechanism based on heat diffusion paths replaces traditional manual or periodic migration strategies, enabling automatic sensing and execution of data hot and cold states and migration operations, significantly reducing manual maintenance costs and improving system stability and intelligence. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 This is a flow chart of a dynamically configurable data cleaning method based on big data of the present invention;

[0059] Figure 2 This is a structural diagram of a dynamically configurable data cleaning system based on big data of the present invention;

[0060] Figure 3 This is a flow chart of the method for automatically scheduling cleaning tasks provided by the present invention. DETAILED DESCRIPTION

[0061] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0062] Example 1

[0063] See also Figure 1 and Figure 3 As shown, Example 1 further illustrates a dynamic configurable data cleaning method based on big data proposed by the present invention, including:

[0064] With the rapid development of big data technologies, data cleaning, as a key step in data preprocessing, is crucial for improving data quality and enhancing data usability. Currently, cleaning tasks in big data environments often face challenges such as massive data volumes, diverse data types, complex structures, and high dynamism. Especially in scenarios involving multi-source, heterogeneous data access, the accuracy and efficiency of cleaning strategies directly impact the effectiveness of subsequent data analysis and intelligent decision-making.

[0065] Existing configurable data cleaning methods and systems generally use rule engines and static configurations to organize and schedule the cleaning process. These methods have achieved some success in cleaning certain structured data. However, in actual big data processing scenarios, the following major technical issues remain to be addressed:

[0066] Heterogeneous data sources are numerous, complex, and their access behaviors are highly dynamic and changeable. Traditional hot and cold data management methods often rely on static metrics such as access frequency or last access time. They categorize data into "hot," "warm," or "cold" states by setting fixed thresholds, and use manual or periodic triggering to migrate and stratify data. When faced with data with complex behavioral patterns and nonlinear value evolution, these methods often struggle to accurately identify the critical points between hot and cold states, leading to errors in identifying hot and cold data, which in turn affects the priority of cleaning tasks and the proper allocation of computing resources.

[0067] The intrinsic value density of data evolves over time and exhibits a "phase change" characteristic, that is, data changes from high value to low value or vice versa may occur within a short time window. Traditional methods fail to effectively model and respond to this dynamic process, resulting in lagging cleaning strategies and reducing the intelligence and adaptability of data processing.

[0068] Data access hotspots in current data storage systems usually show obvious spatial local aggregation, but traditional hot and cold data management strategies often use global granularity and ignore local heat aggregation and diffusion behaviors, resulting in slow response in data migration and resource scheduling, affecting the overall cleaning efficiency and performance of the system.

[0069] Existing systems mostly use fixed cycles or manual intervention to trigger data migration and cleaning operations. They lack the ability to dynamically adjust based on real-time data status and are unable to cope with sudden data changes or high-frequency access fluctuations, further limiting the system's resource utilization and the timeliness of task processing.

[0070] In order to effectively solve the above problems, the present invention proposes a dynamic and configurable data cleaning system based on big data, comprising:

[0071] S1. Collect heterogeneous data sources and integrate them with data source adapter interfaces. Utilize a simulated antigen recognition memory mechanism to identify abnormal patterns in heterogeneous data sources and encode them into antigen fingerprints. Construct a data value density scoring function based on the antigen fingerprints to obtain a data value density score.

[0072] S2. Construct a three-dimensional storage topology space based on data value density scores. A lattice migration algorithm is used to automatically identify the critical points of hot and cold data. When the temperature of heterogeneous data sources drops to a preset threshold, layered migration is triggered. A thermodynamic diffusion mechanism is introduced to simulate the migration process of heterogeneous data sources in the three-dimensional storage topology space as a heat diffusion path.

[0073] S3. Build a data cleaning flowchart by dragging and dropping, determine the execution order of the cleaning operators in the data cleaning flowchart based on topological sorting, and automatically schedule cleaning tasks;

[0074] S4. Construct a Darwinian evolution system for cleaning rules, perform periodic evolution on the cleaning rules, define the fitness function of the cleaning rules, simulate the natural selection mechanism to dynamically optimize the cleaning strategy, and output the evolved rule instruction set;

[0075] S5: Build a heterogeneous processing architecture and customize the VHDL kernel to clean heterogeneous data sources. Collaborate with Spark's batch reading mechanism based on the heat diffusion path and call the evolved rule instruction set to complete the preset data cleaning task.

[0076] S6. Construct a dual-chain structure with an operation evidence chain and a quality certification chain to audit and verify the heterogeneous data sources after cleaning; use a verifiable summary function to record the cleaning credibility of the heterogeneous data sources after cleaning; when the cleaning credibility is less than the preset cleaning credibility threshold, trigger the re-cleaning mechanism and update the cleaning rules.

[0077] Methods for identifying abnormal patterns in heterogeneous data sources and encoding them as antigen fingerprints include:

[0078] Heterogeneous data sources include structured data, semi-structured data, unstructured data, streaming data, and system operation and maintenance data. Standard deviation normalization is performed on heterogeneous data sources to convert them into standardized data formats.

[0079] Constructing a simulated antibody library, defining each antibody in the simulated antibody library as an adaptable and variable antibody template; the antibody template is any one of a K-nearest neighbor model-based template, a cluster center vector-based template, or a small sample neural network-based template;

[0080] The matching degree between the heterogeneous data source and the antibody template is calculated using a similarity detection mechanism; the similarity detection mechanism can be any of cosine similarity, Euclidean distance, or kernel function similarity. If the matching degree is less than a preset matching threshold, it is identified as an abnormal pattern and marked as a simulated antigen.

[0081] The simulated antigen is fingerprinted, and the fingerprint encoding process includes feature dimensionality reduction, fingerprint hash mapping and feature vector normalization to generate an antigen fingerprint containing data identification, anomaly category, timestamp and feature hash value.

[0082] Methods for obtaining data value density scores include:

[0083] Multi-dimensional information extraction is performed on antigen fingerprints to construct data feature vectors. The multi-dimensional information includes antigen occurrence frequency, anomaly level (obtained based on the deviation between the matching degree and the preset matching degree threshold), fingerprint similarity distribution (the similarity distribution between the current antigen fingerprint and historically known antigen fingerprints, used to determine whether the anomaly is a new anomaly or a variant anomaly), historical trigger records (whether this type of fingerprint has triggered a cleaning task in the past and whether it has caused anomalies in the results), and contextual information entropy;

[0084] A data value density scoring function is constructed based on the data feature vector. The data value density scoring function is a nonlinear weighted function. Each antigen fingerprint is input into the corresponding data value density scoring function to calculate the corresponding data value density score.

[0085] The construction method of the three-dimensional storage topological space includes:

[0086] Design a three-dimensional coordinate system for the three-dimensional storage topology space. Combine X, Y, and Z as the three-dimensional coordinates of the three-dimensional storage topology space. Organize all three-dimensional coordinates into an integer grid to form a logical three-dimensional storage topology space. Each dimension is defined as the X-axis, Y-axis, and Z-axis. The X-axis represents the value density dimension, the Y-axis represents the data structure dimension, and the Z-axis represents the time life cycle dimension.

[0087] The value density dimension measures the potential value of data units in cleaning, mining, and analysis, and is derived from the data value density score calculated previously. The data structure dimension identifies the structural type of the data, reflecting heterogeneous data sources. The time life cycle dimension represents the time attribute or active state of the data, and is used to characterize its life cycle stage (such as real-time, near-term, medium-term, and long-term).

[0088] Establish grading and coding rules for each dimension. For the X-axis, normalize all data value density scores and linearly map them to the [0,1] scoring interval. Define stratification levels based on the scoring interval, and preset the first, second, and third thresholds for data value density scores.

[0089] If the data value density score is greater than or equal to the second threshold of the data value density score and less than or equal to the third threshold of the data value density score, it is classified as the high value tier; if the data value density score is greater than or equal to the first threshold of the data value density score and less than the second threshold of the data value density score, it is classified as the medium value tier; if it is less than the first threshold of the data value density score, it is classified as the low value tier;

[0090] For the Y axis, set static mapping codes for the data structure types included in the data structure dimension. The data structure types include structured data, semi-structured data, and unstructured data. For structured data, let Y = 0; for semi-structured data, let Y = 1; for unstructured data, let Y = 2. The data structure dimension is extensible and supports subclasses. When Y = 3, it indicates mixed structure data.

[0091] For the Z axis, the time life cycle stage is defined according to the most recent access time, and the mapping rules are set; the first threshold of the most recent access time, the second threshold of the most recent access time, and the third threshold of the most recent access time are preset; when the most recent access time is less than the first threshold of the most recent access time, Z=0; when the most recent access time is greater than or equal to the first threshold of the most recent access time and less than the second threshold of the most recent access time, Z=1; when the most recent access time is greater than the second threshold of the most recent access time, Z=2.

[0092] Methods for obtaining heat diffusion paths include:

[0093] In traditional hot and cold data management, threshold rules are often set based on dimensions such as access frequency and last access time to classify data into hot and cold data, and then migration strategies are triggered manually or periodically. However, this approach has the following limitations: it cannot accurately identify the critical points of hot and cold changes in multi-source heterogeneous data under complex behavioral patterns; it is difficult to perceive the phase change state of data value density over time; and it lacks the ability to perceive dynamic spatial local aggregation, resulting in delayed migration strategies and low resource utilization.

[0094] Therefore, a lattice migration algorithm is introduced, combined with a three-dimensional storage topology, to achieve a more sensitive and accurate hot and cold migration mechanism. The lattice migration algorithm simulates the behavior of state migration caused by the thermal motion of particles in the solid lattice, and establishes a migration mechanism based on local spatial thermal diffusion and critical phase transition perception.

[0095] In the three-dimensional storage topology space, each coordinate point is regarded as a lattice node, which contains m data units. For each data unit , define its current heat value as ;

[0096] ;in, Indicates the data unit per unit time Number of visits; Indicates the data unit per unit time Frequency of updates (writes or modifications); Represents data units per unit time rate of change of value density; Indicates the current time point; Indicates the index of the data unit; The weight factor representing the frequency of visits; A weight factor indicating the update frequency; The weight factor representing the rate of change of value density; according to the expert experience method, 、 and The value range of is between 0 and 1, and 、 、 The sum is 1. It should be noted that the current popularity value is obtained through a weighted summation formula, which takes into account the access frequency, update frequency and changes in information value. It is a comprehensive measure of data popularity.

[0097] Treat the three-dimensional coordinate system as a spatial lattice grid and define each lattice node The average heat is ; ;in, Represents a node in a three-dimensional storage topology space At the time point Average heat; Represents a lattice node The number of data units contained in ; considering the spatial correlation of heat diffusion, the local heat diffusion coefficient is introduced , establish a lattice heat diffusion coupling model and preset a temperature threshold. When the temperature of the heterogeneous data source drops to the preset temperature threshold, the hierarchical migration judgment is triggered; the heterogeneous data source with a temperature less than or equal to the preset temperature threshold is identified as cold data, and the heterogeneous data source with a temperature greater than the preset temperature threshold is identified as hot data;

[0098] A data migration trigger mechanism is defined, and hierarchical migration operations are performed when the trigger conditions of the data migration trigger mechanism are met. The heat diffusion path of heterogeneous data sources is migrated from high-temperature lattice nodes to low-temperature lattice nodes to simulate the gradual release of heat and the migration of data access hotspots. The heat diffusion path is the spatial trajectory of lattice node temperature changes and migration operations.

[0099] The lattice heat diffusion coupling model is: ;in, Laplacian operator representing local heat; It represents the cooling coupling factor, which adjusts the impact of the difference between heat and temperature threshold. According to the expert experience method, The value range is between 0 and 1; Indicates the temperature threshold;

[0100] It should be noted that the lattice heat diffusion coupling model draws on the classical heat conduction equation to reflect the natural trend of heat diffusion from high temperature area to low temperature area; adding cooling mechanism It is equivalent to a heat dissipation mechanism, which makes the system tend to low temperature; spatial heat diffusion + threshold dissipation jointly determine the evolution of heat, which is the physical basis for determining hot and cold stratification.

[0101] The data migration trigger mechanism is: ;in, Represents a node in a three-dimensional storage topology space At the time point Average heat; The time derivative representing the heat change of heterogeneous data sources; Indicates the cooling delay judgment time window, which is the shortest time that the low temperature must be maintained; Indicates a time point At the time The total temperature integral within ; Indicates that when the trigger conditions of the data migration trigger mechanism are met, the layered migration operation is triggered; is the time integral variable, which means that arrive At any moment in

[0102] It should be noted that the data migration trigger mechanism is a triple judgment mechanism to enhance robustness: Condition 1 ensures that the current temperature is cold; Condition 2 ensures that the temperature is continuously decreasing to avoid short-term fluctuations; Condition 3 ensures that the cooling lasts long enough to eliminate occasional temperature drops. The integral term introduces time stability: the judgment standard is improved from instantaneous value to average behavior over a period of time.

[0103] The following problems existing in the existing technology are solved: in the dynamic and configurable data cleaning process based on big data, the number of heterogeneous data sources is huge, the structure is complex, and the data access patterns are changeable. The traditional hot and cold data management method mainly relies on static thresholds to divide the data into hot and cold states, and performs data migration and hierarchical management through manual or periodic triggering. Multi-source heterogeneous data exhibits highly complex and dynamic behavior patterns in actual cleaning applications. The traditional threshold-based hot and cold division is difficult to accurately capture the critical changes in the hot and cold states of the data, resulting in confusion between hot and cold data, affecting the priority scheduling and resource allocation of cleaning tasks.

[0104] The intrinsic value of data (such as accuracy and completeness) constantly changes during the big data cleaning process. Traditional methods fail to fully account for the phase-change characteristics of value density over time, making it difficult to intelligently adjust cleaning strategies, impacting cleaning effectiveness and efficiency. In large-scale data storage, data access hotspots exhibit distinct spatial localization and aggregation characteristics. However, traditional hot and cold data migration mechanisms ignore this spatial correlation, resulting in slow resource scheduling and migration responses, impacting the overall performance of data cleaning systems. Existing data migration strategies are mostly periodic or manually triggered, lacking dynamic adaptability and inability to respond to data state changes in real time. This results in inefficient utilization of storage and computing resources, impacting the timeliness and accuracy of dynamic data cleaning.

[0105] Compared with existing technologies, this approach offers several advantages: By comprehensively considering access frequency, update frequency, and changes in data value density, it constructs a comprehensive heat measurement metric. Based on a local heat diffusion coupling model, it can capture the hot and cold critical points of multi-source heterogeneous data in a big data environment in real time, significantly improving the accuracy of hot and cold data partitioning and thus optimizing data cleaning priorities and strategy selection. By dynamically reflecting the temporal changes and phase transitions in data value density, the data cleaning system can automatically adjust cleaning rules and task scheduling based on real-time changes in data value, enhancing the adaptability and intelligence of the cleaning process.

[0106] By simulating the spatial diffusion of heat in thermodynamics, it accurately reflects the migration trajectory of data access hotspots in three-dimensional storage space, supports dynamic tiered migration strategies based on heat diffusion paths, effectively avoids migration delays, and improves storage resource utilization and the efficiency of data cleaning tasks. Utilizing preset temperature thresholds and migration trigger mechanisms, it can automatically sense data temperature changes and trigger migration operations, enabling real-time dynamic management of hot and cold data, ensuring that data cleaning tasks are efficiently executed at the appropriate storage level, and enhancing the robustness and responsiveness of the system. The dynamic migration mechanism based on heat diffusion paths replaces traditional manual or periodic migration strategies, enabling automatic sensing and execution of data hot and cold states and migration operations, significantly reducing manual maintenance costs and improving system stability and intelligence.

[0107] For example, in a large-scale intelligent manufacturing platform, more than 4,000 industrial CNC devices are deployed. These devices upload multi-source heterogeneous data through IoT nodes every minute, including:

[0108] Structured data includes operating parameters, temperature, current, vibration values, etc. (recorded at sub-second frequency); semi-structured data includes sensor alarm logs and system alarm status; unstructured data includes infrared / thermal imaging video clips and equipment camera images; interactive data includes engineer operation records and inspection tasks. The platform, based on a data lake architecture, adds over 5TB of data daily. Approximately 70% of this data is high-value data with frequent short-term access (such as vibration trends and critical alarms); approximately 20% of the data becomes inactive within hours of initial collection; and the remaining 10% is periodically active (such as monthly equipment reconciliation records or compliance inspection callbacks).

[0109] The traditional hot and cold tiered cleaning mechanism judges data as cold and cleans or migrates it based on "last access time > 30 days", resulting in some high-value alarm data being mistakenly cleaned before participating in early warning aggregation; some structured sensor data, although inactive, is still used as key data for AI model training and is mistakenly labeled as "cold"; computing resource scheduling is seriously lagging, and the cleaning frequency during peak periods does not match the sudden increase in data.

[0110] Three-dimensional lattice construction: A three-dimensional lattice topology is constructed based on "data source type × time dimension × storage level". Each lattice node (for example, temperature sensor data, 2025-05-15, hot data area) contains an average of approximately 1,000 records.

[0111] The heat of a node in the temperature sensor data is calculated as follows: ; Among them, the data unit per unit time Number of visits 12; data unit per unit time Update frequency 0; data unit per unit time Rate of change of value density is 0.01; then the current heat value If the average heat around the node is 6.5, but its own heat continues to drop to 3.0 and remains above the cooling delay judgment time window (30 minutes), cold data migration is triggered.

[0112] Methods for automatically scheduling cleaning tasks include:

[0113] Preset data cleaning operator library. Operator nodes can be selected from the library by dragging and dropping to build a data cleaning flowchart. Directed edges of the data cleaning flowchart are generated based on the connection relationship between operator nodes. Each directed edge is automatically labeled with data dependency.

[0114] The preset data cleaning operator library includes input operators, transformation operators, filter operators, aggregation operators, and output operators. In response to the drag operation, a data cleaning flow chart formed by connecting corresponding operator nodes is constructed on the blank canvas;

[0115] The data cleaning flow chart is formalized as a directed acyclic graph, and the operator nodes in the directed acyclic graph are sorted based on the topological sorting algorithm to generate an operator execution sequence that satisfies the data dependencies, where the operator nodes without dependencies are marked as executable in parallel; executable task units are generated according to the operator execution sequence, and the cleaning tasks are automatically scheduled for execution in the heterogeneous processing architecture.

[0116] The methods for obtaining the evolved rule instruction set include:

[0117] Preset data cleaning rules and initialize the data cleaning rule set as a rule population. The cleaning rules are represented by encoding, and the initial rule population is generated from the historical cleaning rule library. Define the rule fitness function, which comprehensively considers cleaning accuracy, cleaning coverage, and execution efficiency, and comprehensively evaluates rule performance through weighted summation.

[0118] The fitness function is ;in, Indicates cleaning accuracy; Indicates cleaning coverage; Indicates execution efficiency; Indicates the Cleaning rules; An index indicating the number of cleaning rules;

[0119] It should be noted that the cleaning accuracy is used to measure the ability of the data cleaning rules to identify and repair abnormal or erroneous data. It is obtained as follows: the number of data correctly identified and repaired and the total amount of abnormal data in the cleaning results are counted; the number of data correctly identified and repaired in the cleaning results is divided by the total amount of abnormal data to obtain the cleaning accuracy;

[0120] Cleaning coverage measures the proportion of cleaning rules applicable to the current data stream, primarily reflecting the versatility and breadth of the rules. It is obtained as follows: In a batch of input data, count the number of data records adapted and processed by a rule and the total number of data in the batch. Divide the number of data records adapted and processed by a rule by the total number of data in the batch to obtain the cleaning coverage.

[0121] Execution efficiency is used to measure the performance consumption of cleaning rules during actual operation. The acquisition methods include the following indicators: the processing time of statistical rules on a batch of data And the total number of data in this batch ; Unit data processing time can be introduced as an efficiency indicator, then the execution efficiency ; That is, the shorter the unit data cleaning time, the higher the efficiency and the better the adaptability.

[0122] In each evolution cycle, the rule with the highest fitness is selected and retained according to the rule fitness function; a crossover operation is performed on the selected cleaning rules, and new cleaning rules are generated by combining the characteristics of the cleaning rules; a mutation operation is performed on the new cleaning rules generated by the crossover;

[0123] A fitness threshold is preset, and the cleaning rules with fitness less than the preset fitness threshold are replaced with the newly evolved cleaning rules; after L evolution cycles, the cleaning rules with fitness reaching the preset fitness threshold are selected to form the final rule instruction set.

[0124] Methods for calling the evolved rule instruction set to complete the preset data cleaning task include:

[0125] Build a heterogeneous processing architecture consisting of an FPGA acceleration unit, a Spark distributed processing engine, and a data scheduling manager. The FPGA acceleration unit is used to execute high-concurrency data cleaning operations based on a rule-based instruction set. The Spark distributed processing engine is used for batch scheduling of large-scale heterogeneous data and packaging and output of data cleaning results. The data scheduling manager is used to schedule data batches.

[0126] The data cleaning process is divided into three stages: preprocessing, core cleaning, and output verification. In the preprocessing stage, the Spark distributed processing engine is used to divide heterogeneous data into batches, pre-cache, and bind cleaning rules. In the core cleaning stage, a customized VHDL kernel is used to clean heterogeneous data. In the output verification stage, the Spark distributed processing engine receives the cleaning results from the FPGA acceleration unit and integrates and outputs them, thus completing the preset data cleaning task.

[0127] A VHDL cleaning pipeline consisting of N configurable logic modules is constructed based on a stream processing architecture. The evolved rule instruction set is converted into an instruction format recognizable by the FPGA and mapped to the corresponding configurable logic modules in the VHDL cleaning pipeline based on the instruction content. The configurable logic modules include a field recognition module, a missing value repair module, an outlier replacement module, and a format standardization module. Each module can be enabled on demand to achieve dynamic modular cleaning.

[0128] By connecting to heterogeneous data sources through the Spark distributed processing engine, a data heat-aware scheduling mechanism is built based on the heat diffusion path. The data scheduling manager pushes hot data to the FPGA acceleration unit for cleaning and processing, while cold data is cached and awaits scheduling.

[0129] For each batch of heterogeneous data sources pushed, the corresponding rule instruction set identifier is bound to drive the FPGA acceleration unit to load the corresponding configurable logic module and perform cleaning operations; the cleaned heterogeneous data source is returned from the FPGA acceleration unit to the Spark distributed processing engine for formatting and packaging, and the cleaned heterogeneous data source is output.

[0130] Methods for auditing and verifying cleansed heterogeneous data sources include:

[0131] A dual-chain structure with an operation evidence chain and a quality certification chain is constructed. For each data cleaning operation process, the data cleaning operation timestamp, the called cleaning rule ID, the data cleaning flow chart, the unique identifier of the processed heterogeneous data source, the operator and the algorithm subject ID are recorded in real time. The summary hash value of the data cleaning operation event is calculated using the verifiable summary function Merkle Tree to form the operation fingerprint.

[0132] The operation fingerprint is written into the operation evidence chain as a transaction and linked with the hash of the previous block to form a blockchain structure. The operation evidence chain supports smart contracts. When the data cleaning operation is completed, the smart contract automatically triggers the audit verification of the heterogeneous data source after cleaning based on the operation fingerprint.

[0133] The preset temperature threshold is set by the staff. By collecting different temperatures, the average value of multiple temperatures is taken as the preset temperature threshold; similarly, the preset cleaning credibility threshold, preset matching threshold, preset data value density score first threshold, data value density score second threshold, data value density score third threshold, preset latest access time first threshold, latest access time second threshold, latest access time third threshold and preset fitness threshold are set.

[0134] This embodiment constructs a comprehensive heat measurement metric by comprehensively considering access frequency, update frequency, and changes in data value density. Based on a local heat diffusion coupling model, it can capture the hot and cold critical points of multi-source heterogeneous data in a big data environment in real time, greatly improving the accuracy of hot and cold data classification, thereby optimizing data cleaning priorities and strategy selection. Dynamically reflecting the temporal changes and phase transitions in data value density enables the data cleaning system to automatically adjust cleaning rules and task scheduling based on real-time changes in data value, enhancing the adaptability and intelligence of the cleaning process.

[0135] By simulating the spatial diffusion of heat in thermodynamics, it accurately reflects the migration trajectory of data access hotspots in three-dimensional storage space, supports dynamic tiered migration strategies based on heat diffusion paths, effectively avoids migration delays, and improves storage resource utilization and the efficiency of data cleaning tasks. Utilizing preset temperature thresholds and migration trigger mechanisms, it can automatically sense data temperature changes and trigger migration operations, enabling real-time dynamic management of hot and cold data, ensuring that data cleaning tasks are efficiently executed at the appropriate storage level, and enhancing the robustness and responsiveness of the system. The dynamic migration mechanism based on heat diffusion paths replaces traditional manual or periodic migration strategies, enabling automatic sensing and execution of data hot and cold states and migration operations, significantly reducing manual maintenance costs and improving system stability and intelligence.

[0136] Example 2

[0137] See also Figure 2 As shown, this embodiment provides a dynamically configurable data cleaning system based on big data, including:

[0138] The adaptive detection module is used to collect heterogeneous data sources and integrate them with the data source adapter interface. It uses a simulated antigen recognition memory mechanism to identify abnormal patterns in heterogeneous data sources and encode them into antigen fingerprints. Based on the antigen fingerprints, it constructs a data value density scoring function to obtain a data value density score.

[0139] The intelligent tiered storage module constructs a three-dimensional storage topology space based on data value density scores. It uses a lattice migration algorithm to automatically identify the critical points of hot and cold data, triggering tiered migration when the temperature of heterogeneous data sources drops to a preset threshold. It also introduces a thermodynamic diffusion mechanism to simulate the migration process of heterogeneous data sources in the three-dimensional storage topology space as a heat diffusion path.

[0140] The data cleaning orchestration module builds a data cleaning flowchart by dragging and dropping, determines the execution order of cleaning operators in the data cleaning flowchart based on topological sorting, and automatically schedules cleaning tasks;

[0141] The ecological rule evolution module builds a Darwinian evolution system for cleaning rules, periodically evolves the cleaning rules, defines the fitness function of the cleaning rules, simulates the natural selection mechanism to dynamically optimize the cleaning strategy, and outputs the evolved rule instruction set;

[0142] The hardware-accelerated cleaning module builds a heterogeneous processing architecture and customizes the VHDL kernel to clean heterogeneous data sources. It collaborates with Spark's batch reading mechanism based on the heat diffusion path and calls the evolved rule instruction set to complete the preset data cleaning tasks.

[0143] The trusted cleaning evidence module builds a dual-chain structure with an operation evidence chain and a quality certification chain to audit and verify heterogeneous data sources after cleaning. It uses a verifiable summary function to record the cleaning credibility of the heterogeneous data sources after cleaning. When the cleaning credibility is less than the preset cleaning credibility threshold, the re-cleaning mechanism is triggered and the cleaning rules are updated.

[0144] Since the electronic device introduced in this embodiment is an electronic device used to implement a dynamic configurable data cleaning method based on big data and its system in the embodiment of this application, based on the dynamic configurable data cleaning method based on big data and its system introduced in the embodiment of this application, technical personnel in this field can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not introduced in detail here. As long as technical personnel in this field implement the electronic device used by a dynamic configurable data cleaning method based on big data and its system in the embodiment of this application, it falls within the scope of protection of this application.

[0145] The above formulas are all dimensionless and numerical calculations. The formulas are obtained by collecting a large amount of data and performing software simulation to obtain the most recent real situation. The preset parameters and thresholds in the formulas are set by technicians in this field according to actual conditions.

[0146] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiment. All technical solutions based on the principles of the present invention are within the scope of protection of the present invention. It should be noted that for users of ordinary skill in the art, various improvements and modifications that do not depart from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A dynamic and configurable data cleaning method based on big data, characterized in that: include: S1. Collect heterogeneous data sources and integrate them with data source adapter interfaces. Using a simulated antigen recognition memory mechanism, abnormal patterns in heterogeneous data sources are identified and encoded as antigen fingerprints. A data value density scoring function is constructed based on the antigen fingerprint to obtain a data value density score. S2. Construct a three-dimensional storage topology space based on data value density scores. A lattice migration algorithm is used to automatically identify the critical points of hot and cold data. When the temperature of heterogeneous data sources drops to a preset threshold, layered migration is triggered. A thermodynamic diffusion mechanism is introduced to simulate the migration process of heterogeneous data sources in the three-dimensional storage topology space as a heat diffusion path. S3. Build a data cleaning flowchart by dragging and dropping, determine the execution order of the cleaning operators in the data cleaning flowchart based on topological sorting, and automatically schedule cleaning tasks; S4. Construct a Darwinian evolution system for cleaning rules, perform periodic evolution on the cleaning rules, define the fitness function of the cleaning rules, simulate the natural selection mechanism to dynamically optimize the cleaning strategy, and output the evolved rule instruction set; S5: Build a heterogeneous processing architecture and customize the VHDL kernel to clean heterogeneous data sources. Collaborate with Spark's batch reading mechanism based on the heat diffusion path and call the evolved rule instruction set to complete the preset data cleaning task. S6. Build a dual-chain structure with an operation evidence chain and a quality certification chain to audit and verify heterogeneous data sources after cleaning; A verifiable summary function is used to record the cleaning credibility of the heterogeneous data source after cleaning; when the cleaning credibility is less than the preset cleaning credibility threshold, the re-cleaning mechanism is triggered and the cleaning rules are updated.

2. A dynamic configurable data cleaning method based on big data according to claim 1, characterized in that: The method for identifying abnormal patterns in heterogeneous data sources and encoding them into antigen fingerprints includes: Heterogeneous data sources include structured data, semi-structured data, unstructured data, streaming data, and system operation and maintenance data. Standard deviation normalization is performed on heterogeneous data sources to convert them into standardized data formats. Constructing a simulated antibody library, defining each antibody in the simulated antibody library as an adaptable and variable antibody template; the antibody template is any one of a K-nearest neighbor model-based template, a cluster center vector-based template, or a small sample neural network-based template; The matching degree between the heterogeneous data source and the antibody template is calculated using a similarity detection mechanism; the similarity detection mechanism can be any of cosine similarity, Euclidean distance, or kernel function similarity. If the matching degree is less than a preset matching threshold, it is identified as an abnormal pattern and marked as a simulated antigen. The simulated antigen is fingerprinted, and the fingerprint encoding process includes feature dimensionality reduction, fingerprint hash mapping and feature vector normalization to generate an antigen fingerprint containing data identification, anomaly category, timestamp and feature hash value.

3. A dynamic configurable data cleaning method based on big data according to claim 2, characterized in that: The method for obtaining the data value density score includes: Multi-dimensional information extraction is performed on antigen fingerprints to construct data feature vectors. The multi-dimensional information includes antigen occurrence frequency, abnormality level, fingerprint similarity distribution, historical trigger records and context information entropy. A data value density scoring function is constructed based on the data feature vector. The data value density scoring function is a nonlinear weighted function. Each antigen fingerprint is input into the corresponding data value density scoring function to calculate the corresponding data value density score.

4. A dynamic configurable data cleaning method based on big data according to claim 3, characterized in that: The method for constructing the three-dimensional storage topological space includes: Design a three-dimensional coordinate system for the three-dimensional storage topology space. Combine X, Y, and Z as the three-dimensional coordinates of the three-dimensional storage topology space. Organize all three-dimensional coordinates into an integer grid to form a logical three-dimensional storage topology space. Each dimension is defined as the X-axis, Y-axis, and Z-axis. The X-axis represents the value density dimension, the Y-axis represents the data structure dimension, and the Z-axis represents the time life cycle dimension. Establish grading and coding rules for each dimension. For the X-axis, normalize all data value density scores and linearly map them to the [0,1] scoring interval. Define stratification levels based on the scoring interval, and preset the first, second, and third thresholds for data value density scores. If the data value density score is greater than or equal to the second threshold of the data value density score and less than or equal to the third threshold of the data value density score, it is classified as the high value tier; if the data value density score is greater than or equal to the first threshold of the data value density score and less than the second threshold of the data value density score, it is classified as the medium value tier; if it is less than the first threshold of the data value density score, it is classified as the low value tier; For the Y axis, set static mapping codes for the data structure types included in the data structure dimension. The data structure types include structured data, semi-structured data, and unstructured data. For structured data, let Y = 0; for semi-structured data, let Y = 1; for unstructured data, let Y = 2. The data structure dimension is extensible and supports subclasses. When Y = 3, it indicates mixed structure data. For the Z axis, the time life cycle stage is defined according to the most recent access time, and the mapping rules are set; the first threshold of the most recent access time, the second threshold of the most recent access time, and the third threshold of the most recent access time are preset; when the most recent access time is less than the first threshold of the most recent access time, Z=0; when the most recent access time is greater than or equal to the first threshold of the most recent access time and less than the second threshold of the most recent access time, Z=1; when the most recent access time is greater than the second threshold of the most recent access time, Z=2.

5. A dynamic configurable data cleaning method based on big data according to claim 4, characterized in that: The method for obtaining the heat diffusion path includes: In the three-dimensional storage topology space, each coordinate point is regarded as a lattice node, which contains m data units. For each data unit , define its current heat value as ; Treat the three-dimensional coordinate system as a spatial lattice grid and define each lattice node The average heat is ; Taking into account the spatial correlation of thermal diffusion, the local thermal diffusion coefficient is introduced , establish a lattice heat diffusion coupling model and preset a temperature threshold. When the temperature of the heterogeneous data source drops to the preset temperature threshold, the hierarchical migration judgment is triggered; the heterogeneous data source with a temperature less than or equal to the preset temperature threshold is identified as cold data, and the heterogeneous data source with a temperature greater than the preset temperature threshold is identified as hot data; A data migration trigger mechanism is defined, and hierarchical migration operations are performed when the trigger conditions of the data migration trigger mechanism are met. The heat diffusion path of heterogeneous data sources is migrated from high-temperature lattice nodes to low-temperature lattice nodes to simulate the gradual release of heat and the migration of data access hotspots. The heat diffusion path is the spatial trajectory of lattice node temperature changes and migration operations.

6. A dynamic configurable data cleaning method based on big data according to claim 5, characterized in that: The method for automatically scheduling cleaning tasks includes: Preset data cleaning operator library. Operator nodes can be selected from the library by dragging and dropping to build a data cleaning flowchart. Directed edges of the data cleaning flowchart are generated based on the connection relationship between operator nodes. Each directed edge is automatically labeled with data dependency. The data cleaning flow chart is formalized as a directed acyclic graph, and the operator nodes in the directed acyclic graph are sorted based on the topological sorting algorithm to generate an operator execution sequence that satisfies the data dependencies, where the operator nodes without dependencies are marked as executable in parallel; executable task units are generated according to the operator execution sequence, and the cleaning tasks are automatically scheduled for execution in the heterogeneous processing architecture.

7. A dynamic configurable data cleaning method based on big data according to claim 6, characterized in that: The method for obtaining the evolved rule instruction set includes: Preset data cleaning rules and initialize the data cleaning rule set as a rule population. The cleaning rules are represented by encoding, and the initial rule population is generated from the historical cleaning rule library. Define the rule fitness function, which comprehensively considers cleaning accuracy, cleaning coverage, and execution efficiency, and comprehensively evaluates rule performance through weighted summation. In each evolution cycle, the rule with the highest fitness is selected and retained according to the rule fitness function; a crossover operation is performed on the selected cleaning rules, and new cleaning rules are generated by combining the characteristics of the cleaning rules; a mutation operation is performed on the new cleaning rules generated by the crossover; A fitness threshold is preset, and the cleaning rules with fitness less than the preset fitness threshold are replaced with the newly evolved cleaning rules; after L evolution cycles, the cleaning rules with fitness reaching the preset fitness threshold are selected to form the final rule instruction set.

8. A dynamic configurable data cleaning method based on big data according to claim 7, characterized in that: The method of calling the evolved rule instruction set to complete the preset data cleaning task includes: Build a heterogeneous processing architecture consisting of an FPGA acceleration unit, a Spark distributed processing engine, and a data scheduling manager. Divide the data cleaning process into three stages: preprocessing, core cleaning, and output verification. In the preprocessing stage, the Spark distributed processing engine is used to batch divide, pre-cache, and bind cleaning rules to heterogeneous data. In the core cleaning phase, a customized VHDL kernel is used to clean heterogeneous data. In the output verification phase, the Spark distributed processing engine receives the cleaning results from the FPGA acceleration unit and integrates and outputs them, thus completing the preset data cleaning task. A VHDL cleaning pipeline consisting of N configurable logic modules is constructed based on a stream processing architecture. The evolved rule instruction set is converted into an instruction format recognizable by the FPGA and mapped to the corresponding configurable logic modules in the VHDL cleaning pipeline based on the instruction content. The configurable logic modules include a field recognition module, a missing value repair module, an outlier replacement module, and a format standardization module. Each module can be enabled on demand to achieve dynamic modular cleaning. By connecting to heterogeneous data sources through the Spark distributed processing engine, a data heat-aware scheduling mechanism is built based on the heat diffusion path. The data scheduling manager pushes hot data to the FPGA acceleration unit for cleaning and processing, while cold data is cached and awaits scheduling. For each batch of heterogeneous data sources pushed, the corresponding rule instruction set identifier is bound to drive the FPGA acceleration unit to load the corresponding configurable logic module and perform cleaning operations; the cleaned heterogeneous data source is returned from the FPGA acceleration unit to the Spark distributed processing engine for formatting and packaging, and the cleaned heterogeneous data source is output.

9. A dynamic configurable data cleaning method based on big data according to claim 8, characterized in that: The method for auditing and verifying the cleaned heterogeneous data source includes: A dual-chain structure with an operation evidence chain and a quality certification chain is constructed. For each data cleaning operation process, the data cleaning operation timestamp, the called cleaning rule ID, the data cleaning flow chart, the unique identifier of the processed heterogeneous data source, the operator and the algorithm subject ID are recorded in real time. The summary hash value of the data cleaning operation event is calculated using the verifiable summary function Merkle Tree to form the operation fingerprint. The operation fingerprint is written into the operation evidence chain as a transaction and linked with the hash of the previous block to form a blockchain structure. The operation evidence chain supports smart contracts. When the data cleaning operation is completed, the smart contract automatically triggers the audit verification of the heterogeneous data source after cleaning based on the operation fingerprint.

10. A dynamically configurable data cleaning system based on big data, used to implement a dynamically configurable data cleaning method based on big data according to any one of claims 1 to 9, characterized in that: include: Adaptive detection module, used to collect heterogeneous data sources and integrate heterogeneous data sources with data source adapter interfaces; Using a simulated antigen recognition memory mechanism, abnormal patterns in heterogeneous data sources are identified and encoded as antigen fingerprints. A data value density scoring function is constructed based on the antigen fingerprint to obtain a data value density score. The intelligent tiered storage module constructs a three-dimensional storage topology space based on data value density scores. It uses a lattice migration algorithm to automatically identify the critical points of hot and cold data, triggering tiered migration when the temperature of heterogeneous data sources drops to a preset threshold. It also introduces a thermodynamic diffusion mechanism to simulate the migration process of heterogeneous data sources in the three-dimensional storage topology space as a heat diffusion path. The data cleaning orchestration module builds a data cleaning flowchart by dragging and dropping, determines the execution order of cleaning operators in the data cleaning flowchart based on topological sorting, and automatically schedules cleaning tasks; The ecological rule evolution module builds a Darwinian evolution system for cleaning rules, periodically evolves the cleaning rules, defines the fitness function of the cleaning rules, simulates the natural selection mechanism to dynamically optimize the cleaning strategy, and outputs the evolved rule instruction set; The hardware-accelerated cleaning module builds a heterogeneous processing architecture and customizes the VHDL kernel to clean heterogeneous data sources. It collaborates with Spark's batch reading mechanism based on the heat diffusion path and calls the evolved rule instruction set to complete the preset data cleaning tasks. The trusted cleaning evidence module builds a dual-chain structure with an operation evidence chain and a quality certification chain to audit and verify heterogeneous data sources after cleaning; A verifiable summary function is used to record the cleaning credibility of the heterogeneous data source after cleaning; when the cleaning credibility is less than the preset cleaning credibility threshold, the re-cleaning mechanism is triggered and the cleaning rules are updated.

Images