A big data-oriented data correction fusion system and method

Abstract

The application relates to the technical field of industrial data processing, in particular to a data correction and fusion system and method for big data, which comprises a context-related data correction module. Based on the to-be-corrected industrial manual input items and the captured data, the context-related data correction module utilizes the equipment number and the process stage information attached to the input items. In the application, a dynamic effective dictionary based on context information such as the equipment number and the process stage is constructed, and multi-dimensional similarity calculation such as character conversion editing frequency and pronunciation code comparison is combined to automatically correct the industrial manual input items with high confidence, thereby improving the basic data quality from the source of data generation. Furthermore, the corrected data items and the automatically collected sensor readings are uniformly abstracted into data nodes carrying unique identifiers, and all calculation and fusion operations are recorded as directed operation edges between the nodes to construct a full-cycle data pedigree chart.

Description

Technical Field

[0001] This invention relates to the field of industrial data processing technology, and in particular to a data correction and fusion system and method for big data. Background Technology

[0002] Industrial data processing technology refers to a set of technologies used in industrial production environments such as manufacturing, energy, chemical, and public utilities to systematically collect, transmit, store, clean, fuse, analyze, and visualize multi-source heterogeneous data from various equipment, sensors, control systems, and manual operations.

[0003] The shortcomings of existing technologies lie in their reliance on static, global validation rules and dictionaries for data quality assurance. When processing manually entered industrial data with strong contextual relationships, these methods struggle to dynamically adapt to the effective input range of specific workstations or process stages, resulting in high error recognition rates and weak correlation in correction suggestions. For example, in a large chemical plant, when validating material entry errors in reactor A, the system might use a dictionary containing all materials in the plant. This not only reduces matching efficiency but could also incorrectly correct a misspelled proprietary catalyst name to a material used in workshop B that should never appear there, introducing more serious data logic errors. Therefore, improvements are needed. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by proposing a data correction and fusion system and method for big data.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A data correction and fusion system for big data includes: The context-related data correction module, based on the manually entered industrial entries to be corrected and the captured data, uses the equipment number and process stage information attached to the entered entries to retrieve the associated material batch number records, operation instruction records and fault code records from the historical database, summarizes them into a temporary valid dictionary, calculates the minimum number of single character insertions, deletions and replacements required to convert between the entered entries and each entry in the dictionary, and at the same time converts the entered entries and each entry in the dictionary into coded characters according to preset pronunciation rules. By comparing the consistency of the coded characters, a comprehensive confidence score is generated, and automatic correction or manual confirmation prompts are determined according to preset thresholds to form a corrected data entry set; The data derivation genealogy construction module, based on the corrected data entry set and automatically collected sensor data readings, creates a data node with a unique identifier, value and timestamp for each independent data item. When several data nodes jointly generate a new data item, the generation operation is recorded as an operation edge, and a connection relationship is established from the input data node to the output data node, thus constructing a directed acyclic graph structure and forming a full-cycle data genealogy.

[0006] Preferably, the system further includes: The cleaning operation influence domain analysis module, based on the full-cycle data spectrum diagram, when a cleaning operation instruction for a certain data node is received, places the unique identifier of that node into the processing queue. Then, it cyclically retrieves node identifiers from the queue, searches for all directed edges starting from that node in the spectrum diagram, adds the downstream node identifiers pointed to by the edges to the node identifier set to be updated, and also places the downstream node identifiers into the processing queue. This process is repeated until the processing queue is empty, thus obtaining the complete node identifier set to be updated. The backtracking and recalculation correction module, based on the set of node identifiers to be updated and the preset recalculation strategy, determines a reasonable calculation order according to the dependency relationship between nodes in the phylogenetic graph. First, the source node that is directly corrected is recalculated, triggering the calculation task corresponding to the associated operation edge. After the calculation task is completed and a new data node is generated, all calculation tasks with the new data node as input are triggered step by step, and the process is repeated until all nodes in the set of node identifiers to be updated have been recalculated and updated, resulting in a consistent data update set.

[0007] Preferably, the context-related data correction module includes: The context dictionary construction submodule, based on the manually entered industrial entries to be corrected and the data captured, extracts the equipment number and process stage from the entry metadata as a composite query key, initiates a query request to the historical database, and filters out unique material batch numbers, operation instructions and fault code records from the returned results. Then, the deduplicated record entries are compiled into a key-value pair set to obtain the associated context dictionary. The multidimensional similarity calculation submodule, based on the associated context dictionary, uses a dynamic programming matrix filling method to calculate the minimum number of character operations required for mutual conversion between the manually entered industrial entries to be corrected and each entry in the dictionary. At the same time, it retains the first letter of the entries and words and maps the remaining consonants to numbers for encoding. Then, it compares the matching degree of the encoded strings to generate a comprehensive similarity matrix. The confidence correction decision submodule, based on the comprehensive similarity matrix, performs a weighted summation of the similarity scores in the matrix and the frequency of occurrence of the term in the associated context dictionary, calculates a comprehensive confidence score for each candidate term, and then compares the highest score with two preset thresholds, high and low. If the score is higher than the high threshold, the original term is directly replaced; if it is between the two thresholds, it is marked and a candidate list is provided to obtain the corrected data item set.

[0008] Preferably, the data derivation phylogenetic map construction module includes: The data node definition submodule, based on the corrected data entry set and the automatically collected sensor readings, traverses all data items, instantiates a structured data object for each data item containing a UUID as a unique identifier, the original value as the load, and the collection time as a timestamp, and stores all instantiated data objects into a memory set to establish an initial data node set; The operation relationship recording submodule, based on the initial data node set, captures each calculation event of averaging, filtering, or fusion by deploying interceptors in the data processing flow, and extracts the name of the operation function, the list of unique identifiers of the input data items, and the unique identifier of the output data items from the event context, encapsulates the information into a triplet record, and establishes a data operation relationship list. The genealogical graph link generation submodule, based on the initial data node set and the data operation relationship list, traverses each record in the data operation relationship list, searches for the node object corresponding to the input node identifier and the node object corresponding to the output node identifier in the graph database, and then creates directed edges carrying operation type attributes between the node objects, with the direction of the edges pointing from the input to the output. All nodes and edges are aggregated to obtain the full-cycle data genealogical graph.

[0009] Preferably, the cleaning operation influence domain analysis module includes: The source point cleaning submodule, based on the full-cycle data spectrum diagram, parses the unique identifier of the target data node contained in the cleaning operation instruction, directly retrieves the node object corresponding to the identifier using the index function of the graph database, adds the unique identifier as the first element to the front of a double-ended queue, and initializes a hash set for storing the results to obtain the initial contaminated node queue. The forward tracing submodule for the impact path, based on the initial contaminated node queue and the full-cycle data spectrum, pops a node identifier from the head of the queue as long as the queue is not empty, queries all outgoing edges associated with this node, obtains the unique identifiers of all downstream adjacent nodes pointed to by the edges, adds the identifiers one by one to the tail of the queue, and records the parent-child node pairs traversed, thus establishing the path of the downstream affected nodes.

[0010] Preferably, the cleaning operation influence domain analysis module further includes: The module that summarizes the affected domain set adds the unique identifiers of all downstream adjacent nodes recorded during the tracing process to a pre-initialized hash set based on the path of the downstream affected nodes. It automatically removes duplicates by utilizing the uniqueness of the hash set elements. After the entire traversal process is completed, all elements in the hash set are exported as a list to generate a set of node identifiers to be updated.

[0011] Preferably, the dependency backtracking and recalculation correction module includes: The recomputation sequence planning submodule calculates the in-degree of each node in the set based on the node identifier set to be updated and the full-cycle data spectrum. It puts all nodes with an in-degree of zero into a queue, then takes nodes out of the queue in a loop, adds them to the execution sequence, and decrements the in-degree of all downstream adjacent nodes by one. If the in-degree is zero after decrementing, the downstream node is enqueued to generate an ordered recomputation task list. The cascaded computation execution submodule, based on the ordered recomputation task list, traverses each task identifier in the list in sequence, retrieves the computation operation type and all input nodes corresponding to the task from the full-cycle data spectrum, obtains the latest value of the input node, then calls the corresponding computation function to perform the operation, and stores the operation result with the task identifier to obtain a newly generated data node set.

[0012] Preferably, the dependency backtracking and recalculation correction module further includes: The data status update submodule, based on the newly generated data node set, traverses each new data node in the set, uses a unique identifier to find the corresponding old node in the full-cycle data hierarchy graph, and updates the old node's value, timestamp, and status mark with the corresponding content of the new node. After all update operations are completed, the updated nodes together constitute a consistent data update set.

[0013] This invention also provides a data acquisition and fusion method, comprising the following steps: Based on the manually entered industrial entries to be corrected and the captured data, the system uses the equipment number and process stage information attached to the entered entries to retrieve related material batch number records, operation instruction records and fault code records from the historical database, and summarizes them into a temporary effective dictionary. It calculates the minimum number of single character insertions, deletions and replacements required to convert between the entered entries and each entry in the dictionary, and at the same time converts the entered entries and each entry in the dictionary into coded characters according to preset pronunciation rules. By comparing the consistency of the coded characters, a comprehensive confidence score is generated. Based on the preset threshold, automatic correction or manual confirmation prompts are determined to form a corrected data entry set. Based on the corrected data entry set and the automatically collected sensor data readings, a data node with a unique identifier, value and timestamp is created for each independent data item. When several data nodes jointly generate a new data item, the generation operation is recorded as an operation edge, and a connection relationship is established from the input data node to the output data node to construct a directed acyclic graph structure and form a full-cycle data spectrum diagram. Based on the full-cycle data spectrum, when a cleaning operation instruction for a certain data node is received, the unique identifier of the node is placed in the processing queue. Then, the node identifier is retrieved from the queue in a loop. All directed edges originating from the node are searched in the spectrum. The downstream node identifiers pointed to by the edges are added to the node identifier set to be updated. At the same time, the downstream node identifiers are also placed in the processing queue. This process is repeated until the processing queue is empty, and the complete node identifier set to be updated is obtained. Based on the set of node identifiers to be updated and the preset recalculation strategy, a reasonable calculation order is determined according to the dependency relationship between nodes in the phylogenetic graph. First, the source node that is directly corrected is recalculated, triggering the calculation task corresponding to the associated operation edge. After the calculation task is completed and a new data node is generated, all calculation tasks with the new data node as input are triggered step by step, and the process is repeated until all nodes in the set of node identifiers to be updated have been recalculated and updated, resulting in a consistent data update set.

[0014] Compared with the prior art, the advantages and positive effects of the present invention are as follows: In this invention, a dynamic and effective dictionary based on contextual information such as equipment number and process stage is constructed. Combined with multi-dimensional similarity calculations such as character conversion and editing times and pronunciation encoding comparison, high-confidence automatic correction is performed on manually entered industrial data, improving the quality of basic data from its source. Furthermore, the corrected data entries and automatically collected sensor readings are uniformly abstracted into data nodes carrying unique identifiers. All calculation and fusion operations are recorded as directed operation edges between nodes, constructing a full-cycle data genealogy graph. This makes the implicit and complex derivative relationships between data explicit and structured, providing a solid foundation for data lifecycle traceability. When any data node is cleaned or corrected, efficient and complete graph traversal can be performed along the directed edges of the genealogy graph, quickly defining the set of node identifiers to be updated composed of all affected downstream data, solving the problem of uncertain data contamination spread range. Based on the dependencies revealed by the genealogy diagram, an ordered cascaded recalculation is automatically performed on the affected node set, ensuring that the data state from the source to the end can be automatically restored to logical consistency, guaranteeing the final accuracy and integrity of the entire data set, and avoiding the problem of global data contradictions caused by local data correction. Attached Figure Description

[0015] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0017] Please see Figure 1 This invention provides a technical solution: a data correction and fusion system for big data, comprising: The context-related data correction module, based on the manually entered industrial entries to be corrected and the captured data, uses the equipment number and process stage information attached to the entered entries to retrieve the associated material batch number records, operation instruction records and fault code records from the historical database, summarizes them into a temporary valid dictionary, calculates the minimum number of single character insertions, deletions and replacements required to convert between the entered entries and each entry in the dictionary, and at the same time converts the entered entries and each entry in the dictionary into coded characters according to preset pronunciation rules. By comparing the consistency of the coded characters, a comprehensive confidence score is generated, and automatic correction or manual confirmation prompts are determined according to preset thresholds to form a corrected data entry set; The data derivation genealogy construction module, based on the corrected data entry set and automatically acquired sensor data readings, creates a data node with a unique identifier, value and timestamp for each independent data item. When several data nodes jointly generate a new data item, the generation operation is recorded as an operation edge, and a connection relationship is established from the input data node to the output data node, constructing a directed acyclic graph structure to form a full-cycle data genealogy. The cleaning operation impact domain analysis module, based on the full-cycle data phylogenetic graph, when a cleaning operation instruction for a certain data node is received, places the unique identifier of that node into the processing queue. Then, it cyclically retrieves node identifiers from the queue, searches for all directed edges originating from that node in the phylogenetic graph, adds the downstream node identifiers pointed to by the edges to the node identifier set to be updated, and simultaneously adds the downstream node identifiers to the processing queue. This process is repeated until the processing queue is empty, thus obtaining the complete node identifier set to be updated. The backtracking and recalculation correction module, based on the set of node identifiers to be updated and the preset recalculation strategy, determines a reasonable calculation order according to the dependency relationship between nodes in the phylogenetic graph. First, the source node that is directly corrected is recalculated, triggering the calculation task corresponding to the associated operation edge. After the calculation task is completed and a new data node is generated, all calculation tasks with the new data node as input are triggered step by step, and the process is repeated until all nodes in the set of node identifiers to be updated have been recalculated and updated, resulting in a consistent data update set.

[0018] The context-related data correction module includes: The context dictionary construction submodule, based on the manually entered industrial entries to be corrected and the data captured, extracts the equipment number and process stage from the entry metadata as a composite query key, initiates a query request to the historical database, and filters out unique material batch numbers, operation instructions and fault code records from the returned results. Then, the deduplicated record entries are compiled into a key-value pair set to obtain the associated context dictionary. The multidimensional similarity calculation submodule, based on the associated context dictionary, uses dynamic programming matrix filling to calculate the minimum number of character operations required for mutual conversion between the manually entered industrial entries to be corrected and each entry in the dictionary. At the same time, it retains the first letter of the entries and words and maps the remaining consonants to numbers for encoding. Then, it compares the matching degree of the encoded strings to generate a comprehensive similarity matrix. The confidence correction decision submodule, based on the comprehensive similarity matrix, performs a weighted summation of the similarity scores in the matrix and the frequency of occurrence of the term in the associated context dictionary. It calculates a comprehensive confidence score for each candidate term, and then compares the highest score with two preset thresholds. If the score is higher than the higher threshold, the original term is directly replaced; if it is between the two thresholds, it is marked and a candidate list is provided, thus obtaining the corrected data item set.

[0019] Specifically, firstly, a network packet monitoring program deployed on the aggregation switch or server network card of the industrial control network uses packet capture technology to capture data streams flowing to the human-machine interface or manufacturing execution system application server in real time. The monitoring program is configured with filtering rules for the target server IP address and port (e.g., IP 192.168.10.50, port 443). For traffic encrypted via HTTPS, a man-in-the-middle proxy with decryption capabilities is deployed at the network egress. By pre-installing a trusted root certificate on the operating terminal, the TLS / SSL traffic is decrypted, and the body of the HTTP request message is parsed. The original text entered by the operator is located and extracted from JSON or form data, such as "batch MTRL-00X", along with accompanying metadata such as the equipment number "reactor R-101" and the process stage "mixing stage". As an alternative acquisition method, the system is also configured with a web scraping script based on a browser automation framework. This script is set to automatically start if the primary packet monitoring method fails to capture new data for more than a preset timeout period. The timeout duration is set based on the statistical analysis of historical data entry intervals. Specifically, it is the sum of the average time interval of all manually entered items within 24 consecutive hours and one standard deviation. For example, if the average interval is 30 seconds and the standard deviation is 8 seconds, the timeout duration is set to 38 seconds. After the capture script starts, it will simulate a user logging into the MES system, navigate to the data display page, and accurately capture the latest entered items and their associated metadata according to the predefined XPath path (e.g., / / div[@id='recent-entries'] / table / tbody / tr[1] / td[@class='entry-text']). The polling cycle of the capture script is determined by the historical peak entry frequency and system performance constraints. For example, by analyzing historical data, it is found that there are at most 10 entries per minute during the peak period, that is, one entry every 6 seconds on average. In order to ensure that no data is missed, the basic polling cycle is set to 4 seconds. At the same time, the incremental load of CPU and memory under this frequency is evaluated in the test environment to ensure that its increment does not exceed 5% of the total system load. If it exceeds, it will be 0.The polling cycle increments by 5 seconds until the load requirement is met. After obtaining the manually entered industrial entry to be corrected through any of the above methods, the system immediately extracts the equipment number "Reactor R-101" and the process stage "Mixing Stage" from its metadata. These two pieces of information are combined into a composite query key. This composite query key is used to initiate parallel Structured Query Language (SQL) requests to multiple data tables in the historical database, including the "Production Batch Table," "Operating Procedure Table," and "Equipment Fault Table." The specific query statements are: SELECT DISTINCT Material Batch Number FROM Production Batch Table WHERE Equipment Number = 'Reactor R-101' AND Process Stage = 'Mixing Stage', SELECT DISTINCT Operating Instruction FROM Operating Procedure Table WHERE Equipment Number = 'Reactor R-101' AND Process Stage = 'Mixing Stage', and SELECT DISTINCT Fault Code FROM Equipment Fault Table WHERE For the equipment number 'Reactor R-101', from multiple result sets returned by the database, the system performs secondary deduplication on each result list for three categories: material batch number, operating instruction, and fault code, using the built-in Set data structure. These three lists of deduplicated unique records are then compiled into a structured key-value pair set, where the key is a fixed category name string, and the value is the corresponding unique record list. This ultimately yields a context dictionary specific to the item to be corrected.

[0020] Based on a context-dependent dictionary, such as one containing entries like "MATERIAL_BATCH_X1" and "FAULT_CODE_Z23", for each manually entered industrial entry to be corrected, such as "MTERIAL_BTCH_X1", the system uses a dynamic programming algorithm to calculate the edit distance between it and each entry in the dictionary (e.g., "MATERIAL_BATCH_X1"). Specifically, a two-dimensional matrix is ​​constructed for the two strings, where the number of rows and columns are the lengths of the two strings plus one, respectively. Line 1 Column element values Represents the beginning of the first string The first character is converted into the second string. The minimum number of operations (insertion, deletion, and replacement, each costing 1) required to create n characters is calculated as follows: if the first string's nth character... The character equals the second string's first character. If there are 1 character, then ,otherwise ,in Corresponding deletion operation, Corresponding to the insertion operation, For the corresponding replacement operation, the final value in the bottom right corner of the matrix is ​​the edit distance between the two. For example, the edit distance between "MTERIAL_BTCH_X1" and "MATERIAL_BATCH_X1" is 2 (one replacement of 'E' with 'A', one insertion of 'A'). Normalizing this distance gives the edit distance similarity. Simultaneously, the entry to be corrected and each word in the dictionary undergo speech encoding processing. This involves retaining the first letter of the string and then converting the remaining consonants into numerical codes according to preset mapping rules (e.g., BFPV maps to 1, CGJKQSXZ to 2, DT to 3, L to 4, MN to 5, R to 6, vowels AEIOUHWY and separators are ignored or mapped to 0). For example, "MTERIAL_BTCH_X1" might be encoded as "M3441321", and the dictionary entry "MATERIAL_BATCH_X1" would also be encoded as "M3441321". The matching degree of these two encoded strings is then compared, for example, by calculating their Hamming distance or performing an exact match check. If they are completely identical, the speech similarity is considered... Otherwise, assign values ​​according to the degree of difference (e.g.) The calculated edit distance similarity and speech similarity A weighted fusion is performed to form a comprehensive similarity score, for example... The weight and Based on experience, for example and ,make sure This calculation is performed on both the entry to be corrected and all entries in the dictionary, ultimately forming a comprehensive similarity matrix. The rows of the matrix represent the entry to be corrected, the columns represent the dictionary entries, and the element values ​​are the corresponding comprehensive similarity scores.

[0021] Based on a comprehensive similarity matrix, which stores the comprehensive similarity score between each entry to be corrected and all candidate entries in the associated context dictionary, the system further calculates the comprehensive confidence score of each candidate entry by combining the frequency of occurrence of the entry in the associated context dictionary (i.e., in historical data). The frequency of this term is calculated by counting the number of times it was retrieved from the historical database when constructing the context dictionary, and then normalizing the result by dividing by the total number of occurrences of all terms in that category. For example, if the term "MTRL-001" appears 50 times in the historical data, and the total number of occurrences of all material batch number terms is 500, then its frequency is... For each entry to be corrected and its corresponding candidate terms Its overall confidence score Calculated using the following weighted formula: ,in The term to be corrected and the candidate terms are obtained from the comprehensive similarity matrix. The overall similarity score, It is a candidate term. The normalization frequency of occurrence, weight and The settings are based on the sensitivity and accuracy requirements of the calibration, for example... and ,and After calculating the overall confidence score of all candidate terms, the candidate term with the highest score and its corresponding highest score are selected. Then compare this highest score with a preset high threshold. and low threshold The two thresholds were set based on an evaluation of historical correction performance, for example, by testing on a validation set containing 1000 manually entered entries known to be correct or incorrect, and then adjusting... and To optimize the F1 score or balance precision and recall, if the test shows that when If the automatic correction error rate is less than 0.5%, then set... , if If manual review can effectively identify and correct errors, then set... ,if (For example If the highest-scoring candidate term is used, the system will automatically replace the original entry. (For example If the confidence score is between 0.75 and 0.92, the system will mark the original entry as requiring manual confirmation and provide the user interface with a list containing the original entry and the top few (e.g., 3) candidate terms sorted in descending order of confidence score for the user to choose from. (For example If the original entry is not found, it will be marked as low confidence and the user will be prompted to check it. Through this decision-making process, the corrected set of data entries will eventually be formed.

[0022] The data derivation phylogenetic diagram construction module includes: The data node definition submodule, based on the calibrated data entry set and automatically acquired sensor readings, traverses all data items, instantiates a structured data object for each data item containing a UUID as a unique identifier, the original value as the load, and the acquisition time as a timestamp, and stores all instantiated data objects into a memory set to establish the initial data node set; The operation relationship record submodule, based on the initial data node set, captures each calculation event of averaging, filtering, or fusion by deploying interceptors in the data processing flow, and extracts the name of the operation function, the list of unique identifiers of the input data items, and the unique identifier of the output data items from the event context, encapsulates the information into a triple record, and establishes a list of data operation relationships. The genealogical graph link generation submodule, based on the initial data node set and the data operation relationship list, traverses each record in the data operation relationship list, searches for the node object corresponding to the input node identifier and the node object corresponding to the output node identifier in the graph database, and then creates directed edges carrying operation type attributes between the node objects, with the direction of the edges pointing from the input to the output. By aggregating all nodes and edges, a full-cycle data genealogical graph is obtained.

[0023] Specifically, based on the calibrated data entry set (e.g., containing records such as "Material Batch Number: LOT20240115A, Time: 2024-01-15T10:00:00Z") and automatically acquired sensor readings (e.g., "Temperature Sensor T101: 55.2℃, Time: 2024-01-15T10:00:05Z"), the system processes all data items from these sources one by one. For each independent data item, whether from calibrated manual entry or automatic sensor acquisition, a structured data object instance is created. This data object contains three core attributes: a globally unique identifier (UUID), generated using a UUID generation algorithm, for example, generating "3f9a3d2c-7a58-4b2f-8e9a-1c7e2b8d9f0a" as the unique ID of this data item; and the original value of the data item as the object's payload. For example, for a material batch number record, the payload is the string "LOT20240115A", and for a temperature reading, the payload is the floating-point number 55.2, along with the data item's acquisition or generation time as a timestamp, such as the recording time of manually entered items "2024-01-15T10:00:00Z" or the sampling time of sensor data "2024-01-15T10:00:05Z". Additionally, a type field can be added to the object to distinguish the data source, such as "manual_corrected" or "sensor_reading". All these instantiated structured data objects are then uniformly stored in an in-memory data set that supports fast retrieval, for example, using a hash table (HashMap) structure with UUID as the key and the data object itself as the value, or using a list data structure. This set constitutes the system's initial data node set.

[0024] Based on the initial data node set, which contains all raw and corrected data items, when the data is used in subsequent processing flows to calculate new data items—for example, calculating the average of multiple temperature sensor readings, filtering certain data according to specific conditions, or fusing data from different sources—the system captures these computational events through an interception mechanism pre-deployed on these data processing functions or methods. This interception mechanism can leverage the characteristics of Aspect-Oriented Programming (AOP) or be implemented by wrapping the core computational logic with a layer of monitoring code (such as the decorator pattern). When a data processing operation (e.g., a function named `calculate_average_temperature`) is called, this function receives a list of unique identifiers for several input data nodes (e.g., `['uuid_temp_sensor1', 'uuid_temp_sensor2', 'uuid_temp_sensor3']`) as parameters and performs calculations to generate a new data node (e.g., the average temperature node, whose unique identifier is `uuid_avg_temp`). The interceptor is triggered at this point. Key information is extracted from the execution context of the computation event: the name of the operation (e.g., function name `calculate_average_temperature`), a list of unique identifiers for the input data items used in the computation (['uuid_temp_sensor1', 'uuid_temp_sensor2', 'uuid_temp_sensor3']), and a unique identifier for the new data item (i.e., the output data item) produced by the operation (`uuid_avg_temp`). This extracted information is then encapsulated into a well-defined triple record, typically structured as (operation name, [list of input node IDs], output node ID), for example, ('calculate_average_temperature', ['uuid_temp_sensor1', 'uuid_temp_sensor2', 'uuid_temp_sensor3'], 'uuid_avg_temp'). All such captured triple records are sequentially added to a list, thus establishing a list of data operation relationships.

[0025] Based on the initial data node set (where the data nodes have been pre-loaded into the graph database, and each node contains at least attributes such as UUID, value, and timestamp) and the data operation relationship list (where each record is a triplet in the format (operation name, [list of input node IDs], output node ID)), the system begins to construct the data hierarchy graph. Specifically, this involves traversing each triplet record in the data operation relationship list. For example, processing the record ('calculate_average_temperature', ['uuid_temp_sensor1', 'uuid_temp_...') ... For each input node identifier (e.g., 'uuid_temp_sensor1' and 'uuid_temp_sensor2') in the record, the system uses the query interface provided by the graph database (e.g., Neo4j) (such as the Cypher query statement MATCH(n)WHERE En.id='input node ID'RETURNn) to find and retrieve the corresponding existing node object. Similarly, based on the output node identifier (e.g., 'uuid_avg_temp') in the record, the system finds and retrieves the corresponding existing node object. The system retrieves the corresponding output node object (which is newly created and added to the graph database during the operation). After successfully locating all relevant input and output node objects, the system creates directed edges between them. These edges point from each input data node to the jointly generated output data node, representing the direction of data flow and dependencies. For example, an edge is created from the node with ID 'uuid_temp_sensor1' to the node with ID 'uuid_avg_temp', and another edge is created from the node with ID 'uuid_temp_sensor2' to the node with ID 'uuid_avg_temp'. Each created edge carries an attribute that records the specific operation type that caused this relationship. The attribute value is the operation name in the triple record (e.g., the edge attribute is set to operation: 'calculate_average_temperature'). By repeating this process for all records in the data operation relationship list, all data nodes and the directed edges between them representing data generation and transformation relationships are aggregated in the graph database, ultimately forming a complete and traceable full-cycle data genealogy graph.

[0026] The cleaning operation influence domain analysis module includes: The source point cleaning submodule, based on the full-cycle data genealogy diagram, parses the unique identifier of the target data node contained in the cleaning operation instruction, uses the index function of the graph database to directly retrieve the node object corresponding to the identifier, then adds the unique identifier as the first element to the front of a double-ended queue, and initializes a hash set for storing the results to obtain the initial contaminated node queue. The forward tracing submodule of the impact path, based on the initial contaminated node queue and the full-cycle data genealogy graph, pops a node identifier from the head of the queue as long as the queue is not empty, queries all outgoing edges associated with this node, then obtains the unique identifiers of all downstream adjacent nodes pointed to by the edge, adds the identifiers one by one to the tail of the queue, and records the parent-child node pairs traversed, and establishes the downstream affected node path. The module that summarizes the affected domain set adds the unique identifiers of all downstream adjacent nodes recorded during the tracing process to a pre-initialized hash set based on the path of the downstream affected nodes. It automatically removes duplicates by utilizing the uniqueness of the hash set elements. After the entire traversal process is completed, all elements in the hash set are exported as a list to generate a set of node identifiers to be updated.

[0027] Specifically, based on the full-cycle data hierarchy graph, the received cleaning operation command is first parsed. This command is usually a structured message, such as JSON format {"action":"clean", "targetNodeUUID":"uuid_of_node_to_clean"}. The value of the "targetNodeUUID" field is extracted, which is the unique identifier of the target data node. Then, using the built-in index query function of graph databases (such as Neo4j or JanusGraph), this unique identifier is usually used as the node's primary key or a highly indexed attribute to perform a direct retrieval operation. For example, if using Cypher query, it would be similar to MATCH(n{uuid:"uuid_of_node_to_clean"}). The `RETURN` function quickly locates and retrieves the node object instance in the phylogenetic graph that perfectly corresponds to the identifier. After retrieving the node object, its unique identifier, such as "uuid_of_node_to_clean", is added as the first element to the front or tail of a first-in-first-out (FIFO) double-ended queue (Deque). This queue will serve as the waiting queue for breadth-first search (BFS). Simultaneously, the system initializes an empty hash set (HashSet), which is used to store the unique identifiers of all affected downstream nodes during subsequent impact path tracing and automatically deduplicates them using its properties, thereby constructing the final set of node identifiers to be updated. After this stage is completed, an initial contaminated node queue containing a single initial contaminated source node identifier is obtained.

[0028] Based on the initial contaminated node queue and the full-cycle data hierarchy graph, the system initiates a loop process that continues as long as the initial contaminated node queue is not empty. At the beginning of each loop, a node identifier is popped from the head of the queue, representing the node currently to be processed, for example, node N_current. Then, for this N_current identifier, a query is performed on the full-cycle data hierarchy graph to find all directed edges (outgoing edges) originating from N_current and pointing to other nodes. This query could be, for example, MATCH(parent{uuid:'N_current_uuid'})-[rel]->(child)RETURNchild.uuid,parent.uuid, thereby obtaining a list of unique identifiers for all downstream adjacent data nodes directly pointed to by these outgoing edges. For example, if a list of downstream nodes [uuid_child1, uuid_child2, ...] is obtained, for each unique identifier of a downstream neighboring node in the list, if the identifier has not yet been processed (this can be done by querying an visited set or by relying on a hash set in subsequent steps to remove duplicates), it is added one by one to the tail of the initial polluted node queue for subsequent iterations. At the same time, the system records the parent-child node pairs currently traversed. For example, relationships such as (N_current_uuid, uuid_child1) and (N_current_uuid, uuid_child2) are stored in a temporary list or directly passed to subsequent processing. The set of these records together constitutes the path information describing how data pollution spreads from the initial source to the downstream, thereby establishing the path of the downstream affected nodes.

[0029] Based on the unique identifiers of each downstream neighboring node identified in the affected path, these identifiers are obtained by querying the full-cycle data hierarchy during the traversal of the affected path forward tracing submodule. For example, during the tracing process, if node A affects node B, and node B affects nodes C and D, then the set of unique identifiers of downstream neighboring nodes will contain the identifiers of B, C, and D. The system attempts to add these unique identifiers of all downstream neighboring nodes recorded during the tracing process one by one to the empty hash set (HashSet) pre-initialized in the source point cleaning and locating submodule. Due to the data structure characteristics of the hash set, when attempting to add a... When an element already exists in the set (i.e., the unique identifier of a node), the addition operation is automatically ignored, thus naturally completing the deduplication of the affected node identifiers. This ensures that each affected downstream node appears only once in the final set. This addition process is accompanied by a breadth-first search for the entire impact path forward tracing until the queue to be processed is empty, indicating that all reachable downstream impact paths have been explored. After the entire traversal process is completed, the hash set contains the unique identifiers of all nodes directly or indirectly affected by the initial cleaning operation. Finally, all elements in this hash set are exported as a list structure to generate the set of node identifiers to be updated.

[0030] The dependency backtracking and recalculation correction module includes: The recomputation sequence planning submodule calculates the in-degree of each node in the set based on the node identifier set to be updated and the full-cycle data phylogenetic graph. It puts all nodes with an in-degree of zero into a queue, then iteratively takes nodes out of the queue, adds them to the execution sequence, and decrements the in-degree of all downstream adjacent nodes by one. If the in-degree is zero after decrementing, the downstream node is enqueued, generating an ordered recomputation task list. The cascaded computation execution submodule, based on an ordered recomputation task list, traverses each task identifier in the list in sequence, retrieves the computation operation type and all input nodes corresponding to the task from the full-cycle data spectrum, obtains the latest value of the input node, then calls the corresponding computation function to perform the operation, and stores the operation result with the task identifier to obtain a newly generated set of data nodes. The data status update submodule, based on the newly generated set of data nodes, traverses each new data node in the set, uses a unique identifier to find the corresponding old node in the full-cycle data hierarchy, and updates the old node's value, timestamp, and status flag with the corresponding content of the new node. After all update operations are completed, the updated nodes together constitute a consistent data update set.

[0031] Specifically, based on the node identifier set to be updated (a list containing unique identifiers of all nodes affected by data cleaning operations) and the full-cycle data hierarchy graph, the system first traverses each node identifier in the node identifier set to be updated. For each node, its in-degree is calculated in the full-cycle data hierarchy graph. Here, in-degree specifically refers to the number of directed edges whose endpoints are the current node, and the starting points of these edges must also be nodes in the node identifier set to be updated. For example, if node X is in the node identifier set to be updated and depends on nodes A and B, which are also in the set, then its in-degree with respect to this set is 2. If it also depends on node C, which is not in the set, then this dependency is not included in the current in-degree calculation. After completing the in-degree calculation for all nodes in the node identifier set to be updated, the unique identifiers of all nodes with an in-degree calculation result of zero (i.e., nodes that have no predecessor dependencies within the node identifier set to be updated) are placed into a first-in-first-out (FIFO) queue. The column serves as the initial processing queue for topological sorting. Subsequently, the system enters a loop. As long as the processing queue is not empty, a node identifier is taken from the queue and added to the end of an initially empty execution sequence list. This execution sequence list defines the order of node recomputation. For a node that has just been taken from the queue and added to the execution sequence, the system searches for all its direct downstream neighbors in the full-cycle data phylogenetic graph. For each such downstream neighbor, if it also exists in the original set of node identifiers to be updated, its previously calculated in-degree value is decremented by one. If, after the decrement operation, the in-degree of a downstream neighbor becomes zero, it means that all its predecessor dependencies within the set of node identifiers to be updated have been included in the execution sequence. Therefore, the identifier of the downstream neighbor is added to the end of the processing queue. This loop continues until the processing queue is empty. The final generated execution sequence list is the ordered recomputation task list.

[0032] Based on an ordered recomputation task list, which contains unique identifiers for all nodes in the set of nodes to be updated and arranged according to the topological order of data dependencies, the system iterates through each task identifier in the order of the list. For the currently traversed task identifier (i.e., the unique identifier of a node), the system first uses this identifier to search the full-cycle data hierarchy graph to obtain the computation operation type corresponding to the node and a list of unique identifiers for all its direct input data nodes. The computation operation type is usually stored as an attribute of the node or an attribute of its incoming edges, such as "summation," "average calculation," or "specific model inference." The list of identifiers for input nodes is obtained by querying the node's direct predecessor nodes. After obtaining this information, the system retrieves the latest unique identifier for each input node from the full-cycle data hierarchy graph or a temporary cache storing the results of recomputed nodes. For numerical values, if the input node itself is in the ordered recalculation task list and is located before the current task, its value is the latest value that has already been recalculated. Otherwise, if the input node is not in the recalculation list, its current stable value in the genealogy is used. After collecting the latest values ​​of all input nodes and the calculation operation type, the system dynamically calls or assigns the calculation to a predefined specific calculation function or logic corresponding to the calculation operation type, and passes the obtained input value as a parameter to the function to perform the calculation. After the calculation is completed, a new result value and a possibly updated timestamp are generated. The newly generated calculation result is associated with the current node (task) identifier and temporarily stored, for example, in a mapping table where the key is the node identifier and the value is the new calculation result. After traversing all task identifiers in the ordered recalculation task list and completing the corresponding calculations, these stored associated results together obtain the newly generated data node set.

[0033] Based on the newly generated set of data nodes, which contains the identifiers of all nodes that were recalculated and generated with new values ​​during the cascade computation execution phase, along with their corresponding new data (e.g., values, timestamps), the system begins to traverse each new data node entry in this set. For each entry, it contains a unique identifier for the node, as well as information such as the node's new value and new timestamp after recalculation. Using this unique identifier, the system precisely locates the corresponding old node object instance before the recalculation in the full-cycle data hierarchy graph. After finding the old node, the system performs an update operation, specifically updating the old node object's numerical attributes (payl...). The data node entry (oad) is replaced with the new value in the new data node entry, its timestamp attribute is updated to the new generation time or recalculation completion time in the new data node entry, and its status flag is updated as needed, for example, from "may be outdated" or "to be updated" to "verified", "consistent" or "latest". This overwrite update operation ensures that the node status in the genealogy graph reflects the latest calculation results. This traversal and update process continues until all node entries in the newly generated data node set have been processed. All nodes that have completed the updates of values, timestamps and status flags in the genealogy graph together constitute a consistent data update set.

Claims

1. A data correction and fusion system for big data, characterized in that, The system includes: The context-related data correction module, based on the manually entered industrial entries to be corrected and the captured data, uses the equipment number and process stage information attached to the entered entries to retrieve the associated material batch number records, operation instruction records and fault code records from the historical database, summarizes them into a temporary valid dictionary, calculates the minimum number of single character insertions, deletions and replacements required to convert between the entered entries and each entry in the dictionary, and at the same time converts the entered entries and each entry in the dictionary into coded characters according to preset pronunciation rules. By comparing the consistency of the coded characters, a comprehensive confidence score is generated, and automatic correction or manual confirmation prompts are determined according to preset thresholds to form a corrected data entry set; The data derivation genealogy construction module, based on the corrected data entry set and automatically collected sensor data readings, creates a data node with a unique identifier, value and timestamp for each independent data item. When several data nodes jointly generate a new data item, the generation operation is recorded as an operation edge, and a connection relationship is established from the input data node to the output data node, thus constructing a directed acyclic graph structure and forming a full-cycle data genealogy.

2. The data correction and fusion system for big data according to claim 1, characterized in that, The system also includes: The cleaning operation influence domain analysis module, based on the full-cycle data spectrum diagram, when a cleaning operation instruction for a certain data node is received, places the unique identifier of that node into the processing queue. Then, it cyclically retrieves node identifiers from the queue, searches for all directed edges starting from that node in the spectrum diagram, adds the downstream node identifiers pointed to by the edges to the node identifier set to be updated, and also places the downstream node identifiers into the processing queue. This process is repeated until the processing queue is empty, thus obtaining the complete node identifier set to be updated. The backtracking and recalculation correction module, based on the set of node identifiers to be updated and the preset recalculation strategy, determines a reasonable calculation order according to the dependency relationship between nodes in the phylogenetic graph. First, the source node that is directly corrected is recalculated, triggering the calculation task corresponding to the associated operation edge. After the calculation task is completed and a new data node is generated, all calculation tasks with the new data node as input are triggered step by step, and the process is repeated until all nodes in the set of node identifiers to be updated have been recalculated and updated, resulting in a consistent data update set.

3. The data correction and fusion system for big data according to claim 1, characterized in that, The context-related data correction module includes: The context dictionary construction submodule, based on the manually entered industrial entries to be corrected and the data captured, extracts the equipment number and process stage from the entry metadata as a composite query key, initiates a query request to the historical database, and filters out unique material batch numbers, operation instructions and fault code records from the returned results. Then, the deduplicated record entries are compiled into a key-value pair set to obtain the associated context dictionary. The multidimensional similarity calculation submodule, based on the associated context dictionary, uses a dynamic programming matrix filling method to calculate the minimum number of character operations required for mutual conversion between the manually entered industrial entries to be corrected and each entry in the dictionary. At the same time, it retains the first letter of the entries and words and maps the remaining consonants to numbers for encoding. Then, it compares the matching degree of the encoded strings to generate a comprehensive similarity matrix. The confidence correction decision submodule, based on the comprehensive similarity matrix, performs a weighted summation of the similarity scores in the matrix and the frequency of occurrence of the term in the associated context dictionary, calculates a comprehensive confidence score for each candidate term, and then compares the highest score with two preset thresholds, high and low. If the score is higher than the high threshold, the original term is directly replaced; if it is between the two thresholds, it is marked and a candidate list is provided to obtain the corrected data item set.

4. The data correction and fusion system for big data according to claim 1, characterized in that, The data derivation phylogenetic diagram construction module includes: The data node definition submodule, based on the corrected data entry set and the automatically collected sensor readings, traverses all data items, instantiates a structured data object for each data item containing a UUID as a unique identifier, the original value as the load, and the collection time as a timestamp, and stores all instantiated data objects into a memory set to establish an initial data node set; The operation relationship recording submodule, based on the initial data node set, captures each calculation event of averaging, filtering, or fusion by deploying interceptors in the data processing flow, and extracts the name of the operation function, the list of unique identifiers of the input data items, and the unique identifier of the output data items from the event context, encapsulates the information into a triplet record, and establishes a data operation relationship list. The genealogical graph link generation submodule, based on the initial data node set and the data operation relationship list, traverses each record in the data operation relationship list, searches for the node object corresponding to the input node identifier and the node object corresponding to the output node identifier in the graph database, and then creates directed edges carrying operation type attributes between the node objects, with the direction of the edges pointing from the input to the output. All nodes and edges are aggregated to obtain the full-cycle data genealogical graph.

5. The data correction and fusion system for big data according to claim 2, characterized in that, The cleaning operation influence domain analysis module includes: The source point cleaning submodule, based on the full-cycle data spectrum diagram, parses the unique identifier of the target data node contained in the cleaning operation instruction, directly retrieves the node object corresponding to the identifier using the index function of the graph database, adds the unique identifier as the first element to the front of a double-ended queue, and initializes a hash set for storing the results to obtain the initial contaminated node queue. The forward tracing submodule for the impact path, based on the initial contaminated node queue and the full-cycle data spectrum, pops a node identifier from the head of the queue as long as the queue is not empty, queries all outgoing edges associated with this node, obtains the unique identifiers of all downstream adjacent nodes pointed to by the edges, adds the identifiers one by one to the tail of the queue, and records the parent-child node pairs traversed, thus establishing the path of the downstream affected nodes.

6. The data correction and fusion system for big data according to claim 5, characterized in that, The cleaning operation influence domain analysis module also includes: The module that summarizes the affected domain set adds the unique identifiers of all downstream adjacent nodes recorded during the tracing process to a pre-initialized hash set based on the path of the downstream affected nodes. It automatically removes duplicates by utilizing the uniqueness of the hash set elements. After the entire traversal process is completed, all elements in the hash set are exported as a list to generate a set of node identifiers to be updated.

7. The data correction and fusion system for big data according to claim 2, characterized in that, The dependency backtracking and recalculation correction module includes: The recomputation sequence planning submodule calculates the in-degree of each node in the set based on the node identifier set to be updated and the full-cycle data spectrum. It puts all nodes with an in-degree of zero into a queue, then takes nodes out of the queue in a loop, adds them to the execution sequence, and decrements the in-degree of all downstream adjacent nodes by one. If the in-degree is zero after decrementing, the downstream node is enqueued to generate an ordered recomputation task list. The cascaded computation execution submodule, based on the ordered recomputation task list, traverses each task identifier in the list in sequence, retrieves the computation operation type and all input nodes corresponding to the task from the full-cycle data spectrum, obtains the latest value of the input node, then calls the corresponding computation function to perform the operation, and stores the operation result with the task identifier to obtain a newly generated data node set.

8. The data correction and fusion system for big data according to claim 7, characterized in that, The dependency backtracking and recalculation correction module also includes: The data status update submodule, based on the newly generated data node set, traverses each new data node in the set, uses a unique identifier to find the corresponding old node in the full-cycle data hierarchy graph, and updates the old node's value, timestamp, and status mark with the corresponding content of the new node. After all update operations are completed, the updated nodes together constitute a consistent data update set.

9. The data acquisition and fusion method for a big data-oriented data correction and fusion system according to any one of claims 1-8, characterized in that, Includes the following steps: Based on the manually entered industrial entries to be corrected and the captured data, the system uses the equipment number and process stage information attached to the entered entries to retrieve related material batch number records, operation instruction records and fault code records from the historical database, and summarizes them into a temporary effective dictionary. It calculates the minimum number of single character insertions, deletions and replacements required to convert between the entered entries and each entry in the dictionary, and at the same time converts the entered entries and each entry in the dictionary into coded characters according to preset pronunciation rules. By comparing the consistency of the coded characters, a comprehensive confidence score is generated. Based on the preset threshold, automatic correction or manual confirmation prompts are determined to form a corrected data entry set. Based on the corrected data entry set and the automatically collected sensor data readings, a data node with a unique identifier, value and timestamp is created for each independent data item. When several data nodes jointly generate a new data item, the generation operation is recorded as an operation edge, and a connection relationship is established from the input data node to the output data node to construct a directed acyclic graph structure and form a full-cycle data spectrum diagram. Based on the full-cycle data spectrum, when a cleaning operation instruction for a certain data node is received, the unique identifier of the node is placed in the processing queue. Then, the node identifier is retrieved from the queue in a loop. All directed edges originating from the node are searched in the spectrum. The downstream node identifiers pointed to by the edges are added to the node identifier set to be updated. At the same time, the downstream node identifiers are also placed in the processing queue. This process is repeated until the processing queue is empty, and the complete node identifier set to be updated is obtained. Based on the set of node identifiers to be updated and the preset recalculation strategy, a reasonable calculation order is determined according to the dependency relationship between nodes in the phylogenetic graph. First, the source node that is directly corrected is recalculated, triggering the calculation task corresponding to the associated operation edge. After the calculation task is completed and a new data node is generated, all calculation tasks with the new data node as input are triggered step by step, and the process is repeated until all nodes in the set of node identifiers to be updated have been recalculated and updated, resulting in a consistent data update set.

Images