A method and system for processing multi-source heterogeneous data suitable for rule reconstruction and fusion

Abstract

This invention proposes a method and system for processing multi-source heterogeneous data suitable for rule reconstruction and fusion. It belongs to the interdisciplinary technical fields of industrial big data processing, information fusion, and intelligent decision-making. The method includes: constructing a protocol-independent access layer to compatiblely access multiple data sources and generate a multi-source heterogeneous raw data set; based on the multi-source heterogeneous raw data set, performing physical layer reconstruction using a user-programmable rule engine to generate physical layer reconstructed data; further performing semantic layer reconstruction on the physical layer reconstructed data to generate semantic layer aligned data; and then performing business layer reconstruction based on the semantic layer aligned data to generate business layer reconstructed data. By constructing a protocol-independent access layer, it can be compatible with multiple communication protocols and data formats, eliminating the need to develop dedicated adapters for different combinations, significantly reducing development and maintenance costs, and improving the convenience and flexibility of industrial data access.

Description

Technical Field

[0001] This invention proposes a method and system for processing multi-source heterogeneous data suitable for rule reconstruction and fusion, belonging to the cross-technical fields of industrial big data processing, information fusion and intelligent decision-making. Background Technology

[0002] In today's era of booming smart manufacturing, the data generated in the industrial production process is growing explosively, with a wide range of sources and diverse types, making multi-source heterogeneous data processing a key challenge.

[0003] Currently, the processing of multi-source heterogeneous data in the industrial sector is far from satisfactory. On the one hand, protocol fragmentation is a prominent issue, with multiple communication protocols and data formats often coexisting on the same production line. For example, PLCs use the Modbus TCP protocol, robot controllers use the ROSTopic protocol, MES systems are based on RESTful JSON format, and sensors rely on the CAN bus. Traditional solutions require developing dedicated adapters for each protocol combination, which not only results in long development cycles but also extremely high maintenance costs.

[0004] On the other hand, data fusion processing has many drawbacks. The fusion logic is mostly hard-coded, with rules fixed in the code, making it difficult for process engineers or operations personnel to adjust according to production line changes, resulting in poor flexibility. Furthermore, the lack of a unified semantic layer leads to inconsistent naming of the same entity across different systems, causing distorted fusion results. Moreover, the fusion model is static and simplistic, making it difficult to dynamically select appropriate fusion strategies based on different task contexts, thus failing to meet diverse business needs.

[0005] Most of the existing publicly available information only promotes general concepts such as multi-source data fusion platforms, without achieving deep integration such as programmable rule reconstruction, semantic layer alignment, and dynamic selection of fusion strategies, thus failing to provide truly effective data fusion solutions for intelligent manufacturing. Summary of the Invention

[0006] This invention provides a method and system for processing multi-source heterogeneous data suitable for rule reconstruction and fusion, in order to solve the problems mentioned in the background art above:

[0007] This invention proposes a method for processing multi-source heterogeneous data suitable for rule reconstruction and fusion, the method comprising:

[0008] S1. Construct a protocol-independent access layer to enable compatible access to multiple data sources and generate a diverse set of heterogeneous raw data.

[0009] S2. Based on the diverse and heterogeneous original data set, the physical layer is reconstructed using a user-programmable rule engine to generate physical layer reconstructed data; the semantic layer is further reconstructed on the physical layer reconstructed data to generate semantic layer aligned data; and then the business layer is reconstructed based on the semantic layer aligned data to generate business layer reconstructed data.

[0010] S3. Based on the data reconstructed from the business layer and combined with the task context information, dynamically select the fusion strategy; generate time-series aligned fused data and confidence-weighted fused data;

[0011] S4. Perform correlation analysis on time-series aligned fused data or confidence-weighted fused data to uncover potential relationships between data and generate correlation analysis result data; perform anomaly detection processing on the correlation analysis result data to identify abnormal data and generate anomaly detection result data.

[0012] S5. Based on the anomaly detection results data, perform risk assessment processing to generate risk assessment data; based on the risk assessment data and combined with preset early warning rules, perform risk early warning processing to generate early warning information based on the fusion of multi-source heterogeneous data.

[0013] This invention proposes a system for implementing the multi-source heterogeneous data processing method applicable to rule reconstruction and fusion as described above, the system comprising:

[0014] Collection generation module: Constructs a protocol-independent access layer to enable compatible access to multiple data sources and generate a collection of diverse and heterogeneous raw data.

[0015] Business restructuring module: Based on the diverse and heterogeneous original data set, it uses a user-programmable rule engine to perform physical layer restructuring and generate physical layer restructuring data; it further performs semantic layer restructuring on the physical layer restructuring data to generate semantic layer aligned data; and then performs business layer restructuring based on the semantic layer aligned data to generate business layer restructuring data.

[0016] Strategy selection module: Based on the reconstructed data from the business layer and combined with task context information, dynamically select the fusion strategy; generate time-series aligned fused data and confidence-weighted fused data;

[0017] The detection and processing module performs correlation analysis on time-aligned fused data or confidence-weighted fused data to uncover potential relationships between data and generate correlation analysis results data; it also performs anomaly detection processing on the correlation analysis results data to identify abnormal data and generate anomaly detection results data.

[0018] Early warning processing module: Based on the anomaly detection results data, perform risk assessment processing to generate risk assessment data; based on the risk assessment data and combined with preset early warning rules, perform risk early warning processing to generate early warning information fused from multi-source heterogeneous data.

[0019] The beneficial effects of this invention are as follows: By constructing a protocol-independent access layer, it is compatible with multiple communication protocols and data formats, eliminating the need to develop dedicated adapters for different combinations, significantly reducing development and maintenance costs, and improving the convenience and flexibility of industrial data access. Utilizing a user-programmable rule engine to achieve a three-layer reconstruction of physical, semantic, and business layers allows process engineers and maintenance personnel to independently adjust rules according to production line changes, enhancing the adaptability of data processing to production line variations. Dynamically selecting fusion strategies based on task context satisfies both the high-precision time-series alignment requirements of quality traceability and the latency tolerance of energy consumption analysis, improving the accuracy and practicality of data fusion. A unified semantic layer avoids the distortion of fusion results caused by naming differences between different systems, reducing data errors. Simultaneously, this innovative method avoids the limitations of traditional solutions due to hard-coded rules and static, singular models. Attached Figure Description

[0020] Figure 1 This is a diagram illustrating the steps of the method described in this invention;

[0021] Figure 2 This is a system module diagram of the present invention. Detailed Implementation

[0022] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0023] One embodiment of the present invention, such as Figure 1 As shown, a method for processing multi-source heterogeneous data suitable for rule reconstruction and fusion is provided, the method comprising:

[0024] S1. Construct a protocol-independent access layer to enable compatible access to diverse data sources such as robots, task systems, and devices, generating diverse and heterogeneous raw data sets. This access layer supports multiple communication protocols and data formats such as PLC (ModbusTCP), robot controller (ROSTopic), MES system (RESTfulJSON), and sensors (CAN bus), eliminating the need to develop dedicated adapters for each combination.

[0025] S2. Based on the diverse and heterogeneous original data set, a user-programmable rule engine is used to perform physical layer reconstruction, converting the original byte stream into data with a unified physical unit to generate physical layer reconstruction data; the user-programmable rule engine is then used to further perform semantic layer reconstruction on the physical layer reconstruction data, unifying the naming of the same entity (e.g., workstation A) in different systems to generate semantic layer alignment data; based on the semantic layer alignment data, a user-programmable rule engine is used to perform business layer reconstruction, mapping the data to specific business scenarios to generate business layer reconstruction data.

[0026] S3. Based on the business layer reconstructed data and combined with the task context information, dynamically select the fusion strategy; when the task context is quality traceability, select the time-series alignment fusion strategy to perform high-precision time-series alignment processing on the business layer reconstructed data and generate time-series aligned fused data; when the task context is energy consumption analysis, select the confidence-weighted fusion strategy to perform confidence-weighted processing on the business layer reconstructed data according to the reliability of the data source and generate confidence-weighted fused data.

[0027] S4. Perform correlation analysis on time-series aligned fused data or confidence-weighted fused data to uncover potential relationships between data and generate correlation analysis result data; perform anomaly detection processing on the correlation analysis result data to identify abnormal data and generate anomaly detection result data.

[0028] S5. Based on the anomaly detection results, perform risk assessment processing to generate risk assessment data; based on the risk assessment data and combined with preset early warning rules, perform risk early warning processing to generate early warning information based on multi-source heterogeneous data fusion, providing a new generation of data fusion infrastructure support for intelligent manufacturing.

[0029] The working principle and effects of the above technical solution are as follows: By combining protocol-independent access with layered reconstruction, it can be compatible with multiple data sources and communication formats, avoiding redundant development for different protocol combinations, reducing data access time and cost, and improving the efficiency of cross-system data integration. Layered reconstruction unifies data units and entity naming, avoiding fusion deviations caused by semantic ambiguity, and enhancing data consistency and interoperability. Dynamically selecting fusion strategies based on task context improves the targeting and accuracy of data fusion, adapting to different business needs. Correlation analysis and anomaly detection can promptly uncover potential data relationships and identify anomalies, preventing abnormal data from interfering with production processes; risk warning enhances the risk management capabilities of intelligent manufacturing, preventing losses caused by risk diffusion, ensuring both efficient and stable data processing and improved reliability of production decisions.

[0030] In one embodiment of the present invention, S1 includes:

[0031] S11. Design a distributed protocol parsing kernel, build a protocol-independent access layer framework, and generate a multi-protocol compatible access base.

[0032] S12. Based on a multi-protocol compatible access base, configure an adaptive parsing module for communication protocols, including PLC (ModbusTCP), robot controller (ROSTopic), MES system (RESTfulJSON), and sensors (CAN bus), and generate an adaptive access engine for the protocol.

[0033] S13. Through the protocol adaptive access engine, perform seamless and compatible access to multiple data sources, including robots, task systems, and devices. Collect raw data byte streams and generate diverse heterogeneous raw access data.

[0034] S14. Perform integrity verification and format compliance verification on the diverse and heterogeneous raw access data, filter out invalid and redundant data, and generate a diverse and heterogeneous raw data set; moreover, this access layer does not need to develop a dedicated adapter for each protocol and data format combination, realizing plug-and-play access across system data sources.

[0035] The working principle and effects of the above technical solution are as follows: Through the cooperation of the distributed protocol parsing kernel and the adaptive parsing module, it can be compatible with multiple communication protocols and diverse data sources, eliminating the need to develop dedicated adapters for different protocol format combinations, avoiding resource waste caused by redundant development, and significantly reducing the cost and cycle of data access. Simultaneously, the seamless compatibility access mode allows for plug-and-play use of cross-system data sources, improving access efficiency and reducing cumbersome configuration during the access process. Data integrity verification and compliance verification can filter out invalid and redundant data, preventing such data from interfering with subsequent processing, improving the quality of the original data set, and laying a reliable foundation for subsequent data reconstruction and fusion.

[0036] In one embodiment of the present invention, S2 includes:

[0037] S21. Based on the multi-dimensional heterogeneous raw data set, call the byte stream parsing component of the user-programmable rule engine to extract the mapping relationship between data field identifiers and physical quantities, and generate a physical layer parsing rule set;

[0038] S22. Based on the physical layer parsing rule set, the original byte stream is converted into original data with physical quantity identifiers, and then the unit unification module of the rule engine is used for measurement standardization to generate physical layer reconstructed data.

[0039] S23. Based on the physical layer, reconstruct the data, call the semantic mapping module of the user-programmable rule engine, load the entity naming standardization dictionary (e.g., the unified naming rule for workstation A), match the entity naming differences of different systems, and generate semantic alignment rules.

[0040] S24. Based on the semantic alignment rules, the entity names in the physical layer reconstructed data are uniformly replaced and associated with each other to eliminate semantic ambiguity and generate semantic layer aligned data.

[0041] S25. Based on semantic layer aligned data, combined with the intelligent manufacturing business scenario library (such as production scheduling, quality control, and equipment operation and maintenance), the association rules between data and business scenarios are configured through the business mapping component of the user programmable rule engine to generate a business layer mapping template.

[0042] S26. Based on the business layer mapping template, the semantic layer aligned data is mapped to the indicator system of the specific business scenario to generate business layer reconstructed data.

[0043] The working principle and effects of the above technical solution are as follows: Through a layered reconstruction process driven by a user-programmable rule engine, the mapping relationship between data fields and physical quantities can be accurately extracted, transforming the raw byte stream into data with a unified unit of measurement. This avoids data deviations caused by different units and significantly improves the accuracy of physical layer data. Simultaneously, semantic mapping eliminates differences in entity naming across different systems, resolves semantic ambiguity, enhances data interoperability, and removes obstacles for subsequent fusion processing. Based on this, data is mapped to specific business scenarios, ensuring that the data closely aligns with actual application needs and enhancing its practical value. The entire process guarantees the rigor of data standardization while enabling the reconstructed data to directly serve business decisions, reducing ineffective processing caused by data disconnect from business needs, and laying a solid foundation for efficient data fusion in the future.

[0044] In one embodiment of the present invention, step S22 includes:

[0045] Based on the physical layer parsing rule set, the field matching component of the rule engine is called to perform field identification and data fragment extraction on the raw byte stream in the multi-dimensional heterogeneous raw data set, and generate byte stream data with field identification.

[0046] Based on byte stream data with field identifiers, and according to the physical quantity mapping relationship in the physical layer parsing rule set, the data of each field is numerically converted and bound to physical quantity attributes to generate raw data with physical quantity identifiers.

[0047] Extract unit information from raw data with physical quantity labels, compare it with the standard measurement system built into the rule engine, identify non-standard unit types, and generate a unit conversion comparison table;

[0048] Based on the unit conversion table, the unit unification module of the rule engine performs numerical conversion to unify data with different units of measurement into standard units, ensuring data measurement consistency and generating physical layer reconstructed data.

[0049] The working principle and effects of the above technical solution are as follows: By connecting field matching and hierarchical conversion, valid fields in the original byte stream can be accurately located, avoiding data fragment extraction errors and significantly improving the accuracy of data with field identifiers. Simultaneously, binding field data with physical quantity attributes gives the originally fragmented byte stream a clear physical meaning, eliminating the problem of ambiguous data attributes. Based on this, non-standard units are identified through comparison with a standard measurement system, and then unified conversion is completed, effectively avoiding data measurement confusion caused by different units and ensuring the consistency of physical layer data. The entire process is progressive, ensuring both the rigor of data conversion and the generation of standardized and unified physical layer reconstructed data, reducing obstacles in subsequent semantic reconstruction and business mapping, and laying a solid foundation for the overall efficiency of data processing.

[0050] In one embodiment of the present invention, S3 includes:

[0051] S31. Based on the business layer, reconstruct the data, extract key information of the task context (such as task type, business objectives, and data requirements), and generate a task context feature vector.

[0052] S32. Input the task context feature vector into the fusion strategy decision model, match it with the preset scenario and strategy mapping library, and generate the target fusion strategy identifier.

[0053] S33. If the target fusion strategy is identified as time-aligned fusion (corresponding to task context quality traceability), a high-precision time-series calibration algorithm is invoked. Based on the data acquisition timestamp and the device synchronization clock, the business layer reconstructed data is processed for millisecond-level time-series alignment to generate time-aligned fusion data.

[0054] S34. If the target fusion strategy is identified as confidence-weighted fusion (corresponding to task context energy consumption analysis), calculate the confidence weight of each data source (based on device accuracy, historical data accuracy, and communication stability) through the data source reliability assessment model, and generate a data source confidence weight table.

[0055] S35. Based on the confidence weight table of the data source, perform weighted fusion calculation on the reconstructed data of the business layer to generate confidence weighted fusion data.

[0056] The working principle and effects of the above technical solution are as follows: Through the linkage of task context extraction and dynamic matching of fusion strategies, the core requirements of different business tasks can be accurately captured, avoiding the problem of fusion strategies being out of sync with task objectives and significantly improving the targeting of strategy selection. For quality traceability tasks, millisecond-level time-series alignment effectively eliminates time deviations in data collection, avoids fusion errors caused by time-series chaos, and improves the timeliness and accuracy of fused data. For energy consumption analysis tasks, confidence-weighted fusion, combined with weighting based on data source reliability, filters out interference from low-quality data and enhances the credibility of fusion results. The entire process can flexibly adapt to the fusion needs of different business scenarios while ensuring high-quality fused data, providing reliable support for subsequent correlation analysis and anomaly detection, reducing invalid processing caused by fusion data deviations, and improving overall data processing efficiency.

[0057] In one embodiment of the present invention, S32 includes:

[0058] The task context feature vector is standardized to eliminate the difference in the units of different features and generate a standardized task context feature vector.

[0059] Based on standardized task context feature vectors, a feature retrieval index is constructed, and similarity matching retrieval is performed on the preset scenario and strategy mapping library to filter out scenario-strategy association records with high matching degree and generate a candidate fusion strategy set.

[0060] The candidate fusion strategy set is input into the fusion strategy decision model. The candidate strategies are prioritized and ranked by the built-in weighted voting mechanism of the model to determine the optimal fusion strategy and generate the strategy priority ranking result.

[0061] Based on the strategy priority ranking results, extract the unique identifier information corresponding to the optimal strategy and generate the target fusion strategy identifier.

[0062] The working principle and effects of the above technical solution are as follows: By combining feature vector standardization with hierarchical matching and ranking, the interference of different dimensions of features can be eliminated, avoiding matching deviations caused by differences in dimensions and significantly improving the accuracy of feature matching. Simultaneously, constructing a feature retrieval index to filter highly matched records can quickly identify candidate fusion strategies, reducing the time spent filtering invalid strategies and improving strategy matching efficiency. Utilizing a weighted voting mechanism to rank and select the best candidate strategy accurately determines the optimal strategy that meets the task requirements, avoiding the problem of strategy selection errors. The entire process ensures the rigor of strategy matching while improving the efficiency of generating target fusion strategy identifiers, providing precise guidance for subsequent targeted fusion processing and reducing fusion quality issues caused by strategy deviations.

[0063] In one embodiment of the present invention, step S4 includes:

[0064] S41. Based on time-aligned fused data or confidence-weighted fused data, an adaptive data denoising algorithm (such as wavelet threshold denoising) is used to filter random interference noise, and then the data scale is unified through feature dimension normalization to generate preprocessed fused data.

[0065] S42. Call the graph neural network association analysis model to construct an entity relationship graph of the preprocessed fused data, mine the implicit relationships between data (such as the mapping relationship between equipment operating parameters and product quality), and generate an association relationship matrix;

[0066] S43. Based on the association matrix, extract the core association rules using frequent itemset mining algorithms (such as FP-Growth) to generate association analysis results data;

[0067] S44. Input the correlation analysis results into the improved isolated forest anomaly detection model, set a dynamic anomaly threshold (based on the fluctuation range of the business scenario), identify data points that deviate from the normal distribution, and generate an anomaly data identifier set;

[0068] S45. Combining the abnormal data identifier set with the original data context, output abnormal data information, including location, type, and severity, and generate abnormal detection result data.

[0069] The working principle and effects of the above technical solution are as follows: Through a progressive process of preprocessing, association analysis, and anomaly detection, the adaptive denoising algorithm effectively filters random interference noise, normalizes the data scale to unify it, avoids interference from noise and dimensional differences in subsequent analysis, and significantly improves the purity of preprocessed data. Combining graph neural networks to construct entity relationship graphs with frequent itemset mining to extract core association rules can accurately uncover implicit associations between data, avoid missing association information, and enhance the depth and reliability of association analysis results. The improved isolated forest, combined with dynamic anomaly thresholds, can accurately identify abnormal data points, avoiding misjudgments or omissions caused by fixed thresholds. Combined with the original context, it outputs detailed anomaly information, making anomaly localization more accurate, providing a reliable basis for subsequent risk assessment, and reducing ineffective control caused by anomaly identification bias.

[0070] In one embodiment of the present invention, S43 includes:

[0071] S431. Sparsify the association matrix, filter out low-confidence association edges (set confidence threshold), retain highly correlated data associations, and generate a simplified association matrix;

[0072] S432. Based on the simplified association matrix, extract the itemset combinations of data entities, remove duplicate and invalid itemsets, and generate an initial frequent itemset candidate set;

[0073] S433. Input the initial frequent itemset candidate set into the FP-Growth algorithm model, construct the FP tree and traverse and mine it, count the support of each candidate itemset, and generate itemset support statistics.

[0074] S434. Based on the preset support threshold, filter the support statistics of the itemsets, retain the high-frequency itemsets, and then extract the causal relationships and constraints between the data to generate the core association rule set.

[0075] S435. Supplement the core association rule set with attribute information, including rule confidence and applicable scenarios, organize them into structured data, and generate association analysis result data.

[0076] The working principle and effects of the above technical solution are as follows: Through progressive processing of association matrix simplification, itemset screening, and algorithmic mining, low-confidence association edges can be filtered out, avoiding interference from invalid associations in subsequent analysis and significantly improving the purity of association data. Simultaneously, removing duplicate and invalid itemsets reduces the computational burden of frequent itemset mining, improving mining efficiency. Constructing an FP-tree traversal mining accurately calculates itemset support, avoiding support statistical bias, and filters high-frequency itemsets to extract core association rules, ensuring the effectiveness and relevance of the rules. Supplementing attribute information and organizing it into structured data makes the association analysis results easier to apply. The entire process improves the efficiency and accuracy of association rule mining and ensures good usability of the results, providing accurate association basis for subsequent anomaly detection and reducing detection errors caused by association rule bias.

[0077] In one embodiment of the present invention, S433 includes:

[0078] The initial frequent itemset candidate set is preprocessed by sorting the itemsets in descending order of frequency to eliminate the interference of unordered itemsets on the model input and generate a standardized frequent itemset candidate set.

[0079] The standardized frequent itemset candidate set is input into the FP-Growth algorithm model, the root node of the FP tree is initialized, and each standardized itemset is inserted in sequence to construct the FP tree structure data. The FP tree structure data includes the item header table and the node association relationship.

[0080] Based on the item head table of the FP-tree, traverse the conditional pattern base corresponding to each frequent item, extract the prefix path information associated with the itemset, and generate a set of conditional pattern bases.

[0081] For each conditional pattern base, construct the corresponding conditional FP subtree, mine the locally frequent itemsets in the subtree, and generate a set of locally frequent itemsets;

[0082] Summarize all locally frequent itemsets, count the frequency of each candidate item in the original data, calculate the support value, and generate itemset support statistics.

[0083] The working principle and effects of the above technical solution are as follows: By combining itemset sorting preprocessing with FP-tree hierarchical mining, the interference of unordered itemsets on the model input can be eliminated, avoiding mining bias caused by input chaos and significantly improving the standardization of frequent itemset candidate sets. Simultaneously, constructing an FP-tree structure containing item header tables and node associations makes data relationships clearer, reducing the computational burden of subsequent traversal mining and improving mining efficiency. Traversing the conditional pattern base and constructing conditional FP subtrees can accurately mine local frequent itemsets, avoiding the omission of local association information and ensuring the comprehensiveness of frequent itemset mining. Summarizing local results and statistically analyzing support ensures the accuracy of numerical calculations and avoids support statistical bias. The entire process improves the efficiency and standardization of itemset mining while ensuring the accuracy and reliability of support statistical results, providing a solid basis for the subsequent selection of core association rules and reducing rule extraction errors caused by incomplete mining or statistical bias.

[0084] In one embodiment of the present invention, step S5 includes:

[0085] S51. Based on the anomaly detection results data, construct a multi-dimensional risk quantification indicator system (including impact scope, probability of occurrence, degree of loss, and recovery cost), use the analytic hierarchy process to determine the weight of each indicator, and generate a risk quantification score;

[0086] S52. Based on the risk quantification score, and in accordance with the preset risk level classification standards (low, medium, high, extremely high), determine the risk level and generate risk assessment data;

[0087] S53. Match the risk assessment data with the preset multi-level early warning rule base (matching early warning methods, push targets, and processing time limits according to risk level) to generate early warning execution instructions;

[0088] S54. Based on the early warning execution command, generate multiple forms of early warning information (text prompts, visual charts, sound and light alarms, mobile push notifications) and synchronize them to the intelligent manufacturing management platform, relevant business systems and the responsible person's terminal;

[0089] S55. Record the reception status and processing progress of early warning information to form a closed-loop management data for risk early warning, providing real-time, accurate and traceable next-generation data fusion infrastructure support for intelligent manufacturing.

[0090] The working principle and effects of the above technical solution are as follows: Through the layered advancement of multi-dimensional risk quantification and closed-loop early warning management, it comprehensively covers the key dimensions of risk assessment. Combining the analytic hierarchy process (AHP) to determine indicator weights avoids the bias caused by single indicators, significantly improving the accuracy of risk quantification scoring. By comparing risk levels with standards, the degree of risk becomes clearer, avoiding inappropriate control due to ambiguous levels. Matching multi-level early warning rules generates precise execution instructions, ensuring that early warning methods, targets, and time limits are adapted to risk levels, avoiding problems of chaotic or insufficiently targeted early warnings. Multiple forms of early warnings are synchronized to multiple terminals, allowing relevant personnel to receive information in a timely manner, preventing risk spread due to delayed warnings. Recording the reception status and processing progress forms a closed loop, enhancing the traceability of risk control and avoiding omissions. The entire process improves the timeliness and accuracy of risk early warnings while ensuring the standardization of control, providing reliable data support for intelligent manufacturing and reducing production losses caused by risks.

[0091] One embodiment of the present invention, such as Figure 2 The system described above is used to implement a multi-source heterogeneous data processing method suitable for rule reconstruction and fusion. The system includes:

[0092] Collection generation module: Constructs a protocol-independent access layer to enable compatible access to multiple data sources and generate a collection of diverse and heterogeneous raw data.

[0093] Business restructuring module: Based on the diverse and heterogeneous original data set, it uses a user-programmable rule engine to perform physical layer restructuring and generate physical layer restructuring data; it further performs semantic layer restructuring on the physical layer restructuring data to generate semantic layer aligned data; and then performs business layer restructuring based on the semantic layer aligned data to generate business layer restructuring data.

[0094] Strategy selection module: Based on the reconstructed data from the business layer and combined with task context information, dynamically select the fusion strategy; generate time-series aligned fused data and confidence-weighted fused data;

[0095] The detection and processing module performs correlation analysis on time-aligned fused data or confidence-weighted fused data to uncover potential relationships between data and generate correlation analysis results data; it also performs anomaly detection processing on the correlation analysis results data to identify abnormal data and generate anomaly detection results data.

[0096] Early warning processing module: Based on the anomaly detection results data, perform risk assessment processing to generate risk assessment data; based on the risk assessment data and combined with preset early warning rules, perform risk early warning processing to generate early warning information fused from multi-source heterogeneous data.

[0097] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for processing multi-source heterogeneous data suitable for rule reconstruction and fusion, characterized in that, The method includes: S1. Construct a protocol-independent access layer to enable compatible access to multiple data sources and generate a diverse set of heterogeneous raw data. S2. Based on the diverse and heterogeneous original data set, the physical layer is reconstructed using a user-programmable rule engine to generate physical layer reconstructed data; the semantic layer is further reconstructed on the physical layer reconstructed data to generate semantic layer aligned data; and then the business layer is reconstructed based on the semantic layer aligned data to generate business layer reconstructed data. S3. Based on the data reconstructed from the business layer and combined with the task context information, dynamically select the fusion strategy; generate time-series aligned fused data and confidence-weighted fused data; S4. Perform correlation analysis on time-series aligned fused data or confidence-weighted fused data to uncover potential relationships between data and generate correlation analysis result data; perform anomaly detection processing on the correlation analysis result data to identify abnormal data and generate anomaly detection result data. S5. Based on the anomaly detection results, perform risk assessment processing to generate risk assessment data; based on the risk assessment data and combined with preset early warning rules, perform risk early warning processing to generate early warning information based on the fusion of multi-source heterogeneous data. S1 includes: S11. Design a distributed protocol parsing kernel, build a protocol-independent access layer framework, and generate a multi-protocol compatible access base. S12. Based on the multi-protocol compatible access base, configure the adaptive parsing module of the communication protocol to generate the protocol adaptive access engine; S13. Through the protocol adaptive access engine, perform seamless and compatible access to multiple data sources, collect raw data byte streams, and generate diverse heterogeneous raw access data. S14. Perform integrity verification and format compliance verification on the diverse and heterogeneous raw access data, filter out invalid and redundant data, and generate a diverse and heterogeneous raw data set.

2. The method for processing multi-source heterogeneous data suitable for rule reconstruction and fusion according to claim 1, characterized in that, S2 includes: S21. Based on the multi-dimensional heterogeneous raw data set, call the byte stream parsing component of the user-programmable rule engine to extract the mapping relationship between data field identifiers and physical quantities, and generate a physical layer parsing rule set; S22. Based on the physical layer parsing rule set, the original byte stream is converted into original data with physical quantity identifiers, and then the unit unification module of the rule engine is used for measurement standardization to generate physical layer reconstructed data. S23. Based on the physical layer, reconstruct the data, call the semantic mapping module of the user-programmable rule engine, load the entity naming standardization dictionary, match the entity naming differences of different systems, and generate semantic alignment rules. S24. Based on the semantic alignment rules, uniformly replace and associate the entity names in the physical layer reconstructed data to generate semantic layer aligned data. S25. Based on semantic layer aligned data and combined with the intelligent manufacturing business scenario library, the association rules between data and business scenarios are configured through the business mapping component of the user programmable rule engine to generate a business layer mapping template. S26. Based on the business layer mapping template, the semantic layer aligned data is mapped to the indicator system of the specific business scenario to generate business layer reconstructed data.

3. The method for processing multi-source heterogeneous data suitable for rule reconstruction and fusion according to claim 2, characterized in that, S22 includes: Based on the physical layer parsing rule set, the field matching component of the rule engine is called to perform field identification and data fragment extraction on the raw byte stream in the multi-dimensional heterogeneous raw data set, and generate byte stream data with field identification. Based on byte stream data with field identifiers, and according to the physical quantity mapping relationship in the physical layer parsing rule set, the data of each field is numerically converted and bound to physical quantity attributes to generate raw data with physical quantity identifiers. Extract unit information from raw data with physical quantity labels, compare it with the standard measurement system built into the rule engine, identify non-standard unit types, and generate a unit conversion comparison table; Based on the unit conversion table, the unit unification module of the rule engine performs numerical conversion to unify data with different units of measurement into standard units, generating physical layer reconstruction data.

4. The method for processing multi-source heterogeneous data suitable for rule reconstruction and fusion according to claim 1, characterized in that, The S3 includes: S31. Based on the business layer, reconstruct the data, extract key information of the task context, and generate a task context feature vector; S32. Input the task context feature vector into the fusion strategy decision model, match it with the preset scenario and strategy mapping library, and generate the target fusion strategy identifier. S33. If the target fusion strategy is identified as time-aligned fusion, a high-precision time-series calibration algorithm is invoked. Based on the data acquisition timestamp and the device synchronization clock, the business layer reconstructed data is processed for millisecond-level time-series alignment to generate time-aligned fusion data. S34. If the target fusion strategy is identified as confidence-weighted fusion, calculate the confidence weight of each data source through the data source reliability assessment model and generate a data source confidence weight table. S35. Based on the confidence weight table of the data source, perform weighted fusion calculation on the reconstructed data of the business layer to generate confidence weighted fusion data.

5. The method for processing multi-source heterogeneous data suitable for rule reconstruction and fusion according to claim 1, characterized in that, The S4 includes: S41. Based on time-aligned fused data or confidence-weighted fused data, an adaptive data denoising algorithm is used to filter random interference noise, and then the data scale is unified through feature dimension normalization to generate preprocessed fused data. S42. Call the graph neural network association analysis model to construct an entity relationship graph of the preprocessed fused data, mine the implicit relationships between the data, and generate an association relationship matrix; S43. Based on the association matrix, extract the core association rules through the frequent itemset mining algorithm to generate association analysis result data; S44. Input the correlation analysis results into the improved isolated forest anomaly detection model, set a dynamic anomaly threshold, identify data points that deviate from the normal distribution, and generate an anomaly data identifier set. S45. Combining the abnormal data identifier set with the original data context, output the abnormal data information and generate abnormal detection result data.

6. The method for processing multi-source heterogeneous data suitable for rule reconstruction and fusion according to claim 5, characterized in that, S43 includes: S431. Sparsify the association matrix, filter out low-confidence association edges, retain high-correlation data associations, and generate a simplified association matrix. S432. Based on the simplified association matrix, extract the itemset combinations of data entities, remove duplicate and invalid itemsets, and generate an initial frequent itemset candidate set; S433. Input the initial frequent itemset candidate set into the FP-Growth algorithm model, construct the FP tree and traverse and mine it, count the support of each candidate set, and generate itemset support statistics. S434. Based on the preset support threshold, filter the support statistics of the itemsets, retain the high-frequency itemsets, and then extract the causal relationships and constraints between the data to generate the core association rule set. S435. Supplement the core association rule set with attribute information, organize it into structured data, and generate association analysis result data.

7. The method for processing multi-source heterogeneous data suitable for rule reconstruction and fusion according to claim 6, characterized in that, S433 includes: The initial frequent itemset candidate set is preprocessed by sorting the itemsets in descending order of frequency to eliminate the interference of unordered itemsets on the model input and generate a standardized frequent itemset candidate set. Input the standardized frequent itemset candidate set into the FP-Growth algorithm model, initialize the root node of the FP tree, and insert each standardized itemset in sequence to construct the FP tree structure data. Based on the item head table of the FP-tree, traverse the conditional pattern base corresponding to each frequent item, extract the prefix path information associated with the itemset, and generate a set of conditional pattern bases. For each conditional pattern base, construct the corresponding conditional FP subtree, mine the locally frequent itemsets in the subtree, and generate a set of locally frequent itemsets; Summarize all locally frequent itemsets, count the frequency of each candidate item in the original data, calculate the support value, and generate itemset support statistics.

8. The method for processing multi-source heterogeneous data suitable for rule reconstruction and fusion according to claim 1, characterized in that, The S5 includes: S51. Based on the anomaly detection results data, construct a multi-dimensional risk quantification indicator system, use the analytic hierarchy process to determine the weight of each indicator, and generate a risk quantification score. S52. Based on the risk quantification score, and in accordance with the preset risk level classification standards, determine the risk level and generate risk assessment data; S53. Match the risk assessment data with the preset multi-level early warning rule base to generate early warning execution instructions; S54. Based on the early warning execution command, generate multiple forms of early warning information and synchronize them to the intelligent manufacturing management platform, relevant business systems and the responsible person's terminal; S55. Record the reception status and processing progress of early warning information to form closed-loop management data for risk early warning.

9. A system for implementing the multi-source heterogeneous data processing method for rule reconstruction and fusion as described in claim 1, characterized in that, The system includes: Collection generation module: Constructs a protocol-independent access layer to enable compatible access to multiple data sources and generate a collection of diverse and heterogeneous raw data. Business restructuring module: Based on the diverse and heterogeneous original data set, it uses a user-programmable rule engine to perform physical layer restructuring and generate physical layer restructuring data; it further performs semantic layer restructuring on the physical layer restructuring data to generate semantic layer aligned data; and then performs business layer restructuring based on the semantic layer aligned data to generate business layer restructuring data. Strategy selection module: Based on the reconstructed data from the business layer and combined with task context information, dynamically select the fusion strategy; generate time-series aligned fused data and confidence-weighted fused data; The detection and processing module performs correlation analysis on time-aligned fused data or confidence-weighted fused data to uncover potential relationships between data and generate correlation analysis results data; it also performs anomaly detection processing on the correlation analysis results data to identify abnormal data and generate anomaly detection results data. Early warning processing module: Based on the anomaly detection results data, perform risk assessment processing to generate risk assessment data; based on the risk assessment data and combined with preset early warning rules, perform risk early warning processing to generate early warning information fused from multi-source heterogeneous data.

Images