Multi-source heterogeneous data intelligent fusion analysis method

Abstract

The invention provides a multi-source heterogeneous data intelligent fusion analysis method, which relates to the technical field of industrial multi-source heterogeneous data processing, and comprises five steps of data access and processing, dynamic target analysis and fusion demand generation, data fusion strategy generation, fusion and analysis execution and fusion strategy optimization. Through a target-driven dynamic fusion mode, a data fusion process is combined with a business target, a fusion strategy is automatically adjusted according to real-time changing business requirements, and a fusion rule can be generated and adjusted in real time in the fusion process according to different analysis targets, so that a more accurate and more flexible data fusion mode is provided. Therefore, on the premise that the data quality is guaranteed, the fusion strategy can be automatically and flexibly adjusted according to the real-time data requirement and the service target, redundant calculation is reduced, the calculation efficiency is improved, the fusion process is continuously optimized through a real-time feedback mechanism, and high flexibility and expansibility are achieved.

Description

Technical Field

[0001] This invention relates to the field of industrial multi-source heterogeneous data processing technology, and in particular to an intelligent fusion analysis method for multi-source heterogeneous data. Background Technology

[0002] In modern industry, with rapid technological advancements and increased production automation, the types and sources of industrial data are becoming increasingly diverse. Multi-source heterogeneous data refers to data from multiple different sources with varying structures, formats, and semantics, widely used in manufacturing, energy, transportation, petrochemical, and other industries. This data typically includes, but is not limited to, sensor data, equipment operation data, management system data, and external data. As industrial automation increases, enterprises are gradually incorporating more equipment and production processes into digital and intelligent management. Therefore, how to efficiently process, integrate, and analyze this data from different sources and in different formats has become a crucial issue.

[0003] In past industrial data processing, traditional data fusion methods mostly relied on manually defined data mapping and transformation rules. This involved pre-defining clear data transformation and matching rules, and then integrating data from different sources into a unified format for analysis. While this approach is suitable for situations with relatively simple data types and limited sources, it falls short when dealing with large-scale and multi-source heterogeneous data, mainly due to the following shortcomings: (1) Traditional methods require a lot of data preprocessing and standardization work before integration. They are not adaptable to scenarios such as changes in data sources and the introduction of new data formats, and are difficult to cope with ever-changing industrial scenarios. (2) Traditional frameworks usually use historical data to build static data structures, which limits the system’s ability to process real-time data. Especially when facing data fusion requirements that need to be dynamically adjusted, complex manual intervention is often required. (3) Traditional data fusion usually adopts a process of fusion first and then analysis. All data needs to be fully fused before analysis. This fixed process leads to computational redundancy and performance bottlenecks when facing real-time data streams and large-scale data. Therefore, this invention proposes an intelligent fusion analysis method for multi-source heterogeneous data to solve the problems existing in the prior art. Summary of the Invention

[0004] To address the aforementioned problems, the present invention aims to propose an intelligent fusion analysis method for multi-source heterogeneous data. This method is flexible and intelligent, and can solve the problems existing in the prior art.

[0005] To achieve the objectives of this invention, the invention is implemented through the following technical solution: an intelligent fusion and analysis method for multi-source heterogeneous data, comprising the following steps: Step 1: Data Access and Processing It accesses various industrial data sources based on compatible industrial protocols, acquires equipment industrial data in real time, parses unstructured data, performs preprocessing to obtain preprocessed data, and then stores the preprocessed data using a hierarchical storage architecture. Step 2: Dynamic Target Analysis and Fusion Requirement Generation The system obtains the business objectives input by the user and parses them using natural language processing methods, transforming them into structured data requirements. Then, it maps the structured data requirements to the data within the hierarchical storage architecture to identify the required data sources. Finally, it uses a decision tree algorithm to generate fusion requirements based on the mapping results and business objectives. Step 3: Generate data fusion strategy Based on the fusion requirements in step two, and combining preprocessed data and historical strategy effects, an executable data fusion strategy is dynamically generated using a reinforcement learning model. Step 4: Perform fusion and analysis Based on the fusion requirements in step two, target data is extracted from the hierarchical storage architecture. Then, the data fusion strategy generated in step three is executed to generate fused data oriented towards business objectives. The fused data is then input into a random forest learning model adapted to the business objectives for analysis, and the structured analysis results and feature importance scores corresponding to the user objectives are output. Step 5: Optimization of the fusion strategy Obtain users' judgments on the effectiveness of the structured analysis results, and combine them with the feature importance scores obtained in step four to generate a quantitative value of the historical strategy effect. This value is then used as a negative feedback sample to input into step three to complete the optimization of the fusion strategy.

[0006] A further improvement is made in the following: In step one, the specific preprocessing method is as follows: data anonymization and data cleaning are performed on the data, and then metadata tags are added, including time tags, source tags and quality tags.

[0007] A further improvement is that, in step one, the hierarchical storage architecture includes a distributed file layer for storing raw data, a time-series data layer for storing tagged sensor data, and a graph data layer for storing device topology relationships.

[0008] The further improvement lies in the fact that, in step two, the structured data requirements include key business entities, analytical indicators, and analytical dimensions.

[0009] The further improvement lies in the fact that, in step two, the fusion requirements include data range, key features, and time window.

[0010] The further improvement lies in the fact that, in step three, the data fusion strategy includes data association logic, fusion operation sequence, and resource scheduling scheme.

[0011] A further improvement is that, in step four, the random forest learning model is a pre-trained machine learning model.

[0012] A further improvement is made in step five, where the validity of the structured analysis results is categorized as valid or invalid.

[0013] The beneficial effects of this invention are as follows: This invention, through a goal-driven dynamic fusion approach, combines the data fusion process with business objectives, automatically adjusting the fusion strategy according to real-time changing business needs. Unlike traditional static data structure construction and predefined fusion rules, the dynamic fusion process can generate and adjust fusion rules in real time according to different analytical objectives, thus providing a more accurate and flexible data fusion method. Therefore, this invention can automatically and flexibly adjust the fusion strategy according to real-time data requirements and business objectives while ensuring data quality, reducing redundant calculations, improving computational efficiency, and continuously optimizing the fusion process through a real-time feedback mechanism, exhibiting high flexibility and scalability. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the steps of the present invention. Detailed Implementation

[0015] To enhance understanding of the present invention, the present invention will be further described in detail below with reference to embodiments. These embodiments are only used to explain the present invention and do not constitute a limitation on the scope of protection of the present invention.

[0016] according to Figure 1 As shown in the figure, this embodiment proposes an intelligent fusion and analysis method for multi-source heterogeneous data, including the following steps: Step 1: Data Access and Processing This system accesses various industrial data sources based on compatible industrial protocols, specifically OPC UA / Modbus / MQTT protocols, to access real-time sensor stream data from equipment. It acquires industrial data and parses unstructured data, such as maintenance logs where NLP is used to extract key events like bearing noise; equipment drawings where CV is used to identify topology tags, such as motor M1-coupling end; and work order systems where structured fields are extracted, such as fault codes and handlers. Preprocessing is then performed: data anonymization (ensuring sensitive information such as equipment serial numbers, operator IDs, and internal company data is encrypted or de-identified to comply with data privacy and security standards) and data cleaning (removing constant value signals, such as sensor data continuously at 0, and then using linear interpolation to fill missing values ​​due to communication interruptions). Metadata tags are then added, including time tags (indicating the data's timestamp to ensure correct alignment of time-series data), source tags (indicating the specific equipment or sensor from which the data originates, ensuring data traceability), and quality tags (adding quality indicators to help filter high-quality data and avoid interference from low-quality data during subsequent analysis), resulting in preprocessed data. Furthermore, the source tags include three types: device, location, and parameters, supporting accurate association, while the quality tags include credibility and missing rate, providing a basis for conflict resolution.

[0017] A hierarchical storage architecture is then employed to store the preprocessed data. This architecture includes a distributed file layer for storing raw data (supporting process reverse lookup and accident tracing), a time-series data layer for storing tagged sensor data (meeting real-time monitoring needs), and a graph data layer for storing device topology relationships (enabling fault propagation and deduction). The distributed file layer uses the HDFS distributed file system; the time-series data layer uses the InfluxDB time-series database for storage, suitable for time-series analysis; and the graph data layer uses the Neo4j graph database to store device topology relationships and other data for inter-device relationship analysis.

[0018] After step one, the original byte stream data is completely preserved through the distributed file layer, providing sufficient data support for subsequent backtracking verification and in-depth analysis. The use of lightweight metadata reduces redundant data storage while providing sufficient contextual information (such as time, source, and quality tags), greatly improving data manageability and traceability, and reducing storage requirements. Furthermore, the production line / equipment relationship is explicitly managed through device topology, supporting subsequent correlation analysis.

[0019] Step 2: Dynamic Target Analysis and Fusion Requirement Generation The system acquires user-inputted business objectives (e.g., "equipment failure prediction," "production efficiency optimization," "quality monitoring," etc.) and parses them using natural language processing methods, transforming them into structured data requirements. These requirements include key business entities (e.g., equipment, sensors, production lines), analytical metrics (e.g., equipment failure rate, production efficiency, quality pass rate), analytical dimensions (e.g., daily, hourly in the time dimension), and spatial dimensions (e.g., production line 1, production line 2). The structured data requirements are then mapped to data within a hierarchical storage architecture (using metadata tags added in step one for mapping and matching). This identifies the required data sources. Specifically, metadata tags from the data management platform are used to match data sources related to the business objectives. Based on data tags (e.g., "sensor data," "equipment historical status"), the specific data sources to be accessed are identified, and the corresponding storage layer is located. A decision tree algorithm is then used to generate fusion requirements based on the mapping results and business objectives, including: Data scope: Clearly identify the specific data sources that need to be fused, such as sensor data from the time-series data layer, device topology data from the graph data layer, etc. Key features: Identify the core data fields, attributes, or relationships that need to be extracted or used for fusion analysis. Examples include temperature, pressure, equipment health indices, and inter-equipment relationships. Time window: Defines the time range for data analysis, such as "data from the past month", "data from the past hour", or "real-time data".

[0020] Step 3: Generate data fusion strategy Using a reinforcement learning model, an executable data fusion strategy is dynamically generated based on the data fusion requirements (including data source identifiers, key features, and time windows) generated in step two, preprocessed data (containing metadata tags), and historical strategy effects (based on the execution effects of previously generated fusion strategies, such as prediction accuracy and computation time, serving as feedback signals for reinforcement learning).

[0021] Specifically, the design structure of a reinforcement learning model includes a state space, an action space, and a reward function. Furthermore: State space: The state includes the current data requirements (e.g., data source, features, time window, etc.) and the current state (e.g., computing resource utilization, computing load, etc.), as well as feedback on the effects of historical strategies to help the model understand which strategies have been proven effective; Action space: Actions include data association logic and fusion operation sequence, that is, selecting which data sources to associate with other data sources, such as extracting temperature data from the time series data layer and associating it with pressure data, while the fusion operation sequence corresponds to the selected operation steps to be executed; Reward function: It is dynamically adjusted based on the effectiveness of historical strategies and the achievement of current business goals. That is, if the generated fusion strategy successfully improves the prediction accuracy, the reward function will give a positive reward; if the strategy leads to a waste of computing resources or excessive computing time, it will give a negative reward.

[0022] Furthermore, reinforcement learning models themselves continuously learn and then update and optimize their strategies based on real-time data and new feedback, ensuring that the strategies always meet the current needs and environment.

[0023] Based on the training results of reinforcement learning, an executable data fusion strategy is generated, which includes: Data association logic: Determines how to associate data from different levels. The reinforcement learning model will learn to select the most effective association method, deciding which data should be used in combination based on the relevance of historical data and the matching of business needs. For example, temperature and pressure may be used as input features simultaneously in equipment failure prediction models, while equipment status information may be used in combination with sensor data in equipment health assessment. Data fusion operation sequence: Clearly define the operation steps and order to be performed during the data fusion process, such as data preprocessing (standardization, normalization, missing value imputation, etc.), feature engineering (feature selection or extraction to ensure that the model uses the most effective input features), and data fusion (using methods such as weighted average, data aggregation, and difference merging for data fusion). Resource scheduling scheme: Determine the required computing resources and execution order to optimize computing efficiency.

[0024] The final executable data fusion strategy includes: Data association logic: Specify the data sources and characteristics to be associated, and clarify the role and function of each data source; Fusion operation sequence: Determine the order of data preprocessing, feature extraction, and data aggregation operations; Resource scheduling scheme: Provides a scheme for scheduling computing resources to optimize system operating efficiency.

[0025] Step 4: Perform fusion and analysis Based on the fusion requirements in step two, target data is extracted from the hierarchical storage architecture. Then, the data fusion strategy generated in step three is executed to produce fused data tailored to the business objectives. This fused data is then input into a random forest learning model adapted to the business objectives for analysis. Random forest is a powerful ensemble learning algorithm that automatically selects the optimal features for classification or regression analysis by combining the results of multiple decision trees. The output includes structured analysis results corresponding to the user's objectives (analysis results specific to the user's objectives, such as fault prediction, equipment health assessment, and production optimization) and feature importance scores (helping to understand which data features are most critical to the model's decisions). For the random forest learning model, it can adapt to the hundreds of dimensions of cross-layer features (time series statistics and graph embedding vectors) generated by dynamic fusion, and can solve the complex coupling effect between equipment topology and vibration time series. It can also optimize the fusion strategy in real time through feature importance feedback.

[0026] Therefore, it will be trained based on the fused data to generate multiple decision trees. Each tree will be trained according to different features and data subsets, and finally output the result through voting (classification problem) or averaging (regression problem). If it is a classification task (e.g., whether the equipment is faulty), the model will output the probability of each category and finally select the category with the highest probability; if it is a regression task (e.g., equipment performance prediction), the model will output the predicted value (e.g., the remaining life of the equipment, output, etc.).

[0027] Step 5: Optimization of the fusion strategy After outputting the structured analysis results, the user's judgment on the effectiveness of the structured analysis results is obtained, which is divided into effective (the user believes that the analysis results are correct and can provide valuable decision support) and ineffective (the user believes that the analysis results are inaccurate or do not meet actual needs). Then, combined with the feature importance score obtained in step four, a quantitative value of the historical strategy effect is generated. This value represents the performance of the current fusion strategy in historical data, and its generation is based on the following: (1) If the user feedback is “effective”, the effect score of this strategy is a positive reward; if it is “ineffective”, the score is a negative feedback. (2) Based on the feature importance score, the contribution of each feature will be evaluated by weighting. For example, if a feature dominates in predictions with high effectiveness, its contribution weight will be increased.

[0028] (3) By summarizing all historical feedback, the average performance score of the strategy is obtained, and the performance of the strategy is quantified.

[0029] Therefore, a weighted average method is used to calculate the quantified value of the historical strategy's effectiveness, which is then input into the reinforcement learning model in step three for optimization. With each round of feedback and optimization, the model gradually adapts and adjusts the data fusion strategy to better meet different business objectives and real-time data requirements.

[0030] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Various changes and modifications can be made to the present invention without departing from its framework and scope of application, and all such changes and modifications fall within the scope of the present invention as claimed. The scope of protection of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for intelligent fusion and analysis of multi-source heterogeneous data, characterized in that: Includes the following steps: Step 1: Data Access and Processing It accesses various industrial data sources based on compatible industrial protocols, acquires equipment industrial data in real time, parses unstructured data, performs preprocessing to obtain preprocessed data, and then stores the preprocessed data using a hierarchical storage architecture. Step 2: Dynamic Target Analysis and Fusion Requirement Generation The system obtains the business objectives input by the user and parses them using natural language processing methods, transforming them into structured data requirements. Then, it maps the structured data requirements to the data within the hierarchical storage architecture to identify the required data sources. Finally, it uses a decision tree algorithm to generate fusion requirements based on the mapping results and business objectives. Step 3: Generate data fusion strategy Based on the fusion requirements in step two, and combining preprocessed data and historical strategy effects, an executable data fusion strategy is dynamically generated using a reinforcement learning model. Step 4: Perform fusion and analysis Based on the fusion requirements in step two, target data is extracted from the hierarchical storage architecture. Then, the data fusion strategy generated in step three is executed to generate fused data oriented towards business objectives. The fused data is then input into a random forest learning model adapted to the business objectives for analysis, and the structured analysis results and feature importance scores corresponding to the user objectives are output. Step 5: Optimization of the fusion strategy Obtain users' judgments on the effectiveness of the structured analysis results, and combine them with the feature importance scores obtained in step four to generate a quantitative value of the historical strategy effect. This value is then used as a negative feedback sample to input into step three to complete the optimization of the fusion strategy.

2. The intelligent fusion and analysis method for multi-source heterogeneous data according to claim 1, characterized in that: In step one, the specific preprocessing method is as follows: data anonymization and data cleaning are performed on the data, and then metadata tags are added, including time tags, source tags and quality tags.

3. The intelligent fusion and analysis method for multi-source heterogeneous data according to claim 1, characterized in that: In step one, the hierarchical storage architecture includes a distributed file layer for storing raw data, a time-series data layer for storing tagged sensor data, and a graph data layer for storing device topology relationships.

4. The intelligent fusion and analysis method for multi-source heterogeneous data according to claim 1, characterized in that: In step two, the structured data requirements include key business entities, analytical metrics, and analytical dimensions.

5. The intelligent fusion and analysis method for multi-source heterogeneous data according to claim 1, characterized in that: In step two, the fusion requirements include data range, key features, and time window.

6. The intelligent fusion and analysis method for multi-source heterogeneous data according to claim 1, characterized in that: In step three, the data fusion strategy includes data association logic, fusion operation sequence, and resource scheduling scheme.

7. The intelligent fusion and analysis method for multi-source heterogeneous data according to claim 1, characterized in that: In step four, the random forest learning model is a pre-trained machine learning model.

8. The intelligent fusion and analysis method for multi-source heterogeneous data according to claim 1, characterized in that: In step five, the validity of the structured analysis results is divided into valid and invalid.

Images