Industrial data processing system and industrial data processing method based on full life cycle

Abstract

The invention discloses an industrial data processing system and an industrial data processing method based on a full life cycle, relates to the technical field of industrial data processing, and can accurately provide effective support for an actual production decision so as to meet real-time monitoring and quick response of an industrial field. The system comprises a data acquisition module, a data cleaning module, a data analysis module, a data monitoring module and a system integration module, and the functions of acquisition, cleaning, analysis, tracking feedback and the like of industrial data in the industrial production process are realized by orderly connecting all functional modules. And the data isolation state among the functional modules is broken, so that each functional module can output high-quality industrial data, and the interpretability of a data analysis result is improved.

Description

Technical Field

[0001] The present application relates to the technical field of industrial data processing, and in particular to an industrial data processing system and an industrial data processing method based on the entire life cycle. Background Art

[0002] In the context of big data, data are becoming increasingly diversified, complex and large-scale, and the amount of data on data management platforms is constantly increasing, resulting in complex data relationships, unclear context and uneven data quality, which seriously affects the effectiveness of data.

[0003] In the process of processing industrial data, related technologies usually use machine learning and deep learning algorithms provided by big data platforms to realize energy efficiency analysis of industrial data. However, considering the diverse sources and types of industrial data, the results of energy efficiency analysis of industrial data through machine learning and deep learning algorithms often lack interpretability, making it difficult to provide effective support for actual production decisions, and unable to meet the real-time monitoring and rapid response of industrial sites, limiting the application scenarios of data analysis results. Summary of the invention

[0004] In view of this, the present application provides an industrial data processing system and an industrial data processing method based on the entire life cycle. The main purpose is to solve the problem that the results of energy efficiency analysis of industrial data through machine learning and deep learning algorithms in the prior art often lack interpretability, making it difficult to provide effective support for actual production decisions and unable to meet the real-time monitoring and rapid response of industrial sites.

[0005] According to a first aspect of the present application, there is provided an industrial data processing system based on a full life cycle, comprising: A data acquisition module is used to obtain real-time industrial data of each production process in the industrial production process, and the real-time industrial data has a unified data format; A data cleaning module, used to clean the real-time industrial data according to the process characteristics corresponding to each production process to obtain cleaned industrial data, wherein the cleaning process includes one or more of noise reduction, data filling, abnormal data identification and correction, and preset cleaning rules; A data analysis module, used to perform data benchmarking and parameter optimization adjustment on the cleaned industrial data to obtain data benchmarking results and parameter optimization results; A data monitoring module, used to monitor the data acquisition module, the data cleaning module and the data analysis module through data tracking technology, so as to establish a data feedback mechanism for each production process in the industrial production process according to the data benchmarking results and the parameter optimization results; A system integration module is used to integrate the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module into the system platform through a data connector, so that the system platform can adjust the parameters of each production process in the industrial production process based on the data feedback mechanism.

[0006] Furthermore, the data acquisition module includes a sensor network, which is arranged in each production process in the industrial production process. The sensor network is a multi-parameter sensor constructed using Internet of Things technology, and the multi-parameter sensor includes sensors deployed in each production link in the industrial environment; The data acquisition module is specifically used to use the sensor network to collect indicator data output by heterogeneous data sources, and use standardized data interfaces and protocols to integrate the indicator data output by the heterogeneous data sources to obtain real-time industrial data of each production process in the industrial production process.

[0007] Further, the data cleaning module includes a combination of one or more units selected from a data noise reduction unit, a data filling unit, a data correction unit, and a custom cleaning unit; The data denoising unit is used to decompose the real-time industrial data into multiple modal functions, and to perform denoising on the real-time industrial data by filtering the modal functions whose frequencies are greater than a set value: The data filling unit is used to detect missing values ​​in the real-time industrial data, and for completely random missing data and non-critical data, the missing data is deleted according to a set time granularity, and for random missing data and non-random missing data, the missing data is filled by a random forest algorithm; The data correction unit is used to cluster the real-time industrial data using a clustering algorithm in the industrial production process, obtain the number of data points in the cluster by statistical clustering, determine the data points in the cluster whose number is less than a set threshold as data abnormal points, and use the average interpolation method to perform data correction processing on the data abnormal points; The custom cleaning unit is used to perform cleaning processing on the real-time industrial data using set cleaning rules.

[0008] Furthermore, the system has a data version control function and a data storage function, and the data version control function and the data storage function support the global application of the system; The data version control function is used to record the version of the cleaned industrial data through metadata information, so that the industrial data after version recording has traceability; The data storage function is used to establish a data dictionary so as to store the version-recorded industrial data in a distributed data warehouse through the data dictionary.

[0009] Furthermore, the data analysis module includes a data benchmarking unit and an optimization modeling unit; The data benchmarking unit is used to identify the difference between the cleaned industrial data and the benchmark industrial data, so as to determine the data benchmarking result through the performance data obtained by the difference identification, and the benchmark industrial data is the energy consumption data correspondingly collected when the process equipment is in a steady state in the industrial production process within a set time range; The optimization modeling unit is used to establish a process processing model for each production process in the industrial production process, so as to optimize the parameters of the cleaned industrial data through the process processing model to obtain parameter optimization results, and the parameter optimization results meet the target production conditions.

[0010] Further, the data monitoring module includes a data tracking unit and a data feedback unit; The data tracking unit is used to record the dynamic information of the industrial data of each production process in the data acquisition module, the data cleaning module and the data analysis module through the data tracking technology, so as to ensure the integrity of the industrial data according to the dynamic information: The data feedback unit is used to continuously adjust the production parameters of each production process in the production process based on the parameter optimization results according to the data benchmarking results, so as to establish a data feedback mechanism for each production process in the generation process through the data application results after parameter adjustment.

[0011] Furthermore, the system integration module includes a data interaction unit and a data integration unit: The data interaction unit is used to provide a unified data interaction interface between the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module through a data connector, so as to realize data interaction between different applications according to the unified data interaction interface; The data integration unit is used to integrate the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module into the system platform through the unified data interface using a standardized interface protocol, so that the system platform can adjust parameters of each production process in the industrial production process based on the data feedback mechanism.

[0012] According to a second aspect of the present application, a method for processing industrial data based on the entire life cycle is provided, comprising: Acquire real-time industrial data of each production process in the industrial production process, wherein the real-time industrial data has a unified data format; According to the process characteristics corresponding to each production process, the real-time industrial data is cleaned to obtain cleaned industrial data, wherein the cleaning process includes one or more of noise reduction, data filling, abnormal data identification and correction, and preset cleaning rules; Performing data benchmarking and parameter optimization adjustment on the cleaned industrial data to obtain data benchmarking results and parameter optimization results; Monitor the data acquisition module, the data cleaning module and the data analysis module through data tracking technology, so as to establish a data feedback mechanism for each production process in the industrial production process according to the data benchmarking results and the parameter optimization results; The data acquisition module, the data cleaning module, the data analysis module and the data monitoring module are integrated into the system platform through a data connector, so that the system platform can adjust the parameters of each production process in the industrial production process based on the data feedback mechanism.

[0013] According to a third aspect of the present application, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the method described in the third aspect when executing the computer program.

[0014] According to a fourth aspect of the present application, a readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the method described in the third aspect are implemented.

[0015] Through the above-mentioned technical scheme, the present application provides an industrial data processing system and industrial data processing method based on the entire life cycle. Compared with the current existing technology that uses machine learning and deep learning algorithms provided by a big data platform to realize energy efficiency analysis of industrial data, the industrial data processing system based on the entire life cycle in the present application includes a data acquisition module, a data cleaning module, a data analysis module, a data monitoring module and a system integration module. By orderly connecting various functional modules, the functions of industrial data collection, cleaning, analysis, tracking and feedback in the industrial production process are realized, breaking the data isolation between various functional modules, so that each functional module can output high-quality industrial data and improve the interpretability of data analysis results. This process realizes data monitoring and data optimization of each production process in the industrial production process by introducing advanced data mining and artificial intelligence technologies, which can accurately provide effective support for actual production decisions, thereby meeting real-time monitoring and rapid response of industrial sites.

[0016] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 It is a structural diagram of an industrial data processing system based on the entire life cycle provided by an embodiment of the present application; Figure 2 It is a schematic diagram of the algorithm flow of signal noise reduction provided by an embodiment of the present application; Figure 3 This is a schematic diagram of the algorithm flow of the random forest provided in one embodiment of the present application; Figure 4 This is a schematic diagram of an asynchronous replication execution process provided by an embodiment of the present application; Figure 5 It is a structural diagram of data analysis and optimization in one embodiment of the present application; Figure 6 It is a flowchart of realizing data integration through the mechanism, algorithm model and data model in the IEMS data connector provided by an embodiment of the present application; Figure 7 It is a flowchart of an industrial data processing method based on the entire life cycle in one embodiment of the present application; Figure 8 It is a schematic diagram of the device structure of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] Now the content of the present invention will be discussed with reference to several exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and thus implement the content of the present invention, rather than implying any limitation on the scope of the present invention.

[0019] As used herein, the term "including" and variations thereof are to be interpreted as open-ended terms meaning "including but not limited to." The term "based on" is to be interpreted as "based, at least in part, on." The terms "one embodiment" and "an embodiment" are to be interpreted as "at least one embodiment." The term "another embodiment" is to be interpreted as "at least one other embodiment."

[0020] In the process of processing industrial data, related technologies usually use machine learning and deep learning algorithms provided by big data platforms to realize energy efficiency analysis of industrial data. However, considering the diverse sources and types of industrial data, the results of energy efficiency analysis of industrial data through machine learning and deep learning algorithms often lack interpretability, making it difficult to provide effective support for actual production decisions, and unable to meet the real-time monitoring and rapid response of industrial sites, limiting the application scenarios of data analysis results.

[0021] In order to solve this problem, this embodiment provides an industrial data processing system based on the entire life cycle, such as Figure 1 As shown, the system includes: a data acquisition module 11, a data cleaning module 12, a data analysis module 13, a data monitoring module 14 and a system integration module 15; The data acquisition module 11 is used to obtain real-time industrial data of each production process in the industrial production process, and the real-time industrial data has a unified data format; The data cleaning module 12 is used to clean the real-time industrial data according to the process characteristics corresponding to each production process to obtain cleaned industrial data, wherein the cleaning process includes one or more of noise reduction, data filling, abnormal data identification and correction, and preset cleaning rules; The data analysis module 13 is used to perform data benchmarking and parameter optimization adjustment on the cleaned industrial data to obtain data benchmarking results and parameter optimization results; A data monitoring module 14, used to monitor the data acquisition module, the data cleaning module and the data analysis module through data tracking technology, so as to establish a data feedback mechanism for each production process in the industrial production process according to the data benchmarking results and the parameter optimization results; The system integration module 15 is used to integrate the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module into the system platform through a data connector, so that the system platform can adjust the parameters of each production process in the industrial production process based on the data feedback mechanism.

[0022] In this embodiment, the data processing process based on the full life cycle of industrial data can serve the in-depth analysis and full life cycle management of industrial data in the era of big data analysis.

[0023] Taking the production and energy data analysis of the steel industry as an example, the collection, cleaning, governance, analysis, tracking and feedback, interaction and management of energy consumption data of key processes in the steel industry are carried out. At this time, the industrial data processing process based on the whole life cycle includes strengthening system functions such as data collection, data statistics, data analysis, data verification and data benchmarking, and using system functions and analysis models to support the energy efficiency benchmarking, energy efficiency diagnosis and energy efficiency carbon reduction of industry enterprises. This system will optimize the data management process and play a vital role in realizing intelligent manufacturing and energy conservation in the steel industry. Unlike previous data management systems, the industrial data processing process based on the whole life cycle in this embodiment breaks the isolated state between the various links of data management, realizes the seamless connection of various links, and introduces advanced data mining and artificial intelligence technologies to establish data analysis models to support enterprises to achieve more refined management and decision-making optimization, and help enterprises achieve energy conservation and carbon reduction in the production process.

[0024] Specifically, the data acquisition module includes a sensor network, which is arranged in each production process in the industrial production process. The sensor network is a multi-parameter sensor constructed using Internet of Things technology. The multi-parameter sensor includes sensors deployed in each production link in the industrial environment; The data acquisition module is specifically used to use the sensor network to collect indicator data output by heterogeneous data sources, and use standardized data interfaces and protocols to integrate the indicator data output by the heterogeneous data sources to obtain real-time industrial data of each production process in the industrial production process.

[0025] In the above data acquisition module, the SCADA system can be used to collect data in real time from various links in the steel production process (such as blast furnace ironmaking, converter steelmaking, rolling, etc.). By deploying high-precision, multi-parameter sensors and using the Internet of Things technology to build an intelligent sensor network, accurate monitoring and high-frequency sampling of key parameters such as temperature, pressure, flow, and composition can be achieved. Then, standardized data interfaces and protocols are used to uniformly process and convert heterogeneous data sources to form a standardized and consistent data format. Real-time and stable data transmission is achieved through high-speed wireless and wired networks, and data verification and error correction mechanisms are introduced to ensure data accuracy and reliability. Secure encrypted transmission and strict permission management ensure the security of data during transmission and access. Finally, data from different sources are integrated to form a unified data set, providing high-quality data support for data cleaning, governance, and analysis, improving the utilization value of data, and helping enterprises achieve refined management and intelligent production.

[0026] In actual application scenarios, the data collection environment is complex and diverse. Data sources in industrial environments often include PLC, DCS, sensors, RFID, barcode systems, etc. These devices and systems use different communication protocols and data formats, which increases the complexity of data collection. In industrial environments with high real-time requirements, data collection delays and data loss problems still exist, affecting the timeliness and accuracy of data. Although edge computing can perform preliminary processing of data locally, the integration with the cloud computing platform is not tight enough, resulting in low data transmission and processing efficiency. For the data collection module improvement process, the data collection module in this embodiment includes multi-source data collection, data fusion, data filtering and transmission.

[0027] The specific multi-source data collection process includes the following steps: Using advanced sensor technology and Internet of Things (IoT) technology, the SCADA system collects data from each process of steel production in real time. The collected data includes key production parameters such as temperature, pressure, flow, and energy consumption.

[0028] The specific data fusion process includes the following steps: By using ETL tools, data from different sources and different formats are converted into a unified data type and merged into a unified data set. During the data fusion process, the system formats the data to ensure data consistency and compatibility.

[0029] The specific data filtering and transmission process includes the following steps: the system uses high-precision sensors and low-latency data transmission technology to ensure the real-time and accuracy of data collection, and uses industrial Ethernet or field bus to transmit the collected data to the central control system or data center, and uses wireless communication technologies such as Wi-Fi, ZigBee, LoRa, etc. for data transmission on difficult-to-wire or mobile devices. Through edge computing technology, preliminary processing and filtering are performed at the data collection end to reduce the transmission of invalid data and improve data processing efficiency.

[0030] Specifically, the data cleaning module includes a combination of one or more units among a data noise reduction unit, a data filling unit, a data correction unit, and a custom cleaning unit; The data denoising unit is used to decompose the real-time industrial data into multiple modal functions, and to perform denoising on the real-time industrial data by filtering the modal functions whose frequencies are greater than a set value: The data filling unit is used to detect missing values ​​in the real-time industrial data, and for completely random missing data and non-critical data, the missing data is deleted according to a set time granularity, and for random missing data and non-random missing data, the missing data is filled by a random forest algorithm; The data correction unit is used to cluster the real-time industrial data using a clustering algorithm in the industrial production process, obtain the number of data points in the cluster by statistical clustering, determine the data points in the cluster whose number is less than a set threshold as data abnormal points, and use the average interpolation method to perform data correction processing on the data abnormal points; The custom cleaning unit is used to perform cleaning processing on the real-time industrial data using set cleaning rules.

[0031] In the above data cleaning module, when the collected real-time industrial data is put into actual production data, there are often duplicate, missing, abnormal and inconsistent time granularity data. The main goal of data cleaning is to improve the data and lay a data foundation for establishing the processing model of each process of steel production. The data noise reduction processing can also reduce the complexity of the model, improve the efficiency and performance of the model, reduce the risk of overfitting, and improve the generalization ability of the model.

[0032] In the above data cleaning module, data processing is performed on the collected data to ensure the integrity of the data. The data cleaning module intelligently fills in the missing data, that is, a data-driven machine learning algorithm is used for important data, including any one or a combination of K-Nearest Neighbors (KNN), Multiple Imputation, and Random Forest (RF), and missing values ​​are predicted and filled according to the historical patterns and correlations of the data, so as to maintain the original characteristics and trends of the data to the greatest extent. For the identification and correction of abnormal data, the data cleaning module uses complex anomaly detection algorithms, statistical detection methods including Z-Score, Interquartile Range, etc., clustering algorithms including K-means algorithm (KMeans), density-based clustering algorithm (DBSCAN), etc., as well as time series analysis and deep learning models, which can accurately identify abnormal points in the data. The identified abnormal data will be further analyzed and verified, and automatically or semi-automatically corrected or eliminated to ensure the reliability and consistency of the data.

[0033] In the above data cleaning module, data cleaning rule management and visual ETL (Extract, Transform, Load) tools are also designed to support user-defined cleaning rules and parameter adjustment, and intuitively display the process and results of data cleaning through a graphical interface, which is convenient for users to monitor and tune. Through the logging and report generation functions, the data cleaning module can record the operation and effect of each data processing in detail, provide comprehensive auditing and traceability capabilities, and ensure the transparency and controllability of the data cleaning process. Data that has been strictly cleaned can provide a basis for subsequent data governance, analysis, and decision-making, and improve the utilization value of data and the reliability of analysis.

[0034] In actual application scenarios, the data cleaning process is not automated and intelligent enough, there is a lack of automated data cleaning tools, data quality is difficult to guarantee, and there is a lot of noise and redundant information; data cleaning rules need to be customized according to specific business scenarios, the definition and management of rules are relatively complex, and lack flexibility and versatility; for real-time streaming data, traditional batch cleaning methods are difficult to apply, real-time cleaning technology is not yet mature, and processing performance and effects need to be improved. For the improvement process of the data cleaning module, the data cleaning module in this embodiment includes any one or more of data noise reduction processing, missing data processing, abnormal data identification and correction, data deduplication, and data standardization.

[0035] The specific data denoising process includes the following steps: using the empirical mode decomposition (EMD) method to decompose the data into a series of intrinsic mode functions (IMFs), and achieving data denoising by discarding high-frequency IMF components.

[0036] The EMD denoising process is as follows: Perform EMD decomposition to obtain IMF components and the residual , set the threshold k to divide the IMF component into high-frequency components and low frequency components , only retain the low-frequency components and residuals, and recombine them to obtain the denoised signal , the detailed signal noise reduction algorithm process is as follows Figure 2 As shown in the figure, the noisy signal is decomposed into high-frequency IMF and low-frequency IMF by EMD decomposition, and then the high-frequency IMF is decomposed by wavelet to obtain high-frequency coefficients and low-frequency coefficients. Then the high-frequency coefficient is threshold quantized and reconstructed by wavelet, and the high-frequency IMF reconstructed by wavelet is combined with the low-frequency IMF to reconstruct the signal and obtain the denoised signal. The specific reconstructed data can be calculated by the following formula:

[0037] The specific missing data processing process includes the following steps: the system automatically detects missing values ​​in the data set, and for completely random missing data (MCAR) and non-critical data, it is processed by deleting them according to the unified time granularity of the model, thereby saving data processing time; when processing random missing data (MAR) and non-random missing data (MNAR), the random forest method is used to fill in the missing data, thereby fully restoring the true situation of the original data.

[0038] The random forest method fills missing data as follows: identify the location of missing values ​​in the data set, extract all samples containing the feature for each feature with missing values, and first perform preliminary filling. Let the jth feature There are missing values, and the matrix after initial filling is The detailed random forest algorithm process is as follows Figure 3 As shown, firstly, some features are randomly extracted as candidate features, and the test features are determined in the candidate features using the Gini index. After the node is generated, it is determined whether the node can become a leaf node. If it cannot become a leaf node, branching is performed and each branch node is recursively performed. If it can become a leaf node, it is determined whether the decision tree stops growing. If it does not stop growing, branching is performed and each branch node is recursively performed. If it stops growing, the decision tree is stored and it is determined whether the number of decision trees meets the requirements. If yes, a random forest is generated. If not, randomly extracting features as candidate features is performed again. The specific mean filling can be calculated by the following formula:

[0039] In the above formula, is the sample set with no missing values ​​for the jth feature, Is a collection size.

[0040] Build a random forest model, for each feature containing missing values , build a random forest model Using other features As input, specific prediction The value of can be calculated by the following formula:

[0041] In the above formula, t represents the number of iterations.

[0042] The above iterative optimization process can repeat the above steps until convergence. The specific convergence standard can be calculated by the following formula:

[0043] In the above formula, Indicates the preset convergence standard.

[0044] The specific process of abnormal data identification and correction includes the following steps: During the normal production of each process, the sensor may receive false signals such as start and end for some reasons, resulting in abnormal values ​​in the database. The KMeans clustering algorithm improved by a clustering effective index BWP is used to cluster the data, count the number of data points in each cluster, and set the threshold , if the number of clusters is less than the threshold , then the data point is considered to be an outlier and the mean value interpolation method is used for correction.

[0045] The BWP-KMeans clustering algorithm process is as follows: is the clustering space, where , assuming The sample objects are clustered into c categories, and the inter-class and intra-class partitioning index BWP (j, i) of the i-th sample of the j-th category is defined as the ratio of the cluster deviation distance and the cluster distance of the sample, which is calculated by the following formula:

[0046]

[0047] In the above formula, set the search range of the number of clusters ,from Loop to , select the initial cluster center according to the principle of maximum distance and minimum distance; use the KMeans clustering algorithm to update the cluster center until the distance function converges; through the above formula, the average value of the BWP index of all samples calculated by the clustering results can be calculated by the following formula:

[0048] In the above formula, compare the The size of The k value corresponding to the maximum value is the optimal number of clusters. Use the optimal k value to perform KMeans clustering on the data, divide the data into k clusters, count the data points of each cluster, find the cluster that may contain abnormal data, and if the number of data points in the cluster is less than the previously set threshold , then the data points in this cluster are abnormal data.

[0049] The specific data deduplication process includes the following steps: In the actual production process, special circumstances may occur that cause the sensor to repeatedly transmit the same data to the database, so the system needs to identify and delete the duplicate data to ensure the uniqueness and accuracy of the data set. Through hash algorithms and similarity matching technology, the system can efficiently identify and remove duplicate data.

[0050] The specific data standardization process includes the following steps: the system standardizes the data, including unit conversion, dimension unification, etc., to ensure the consistency and comparability of the data.

[0051] In actual application scenarios, in order to solve the data inconsistency phenomenon existing in industrial data, the system further has a data version control function and a data storage function, and the data version control function and the data storage function support the global application of the system; The data version control function is used to record the version of the cleaned industrial data through metadata information, so that the industrial data after version recording has traceability; The data storage function is used to establish a data dictionary so as to store the industrial data recorded in the version in a distributed data warehouse through the data dictionary.

[0052] The above-mentioned data version control function and data storage function can be realized through data management. Through data consistency verification, data version control and other measures, industrial data can be managed in a systematic and standardized manner, and then the data can be stored in the database to ensure the comprehensiveness and traceability of the data, which is convenient for unified data management.

[0053] Specifically, the data version control function can implement data consistency verification to automatically detect and correct consistency problems in the data, strictly manage and record different versions of the data, support historical tracking and rollback of data, and ensure data traceability. Ultimately, a comprehensive data dictionary and a sound metadata management system will be established to systematically record and manage all aspects of data collection, conversion, storage, use, and destruction to ensure the comprehensiveness of data governance.

[0054] Specifically, for the data storage module, distributed database technology can be used to efficiently store and manage data, support horizontal expansion and high availability of data, and use data compression technology and optimize index structure to reduce storage space usage to improve data query and processing efficiency. An integrated data warehouse is used to support data separation storage solutions, and a complete data backup and recovery mechanism is implemented to ensure data security and recoverability.

[0055] In actual application scenarios, the mechanism of data management is not sound, data islands are serious, and data integrity and consistency are insufficient, which cannot provide a reliable basis for subsequent analysis; in the data management process, there is a lack of strict version control and consistency verification, which leads to data redundancy and conflict, affecting the integrity and reliability of the data. The data management link here can be improved through data version control and data storage functions. The improvement of data version control mainly includes data dictionary and metadata management; the improvement of data storage function mainly includes distributed database design.

[0056] The specific data dictionary and metadata management process includes the following steps: establish a data dictionary, record the type, unit and other information of the data, and provide data query and usage functions; record metadata information such as the source, collection time, and collection method of the data to ensure the transparency and traceability of the data. Through metadata information, the source and collection process of the data can be traced. When data anomalies occur, the metadata can be traced back to the specific sensor device and collection time to find out the cause of the problem and solve the data anomaly problem.

[0057] The specific distributed database design process includes the following steps: using MySQL's distributed architecture to implement the above storage function, and using MySQL's own master-slave replication mechanism and asynchronous replication mode. The execution process of the asynchronous replication mode is as follows: Figure 4 As shown in the figure, the asynchronous replication mode can make MySQL's distributed architecture have the advantages of high fault tolerance, support for multiple storage engines (such as MyISAM, NDB, InnoDB), support for multiple index types, and rich management and monitoring tools, meeting the needs of modern enterprises for large-scale data management and high-performance database systems.

[0058] Specifically, the data analysis module includes a data benchmarking unit and an optimization modeling unit; The data benchmarking unit is used to identify the difference between the cleaned industrial data and the benchmark industrial data, so as to determine the data benchmarking result through the performance data obtained by the difference identification, and the benchmark industrial data is the energy consumption data correspondingly collected when the process equipment is in a steady state in the industrial production process within a set time range; The optimization modeling unit is used to establish a process processing model for each production process in the industrial production process, so as to optimize the parameters of the cleaned industrial data through the process processing model to obtain parameter optimization results, and the parameter optimization results meet the target production conditions.

[0059] The above-mentioned data analysis module is an important tool to support energy efficiency benchmarking, energy efficiency diagnosis and energy efficiency carbon reduction for industry enterprises. First, data benchmarking is implemented to identify the relative performance and room for improvement by comparing and evaluating the differences between different data sets and data benchmarks (representative and excellent performance standards such as energy efficiency data of industry benchmark enterprises as comparison benchmarks). Then the module uses big data mining technology and data-driven modeling to conduct in-depth analysis of massive data in each process of steel production, tap its potential for energy saving and carbon reduction, and transmit real-time data to each process processing model through a third-party API subsystem to ensure seamless integration and interoperability of industrial data, realize real-time monitoring and optimization of data in each process of the production process, and apply machine learning and artificial intelligence algorithms to help improve the intelligent process of each process and achieve energy saving and carbon reduction goals.

[0060] The specific data analysis and optimization process is as follows: Figure 5 As shown, Figure 5 The analysis and optimization process is mainly aimed at the quality system, equipment system, energy system and production system. On the one hand, the big data analysis process within and between systems mainly includes multi-source data fusion, time series feature extraction, correlation analysis and operation performance prediction. Through the big data analysis within and between systems, the correlation between parameter performance in a single system can be obtained. The correlation between parameter performance in a single system can be applied to intelligent learning and scheduling optimization, and a single system intelligent strategy can be generated to achieve multi-system collaborative optimization of energy optimization, equipment operation and maintenance, and quality control. On the other hand, the big data analysis within and between systems can establish a fault maintenance plan through the association between systems, and realize information sharing through collaborative optimization based on the correlation between the fault status maintenance plan and the parameter performance in a single system.

[0061] In actual application scenarios, the technologies and tools used in the data analysis link have limitations, and the analysis methods and tools are relatively simple, making it difficult to conduct in-depth mining and intelligent analysis of complex industrial data, and unable to effectively support refined management and decision optimization; although some advanced machine learning and deep learning algorithms can achieve good analysis results, their results often lack interpretability and are difficult to provide effective support for actual production decisions; in the case of high-frequency data streams and large-scale data volumes, real-time analysis technology is not yet mature, and it is difficult to meet the real-time monitoring and rapid response requirements of industrial sites; data visualization tools have single functions and are difficult to meet the needs of complex industrial application scenarios for data display and interaction, limiting the application and promotion of data analysis results. For the data analysis module improvement process, the data analysis module in this embodiment includes one or more combinations of data benchmarking and optimization modeling.

[0062] The specific data benchmarking process includes the following steps: select the stable operation time of the process equipment, the lowest energy efficiency data during this period as the comparison benchmark, and organize the benchmark data to establish a benchmark database. The system analyzes the company's own data and benchmark data through benchmarking to identify differences and room for improvement.

[0063] The specific optimization modeling process includes the following steps: establish a processing model for each process in steel production, which can use historical data to train the model, use model evaluation indicators to evaluate and optimize the model, and use a third-party API subsystem to transfer real-time data to each process processing model to ensure seamless integration and interoperability of data, and realize real-time monitoring and optimization of data in each process of the production process. It should be noted that each evaluation indicator in the model evaluation index is different, each process is different, and each method is also different. For example, the energy efficiency improvement rate can be achieved through energy efficiency benchmarking or optimization. The model can help identify improvement space and propose optimization plans. The energy efficiency improvement rate reflects the contribution of the model to improving energy efficiency. For example, the contribution to carbon emission reduction can be evaluated by comparing the carbon emissions before and after optimization, which can achieve the effect of the model in energy saving and carbon reduction.

[0064] Specifically, the data monitoring module includes a data tracking unit and a data feedback unit; The data tracking unit is used to record the dynamic information of the industrial data of each production process in the data acquisition module, the data cleaning module and the data analysis module through the data tracking technology, so as to ensure the integrity of the industrial data according to the dynamic information: The data feedback unit is used to continuously adjust the production parameters of each production process in the production process based on the parameter optimization results according to the data benchmarking results, so as to establish a data feedback mechanism for each production process in the generation process through the data application results after parameter adjustment.

[0065] In the above-mentioned data monitoring module, data monitoring is to achieve full-process data tracking and feedback, ensuring the visualization and controllability of data in all links. Specifically, through advanced data tracking technology, this module can monitor and record the entire process from data collection to data analysis in real time, provide detailed log and audit functions, and ensure the integrity and transparency of the data. At the same time, an efficient data feedback mechanism is established to apply the results of data analysis to the production process in a timely manner, dynamically adjust and optimize production parameters, thereby improving production efficiency and product quality. In general, the data monitoring module can provide enterprises with a comprehensive, reliable and efficient data management and feedback system, supporting enterprises to achieve continuous improvement and optimization in the process of digital transformation and intelligent manufacturing.

[0066] In actual application scenarios, the data monitoring module includes data tracking and data feedback. The specific data tracking process includes the following steps: record the timestamp, source, processing process and application results of each data point to form a complete data tracking log, realize the full process tracking of production data, and monitor the production status in real time through data visualization tools. The specific data feedback process includes the following steps: transmit the analysis and optimization results to the control system of each process through the API interface, adjust the production parameters in time, optimize the production process, improve production efficiency, reduce energy consumption, and continuously update and improve the data analysis model through feedback data to realize the feedback process.

[0067] Specifically, the system integration module includes a data interaction unit and a data integration unit: The data interaction unit is used to provide a unified data interaction interface between the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module through a data connector, so as to realize data interaction between different applications according to the unified data interaction interface; The data integration unit is used to integrate the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module into the system platform through the unified data interface using a standardized interface protocol, so that the system platform can adjust parameters of each production process in the industrial production process based on the data feedback mechanism.

[0068] Among the above-mentioned system integration modules, the system integration module is a module that develops and integrates the above-mentioned functions. The main goal of this module is to ensure the organic combination of various functional modules, form a highly integrated and collaborative system platform, and provide a friendly user interface and convenient data operation tools.

[0069] The above system integration module can be applied to the system to support multi-user collaborative work through the system platform. It has permission management and user allocation functions to ensure that different users can access and operate data efficiently and securely.

[0070] In this embodiment, the system integration module adopts a modular design concept to seamlessly integrate data acquisition, data cleaning, data management, data analysis and data monitoring modules to form an efficient and reliable industrial data processing system. The system after module integration not only focuses on the functional realization of each module, but also pays attention to the overall performance, stability and scalability to ensure that the system can operate stably under high load and complex environment.

[0071] In this embodiment, the system integration module fully considers the existing information infrastructure and management system of the enterprise, especially the key business systems such as the Enterprise Resource Planning System (ERP) and the Manufacturing Execution System (MES), and achieves seamless connection with the existing system by adopting standardized interface protocols and APIs to ensure the smooth transmission of industrial data and the integration of business processes. In addition, the system integration module also includes a complete user authority management, security control and data backup and recovery mechanism to ensure the security of the system and the integrity of the data.

[0072] In actual application scenarios, the data integration process is realized through the mechanism, algorithm model and data model in the data connector. Figure 6 As shown, in Figure 6 In the data connector, the data model and the model of the data generated by the integrated energy system are transmitted to the integrated energy system through objects. As the data generated by the integrated energy system, the IES scheduling optimization model, energy consumption prediction model, gas power generation prediction model, power load prediction model, energy storage scheduling optimization model, etc. are input from the integrated energy management system to the integrated energy system through objects through the data-driven mechanism / algorithm model, and the data generated by the integrated energy system is transmitted to the integrated energy management system.

[0073] In actual application scenarios, the point-to-point integration method requires each application provider to customize the definition of basic data, attributes, data models, data twins, devices and data exchange interfaces. In this embodiment, the system integration module uses the integrated energy management system (IEMS) as an integration method for data connectors to achieve data exchange between different applications. Specifically, basic data, attributes and data models can be shared within the data connector and gradually converge through the operation strategy of the data connector; data twins and devices are uniformly managed within the same data connector as a blueprint for communication between applications; the API interface maintains consistency for all applications, data connectors and data models, avoiding repeated interface development work. This structure and strategy achieves interoperability of data exchange between applications in the data connector and improves the efficiency of data exchange.

[0074] The specific evaluation criteria for IEMS data connectors are shown in the following table:

[0075] Furthermore, as a personalized function setting of the industrial data processing system, the industrial data processing system in this embodiment introduces dynamic simulation software that integrates deep learning, performs real-time analysis and learning of industrial data in the production process, and constructs dynamic simulation models. These dynamic simulation models can predict abnormal situations that may occur in the production process and provide preventive maintenance suggestions, thereby reducing downtime and maintenance costs. At the same time, the dynamic simulation software can optimize production parameters and improve production efficiency and product quality.

[0076] Furthermore, as a personalized function setting of the industrial data processing system, the industrial data processing system in this embodiment integrates a data management module, and formulates an optimal energy use plan through big data analysis and optimization algorithms. The data monitoring module can monitor the energy consumption of each production unit in real time, and dynamically adjust the energy allocation plan in combination with production demand and energy supply, reduce energy consumption costs, and achieve green manufacturing. At the same time, the system integrates an intelligent scheduling algorithm to optimize the production scheduling plan by analyzing multiple factors such as production tasks, equipment status, and staffing. The system can automatically generate a scheduling plan and dynamically adjust it according to the actual situation of the production site to improve production flexibility and response speed.

[0077] Furthermore, as a personalized function setting of the industrial data processing system, the industrial data processing system in this embodiment provides interactive charts and reports to facilitate managers to filter and manage data. Create multiple customized dashboards to display key production parameters, energy consumption indicators, key performance indicators (KPIs) and other information of different processes and production lines, and use a variety of visualization components to improve the intuitiveness of data display.

[0078] Furthermore, as a personalized function setting of the industrial data processing system, the industrial data processing system in this embodiment has real-time data display and historical data analysis functions. Real-time data, including production status, equipment status, total energy consumption, etc., are displayed on the dashboard to ensure that managers can monitor production in real time. The system provides a historical data analysis function, which allows users to view historical data trends and changes by selecting different time ranges, and store real-time data in a historical database.

[0079] Furthermore, as a personalized function setting of the industrial data processing system, the industrial data processing system in this embodiment has the functions of command push and report generation and export. Push the optimization scheduling instructions to the control system or operator terminal at the production site to ensure timely adjustment of production parameters. By providing command confirmation and execution feedback functions, the accurate execution of scheduling instructions is ensured. By providing an automated report generation function, production reports, energy consumption reports, KPI reports, etc. are generated according to the set time period, and data and reports can be exported to multiple formats (such as Excel, PDF, CSV), which is convenient for managers to conduct offline analysis and sharing.

[0080] Further, as Figure 1 The specific implementation of the industrial data processing system based on the entire life cycle, the embodiment of the present application provides an industrial data processing method based on the entire life cycle, such as Figure 7 As shown, the method includes: 701. Acquire real-time industrial data of each production process in an industrial production process, where the real-time industrial data has a unified data format.

[0081] 702. According to the process characteristics corresponding to each production process, the real-time industrial data is cleaned to obtain cleaned industrial data. The cleaning process includes a combination of one or more of noise reduction, data filling, abnormal data identification and correction, and preset cleaning rules.

[0082] 703. Perform data benchmarking and parameter optimization adjustment on the cleaned industrial data to obtain data benchmarking results and parameter optimization results.

[0083] 704. Monitor the data acquisition module, the data cleaning module and the data analysis module through data tracking technology to establish a data feedback mechanism for each production process in the industrial production process according to the data benchmarking results and the parameter optimization results.

[0084] 705. Integrate the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module into the system platform through a data connector, so that the system platform can adjust parameters of each production process in the industrial production process based on the data feedback mechanism.

[0085] Based on the above Figure 7 The method shown in the embodiment of the present application accordingly provides a storage medium on which a computer program is stored, and when the program is executed by a processor, the above-mentioned Figure 1-Figure 5 The industrial data processing method based on the entire life cycle is shown.

[0086] Based on this understanding, the technical solution of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.), and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each implementation scenario of the present application.

[0087] Based on the above Figure 7 The method shown, and Figure 1 The embodiment of the industrial data processing system based on the whole life cycle is shown. In order to achieve the above-mentioned purpose, the embodiment of the present application also provides a physical device based on the industrial data processing of the whole life cycle, which can be a computer, a smart phone, a tablet computer, a smart watch, a server, or a network device, etc. The physical device includes a storage medium and a processor; the storage medium is used to store a computer program; the processor is used to execute the computer program to achieve the above-mentioned Figure 7 The industrial data processing method based on the entire life cycle is shown.

[0088] Optionally, the physical device may also include a user interface, a network interface, a camera, a radio frequency (RF) circuit, a sensor, an audio circuit, a WI-FI module, etc. The user interface may include a display, an input unit such as a keyboard, etc., and the optional user interface may also include a USB interface, a card reader interface, etc. The network interface may optionally include a standard wired interface, a wireless interface (such as a WI-FI interface), etc.

[0089] In an exemplary embodiment, see Figure 8 The physical device includes a communication bus, a processor, a memory and a communication interface, and may also include an input / output interface and a display device, wherein each functional unit can communicate with each other through the bus. The memory stores a computer program, and the processor is used to execute the program stored in the memory and execute the industrial data processing method based on the full life cycle in the above embodiment.

[0090] Those skilled in the art will understand that the physical device structure based on full life cycle industrial data processing provided in this embodiment does not constitute a limitation on the physical device, and may include more or fewer components, or a combination of certain components, or different component arrangements.

[0091] The storage medium may also include an operating system and a network communication module. The operating system is a program that manages the hardware and software resources of the physical device for industrial data processing based on the entire life cycle, and supports the operation of information processing programs and other software and / or programs. The network communication module is used to achieve communication between components within the storage medium, as well as communication with other hardware and software in the information processing physical device.

[0092] Through the description of the above implementation methods, those skilled in the art can clearly understand that the present application can be implemented by means of software plus necessary general hardware platforms, or by hardware. By applying the technical solution of the present application, compared with the current existing methods, the present application connects the various functional modules in an orderly manner to realize the functions of collecting, cleaning, analyzing, tracking and feedback of industrial data in the industrial production process, breaking the data isolation state between the various functional modules, so that each functional module can output high-quality industrial data, and improve the interpretability of data analysis results. The process realizes data monitoring and data optimization of each production process in the industrial production process by introducing advanced data mining and artificial intelligence technologies, which can accurately provide effective support for actual production decisions, thereby meeting the real-time monitoring and rapid response of industrial sites.

[0093] Those skilled in the art will appreciate that the accompanying drawings are only schematic diagrams of a preferred implementation scenario, and the modules or processes in the accompanying drawings are not necessarily necessary for implementing the present application. Those skilled in the art will appreciate that the modules in the devices in the implementation scenario can be distributed in the devices of the implementation scenario according to the description of the implementation scenario, or can be changed accordingly and located in one or more devices different from the present implementation scenario. The modules of the above-mentioned implementation scenario can be combined into one module, or can be further split into multiple submodules.

[0094] The above serial numbers of this application are only for description and do not represent the advantages and disadvantages of the implementation scenarios. The above disclosure is only a few specific implementation scenarios of this application, but this application is not limited to them, and any changes that can be thought of by technicians in this field should fall within the scope of protection of this application.

Claims

1. An industrial data processing system based on the entire life cycle, characterized in that: include: A data acquisition module is used to obtain real-time industrial data of each production process in the industrial production process, and the real-time industrial data has a unified data format; A data cleaning module, used to clean the real-time industrial data according to the process characteristics corresponding to each production process to obtain cleaned industrial data, wherein the cleaning process includes one or more of noise reduction, data filling, abnormal data identification and correction, and preset cleaning rules; A data analysis module, used to perform data benchmarking and parameter optimization adjustment on the cleaned industrial data to obtain data benchmarking results and parameter optimization results; A data monitoring module, used to monitor the data acquisition module, the data cleaning module and the data analysis module through data tracking technology, so as to establish a data feedback mechanism for each production process in the industrial production process according to the data benchmarking results and the parameter optimization results; A system integration module is used to integrate the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module into the system platform through a data connector, so that the system platform can adjust the parameters of each production process in the industrial production process based on the data feedback mechanism.

2. The industrial data processing system based on the entire life cycle according to claim 1 is characterized in that: The data acquisition module includes a sensor network, which is arranged in each production process in the industrial production process. The sensor network is a multi-parameter sensor constructed using the Internet of Things technology. The multi-parameter sensor includes sensors deployed in each production link in the industrial environment; The data acquisition module is specifically used to use the sensor network to collect indicator data output by heterogeneous data sources, and use standardized data interfaces and protocols to integrate the indicator data output by the heterogeneous data sources to obtain real-time industrial data of each production process in the industrial production process.

3. The industrial data processing system based on the entire life cycle according to claim 1 is characterized in that: The data cleaning module includes a combination of one or more units selected from a data noise reduction unit, a data filling unit, a data correction unit, and a custom cleaning unit; The data denoising unit is used to decompose the real-time industrial data into multiple modal functions, and to perform denoising on the real-time industrial data by filtering the modal functions whose frequencies are greater than a set value: The data filling unit is used to detect missing values ​​in the real-time industrial data, and for completely random missing data and non-critical data, the missing data is deleted according to a set time granularity, and for random missing data and non-random missing data, the missing data is filled by a random forest algorithm; The data correction unit is used to cluster the real-time industrial data using a clustering algorithm in the industrial production process, obtain the number of data points in the cluster by statistical clustering, determine the data points in the cluster whose number is less than a set threshold as data abnormal points, and use the average interpolation method to perform data correction processing on the data abnormal points; The custom cleaning unit is used to perform cleaning processing on the real-time industrial data using set cleaning rules.

4. The industrial data processing system based on the entire life cycle according to claim 1 is characterized in that: The system has a data version control function and a data storage function, and the data version control function and the data storage function support the global application of the system; The data version control function is used to record the version of the cleaned industrial data through metadata information, so that the industrial data after version recording has traceability; The data storage function is used to establish a data dictionary so as to store the version-recorded industrial data in a distributed data warehouse through the data dictionary.

5. The industrial data processing system based on the entire life cycle according to claim 1 is characterized in that: The data analysis module includes a data benchmarking unit and an optimization modeling unit; The data benchmarking unit is used to identify the difference between the cleaned industrial data and the benchmark industrial data, so as to determine the data benchmarking result through the performance data obtained by the difference identification, and the benchmark industrial data is the energy consumption data correspondingly collected when the process equipment is in a steady state in the industrial production process within a set time range; The optimization modeling unit is used to establish a process processing model for each production process in the industrial production process, so as to optimize the parameters of the cleaned industrial data through the process processing model to obtain parameter optimization results, and the parameter optimization results meet the target production conditions.

6. The industrial data processing system based on the entire life cycle according to claim 1 is characterized in that: The data monitoring module includes a data tracking unit and a data feedback unit; The data tracking unit is used to record the dynamic information of the industrial data of each production process in the data acquisition module, the data cleaning module and the data analysis module through the data tracking technology, so as to ensure the integrity of the industrial data according to the dynamic information: The data feedback unit is used to continuously adjust the production parameters of each production process in the production process based on the parameter optimization results according to the data benchmarking results, so as to establish a data feedback mechanism for each production process in the generation process through the data application results after parameter adjustment.

7. The industrial data processing system based on the entire life cycle according to any one of claims 1 to 6, characterized in that: The system integration module includes a data interaction unit and a data integration unit: The data interaction unit is used to provide a unified data interaction interface between the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module through a data connector, so as to realize data interaction between different applications according to the unified data interaction interface; The data integration unit is used to integrate the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module into the system platform through the unified data interface using a standardized interface protocol, so that the system platform can adjust parameters of each production process in the industrial production process based on the data feedback mechanism.

8. A method for processing industrial data based on the entire life cycle, characterized in that: include: The data acquisition module acquires real-time industrial data of each production process in the industrial production process, and the real-time industrial data has a unified data format; The data cleaning module cleans the real-time industrial data according to the process characteristics corresponding to each production process to obtain cleaned industrial data. The cleaning process includes one or more of noise reduction, data filling, abnormal data identification and correction, and preset cleaning rules. The data analysis module performs data benchmarking and parameter optimization adjustment on the cleaned industrial data to obtain data benchmarking results and parameter optimization results; The data monitoring module monitors the data acquisition module, the data cleaning module and the data analysis module through data tracking technology, so as to establish a data feedback mechanism for each production process in the industrial production process according to the data benchmarking result and the parameter optimization result; The system integration module integrates the data acquisition module, the data cleaning module, the data analysis module and the data monitoring module into the system platform through a data connector, so that the system platform can adjust the parameters of each production process in the industrial production process based on the data feedback mechanism.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the industrial data processing method based on the entire life cycle of claim 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the industrial data processing method based on the entire life cycle as claimed in claim 8 are implemented.

Images