A method and system for integrating and analyzing data of the entire process industry manufacturing chain

Abstract

The present application discloses a method and system for data integration and analysis of the entire chain of process industry manufacturing, the method comprising: obtaining data from various business application systems and hardware devices in the process industry; classifying and aggregating the acquired data according to different subject businesses, wherein each type of classified and aggregated data corresponds to a subject business; receiving data analysis requirements; obtaining the classified and aggregated data required for the data analysis requirements according to the target of the data analysis requirements, and analyzing the acquired classified and aggregated data to obtain the target result. The present application solves the problem in the prior art caused by the mutual independence of systems between different enterprises in the entire chain of process industry data, thereby realizing cross-domain integration of multi-source heterogeneous data in the process industry and dynamic perception of the entire manufacturing chain.

Description

Technical Field

[0001] The present application relates to the field of intelligent manufacturing, and specifically, to a method and system for integrating and analyzing data of the entire process industrial manufacturing chain. Background Art

[0002] Process industry is an important industry. With the development of the Internet, process industry enterprises are also rapidly developing in the direction of informatization, forming many information systems with unified standards, such as ERP, MES, and DCS, to meet the basic application needs of enterprises in process control, production management, financial control, etc.

[0003] However, the systems of different enterprises in the entire chain of the same process industry are independent of each other, with inconsistent data formats, mismatched time and space, inconsistent models, unintegrated information, and lack of sharing. As a result, the large amount of data accumulated and stored in the enterprise informatization construction is not fully utilized, and there is a lack of deeper data mining and application, and it cannot be directly used for enterprise intelligent decision-making in the entire manufacturing chain. Summary of the invention

[0004] The embodiments of the present application provide a method and system for integrating and analyzing data of the entire chain of process industry manufacturing, so as to at least solve the problem in the prior art caused by the mutual independence of the systems of different enterprises in the entire chain of process industry data.

[0005] According to one aspect of the present application, a method for data integration and analysis of the entire chain of process industry manufacturing is provided, including: acquiring data from various business application systems and hardware devices in the process industry industry, wherein the various business application systems are systems that provide software support for the process industry industry, and the hardware devices are production equipment in the process industry industry; classifying and aggregating the acquired data according to different subject businesses, wherein each type of classified and aggregated data corresponds to a subject business; receiving data analysis requirements; acquiring the classified and aggregated data required for the data analysis requirements according to the objectives of the data analysis requirements, and analyzing the acquired classified and aggregated data to obtain target results.

[0006] Furthermore, acquiring data from various business application systems and hardware devices in the process industry includes: acquiring real-time data in the process industry; and acquiring historical data in the process industry.

[0007] Furthermore, acquiring the historical data in the process industry includes: acquiring the historical data from an enterprise relational database and / or log in the process industry.

[0008] Furthermore, the target result includes at least one of the following of the products in the process industry: quality, energy consumption, material consumption, price, and supply and demand.

[0009] Furthermore, analyzing the acquired classified and summarized data according to the target of the data analysis requirement to obtain the target result includes: selecting a corresponding neural network-based machine learning model according to the target result to be analyzed, wherein the machine learning model is trained based on the acquired classified and summarized data, and the machine learning model outputs the target result; calling the machine learning model to obtain the target result according to the data to be analyzed carried in the data analysis requirement.

[0010] According to another aspect of the present application, a full-chain data integration and analysis system for process industry manufacturing is also provided, including: a data acquisition module, used to obtain data from various business application systems and hardware equipment in the process industry industry, wherein the various business application systems are systems that provide software support for the process industry industry, and the hardware equipment is production equipment in the process industry industry; a data integration module, used to classify and summarize the acquired data according to different theme businesses, wherein each type of classified and summarized data corresponds to a theme business; a data analysis module, used to receive data analysis requirements; according to the objectives of the data analysis requirements, the classified and summarized data required for the data analysis requirements are acquired, and the acquired classified and summarized data are analyzed to obtain the target results.

[0011] Furthermore, the data acquisition module is used to: obtain real-time data in the process industry; obtain historical data in the process industry.

[0012] Furthermore, the data acquisition module is used to obtain the historical data from an enterprise relational database and / or log in the process industry.

[0013] Furthermore, the target result includes at least one of the following of the products in the process industry: quality, energy consumption, material consumption, price, and supply and demand.

[0014] Furthermore, the data analysis module is used to: select a corresponding neural network-based machine learning model according to the target result to be analyzed, wherein the machine learning model is trained based on the acquired classified summary data, and the machine learning model outputs the target result; call the machine learning model to obtain the target result according to the data to be analyzed carried in the data analysis requirements.

[0015] In the embodiment of the present application, the data in various business application systems and hardware devices in the process industry are obtained, wherein each business application system is a system that provides software support for the process industry, and the hardware device is the production equipment in the process industry; the obtained data is classified and summarized according to different subject businesses, wherein each classified and summarized data corresponds to a subject business; data analysis requirements are received; according to the target of the data analysis requirement, the classified and summarized data required for the data analysis requirement is obtained, and the obtained classified and summarized data is analyzed to obtain the target result. This application solves the problem caused by the mutual independence of the systems of different enterprises in the whole chain of process industry in the prior art, thereby realizing the cross-domain integration of multi-source heterogeneous data in the process industry and the dynamic perception of the whole manufacturing chain. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0017] Figure 1 It is a flow chart of a method for integrating and analyzing data of the entire process industrial manufacturing chain according to an embodiment of the present application;

[0018] Figure 2 It is a structural diagram of a process industry manufacturing full-chain data integration and analysis system according to an embodiment of the present application. DETAILED DESCRIPTION

[0019] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0020] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0021] In this embodiment, a method for integrating and analyzing data of the entire process industry manufacturing chain is provided. Figure 1 is a flowchart of a method for integrating and analyzing data of the entire process industry manufacturing chain according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0022] Step S102, acquiring data from various business application systems and hardware devices in the process industry, wherein each business application system is a system that provides software support for the process industry, and the hardware device is a production equipment in the process industry;

[0023] In this step, real-time data and / or historical data in the process industry are obtained. Optionally, obtaining the historical data in the process industry includes: obtaining the historical data from an enterprise relational database and / or log in the process industry.

[0024] Step S104, classifying and aggregating the acquired data according to different subject businesses, wherein each type of classified and aggregated data corresponds to a subject business;

[0025] Step S106, receiving data analysis requirements;

[0026] Step S108, acquiring the classified and summarized data required by the data analysis requirement according to the target of the data analysis requirement, and analyzing the acquired classified and summarized data to obtain the target result.

[0027] In this step, the target result includes at least one of the following of the products in the process industry: quality, energy consumption, material consumption, price, and supply and demand.

[0028] In this step, the target result can be obtained through machine learning. For example, a corresponding neural network-based machine learning model is selected according to the target result that needs to be analyzed, wherein the machine learning model is trained based on the acquired classified summary data, and the machine learning model outputs the target result; the machine learning model is called to obtain the target result according to the data to be analyzed carried in the data analysis requirements.

[0029] This is illustrated below with an example. In this example, real-time production data of the production equipment is obtained, and the target result currently desired is the quality change of the product based on the production data. The production data is input into the corresponding machine learning model, and the machine learning model outputs the quality change of the product, wherein the machine learning model is obtained based on historical data training, the production data in the historical data is used as input and output, the quality change corresponding to the production data is used as output data, a pair of input data and output data is used as a set of training data, and the machine learning model can be obtained by training with multiple sets of the training data.

[0030] The above steps solve the problem in the prior art that the data of the entire chain of the process industry is independent of each other in the systems of different enterprises, thereby realizing the cross-domain integration of multi-source heterogeneous data in the process industry and the dynamic perception of the entire manufacturing chain.

[0031] The following is a description of an optional embodiment. In this optional embodiment, a process industry manufacturing full-chain data integration and analysis system is provided, including:

[0032] The data collection module extracts data from various business application systems and production equipment in the process industry cluster area to achieve preliminary aggregation and real-time update of basic data in the data warehouse and database. Optionally, the collection of historical offline data is generally extracted from the enterprise's relational database and business log; the collection of real-time production data is to collect key process data that affects upper-level decision-making.

[0033] The data integration module designs the ODS layer, DWD layer, and DWS layer in the data warehouse, and then designs the DWM layer to organize the data into storage structures for different subject business dimensions based on OLAP. Optionally, the ODS data access layer basically maintains the same format as the data source, does not perform excessive verification, and ensures that there is no data missing; the DWD data detail layer performs data cleaning and conversion, dimension completion, and standardization.

[0034] The DWS data aggregation layer builds statistical indicators with consistent naming standards and caliber based on the analysis of upper-layer application themes.

[0035] The DWM data mart layer aggregates relevant data for specific application topics.

[0036] The data analysis module performs feature extraction and correlation analysis on the data required for upper-level decision-making analysis; based on deep neural networks, it conducts model training for intelligent prediction and intelligent perception of product quality, energy consumption, material consumption, price, supply and demand in the industry, including data preprocessing, model training, model verification and testing.

[0037] The data interaction module uses three different types of servers to build the background for security and scalability, and implements user management, authority management, security management, log management, and function management; for example, the server may include at least one of the following: a logic processing server, a static processing server, and a dynamic forwarding server.

[0038] The task scheduling system uses open source task scheduling tools to design complete task scheduling management for real-time and offline collection, integration, analysis, and interaction of data warehouses to ensure high-performance operation of the system.

[0039] In this embodiment, in order to further manage the system better, at least one of the following modules may be included:

[0040] The cluster management module provides cluster services for the system background distributed computer cluster.

[0041] The permission management module sets system access rights not only for users at the system application layer, but also for system development and maintenance personnel to ensure system security.

[0042] The log management module classifies different events of the system into different levels and saves or prompts them in log form to different degrees.

[0043] The present embodiment is described below with reference to the accompanying drawings. Figure 2 This is a schematic diagram of a process industry manufacturing full-chain data integration and analysis system according to an embodiment of the present application, referring to Figure 2 ,The system is generally divided into six parts: data source, data collection, data integration, data analysis, data interaction, and system application.

[0044] Data source: System data comes from upstream and downstream enterprises in the process industry. Data types include real-time updated data in hardware equipment during the production process of the upstream, midstream and downstream industries, and production business data in ERP, CRM, EAM, SCM and other application systems.

[0045] Data collection: Based on the OPC-UA protocol, real-time data of hardware devices can be directly read; using Sqoop and Flume open source data extraction tools, data from enterprise relational databases and enterprise business logs can be imported into the HDFS file storage system respectively.

[0046] Data integration: After data collection, a complete system warehouse should be built. Data integration is divided into four layers: ODS data access layer, for data synchronization, basically keeping the format consistent with the source data to ensure that there is no data missing; DWD layer data detail layer, for data cleaning and conversion, incomplete dimensions, standardization, exception processing, etc.; DWS data summary layer, providing a subject-oriented, complete historical description for the data source; DWM data mart layer, for intelligent prediction and intelligent perception of product quality, energy consumption, material consumption, price, supply and demand in the industry.

[0047] Specifically, the data integration and storage processes are based on the distributed architecture of the Hadoop open source ecosystem. The collected data is stored in HDFS. The background computing engine uses Hive for offline processing and Flink for real-time processing. The processed upper-level data is stored in MySQL and cached through Redis and MangoDB for external access.

[0048] Optionally, the above data integration is based on the Hadoop distributed architecture, and a cluster management module can be developed to provide cluster management for cluster servers composed of multiple high-performance computers in the system.

[0049] Data analysis: The analysis process is developed based on Python language and uses open source machine learning algorithm libraries. The analysis process includes:

[0050] Use principal component analysis, autoencoder and other methods to reduce the dimension of the data and extract features, and then perform association analysis based on Aprior, FP-G and other algorithms.

[0051] Based on deep neural networks, model training is carried out for intelligent prediction and intelligent perception of product quality, energy consumption, material consumption, price, supply and demand in the industry, including data preprocessing, model training, model verification and testing.

[0052] Specifically, when time series-related indicators such as energy consumption, material consumption, and price are used as input data, the variational mode decomposition (VMD) signal decomposition method is used to decompose the time series of the input data into feature time series, including trend time series (characterizing the long-term change trend of the input data), periodic time series (characterizing the periodic change trend of the input data over time), and fluctuation / noise time series (characterizing the unpredictable fluctuations of the input data caused by various emergencies); secondly, the recurrent neural network (RNN), long short-term memory artificial neural network (LSTM), gated learning unit (GRU), Transformer model and other methods are used to learn the implicit features in different sequences, and the AIC information criterion, BIC information criterion, L1 regularization, L2 regularization, Dropout, Bayesian method and other methods are used to balance the model training effect and prediction effect to prevent the model from overfitting; the prediction model is trained separately based on each set of feature sequences obtained by decomposition, and the model output is weighted to obtain the final prediction result of the input data.

[0053] When indicators such as supply and demand that directly affect corporate decision-making are used as input data, the basic influencing factors that affect the input data are first selected, and then deep neural networks and other methods are used to explore the deep-level characteristics and change modes of the basic influencing factors. Heuristic algorithms and AIC information criteria are used to improve model training efficiency, and finally an intelligent perception model is obtained to predict changes in input data trends, thereby guiding corporate business decisions.

[0054] Data interaction: Use Django+Nginx+uWSGI to build a Web server to provide API interface for the application layer, including user management, permission management, security management, log management, function management, etc.

[0055] For example, Django, as a logical interface server, provides logical operations of application function interfaces; Nginx, as a static processing server, can act as a reverse proxy and play a role in load balancing; uWSGI, as a dynamic forwarding server, handles static requests at the application layer directly by Nginx, and dynamic requests are forwarded to Django after being transferred by uWSGI for processing.

[0056] System application: The front-end application realizes complex functions through the combination of different algorithm components. It is divided into three major sections: collaborative design, collaborative manufacturing, and collaborative services, covering the vast majority of application scenarios of process industrial enterprises.

[0057] The task scheduling system uses the open source scheduling tool AirFlow to develop a complete task scheduling system for the system data collection module, integration module, analysis module, and interaction module to ensure efficient and automated operation of the system.

[0058] Optionally, based on the above embodiment, a rights management module is further included to set system access rights not only for users of the system application layer but also for system development and maintenance personnel to ensure system security.

[0059] Optionally, based on the above embodiment, a log management module is also included to classify different events of the system into different levels and manage them in the form of file output, screen output, email reminder, etc.

[0060] Based on the above specific implementation, the overall design of the system tries to adopt a low-coupling and high-cohesion approach to develop cross-domain data sharing, cross-domain management of manufacturing resources, and cross-domain integration and analysis systems for the entire chain of process industry manufacturing, so that the system can basically meet the application of most standardized enterprises in the process industry chain.

[0061] In this embodiment, an electronic device is provided, including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to execute the method in the above embodiment.

[0062] The above program can be run in the processor, or it can be stored in the memory (or computer-readable medium), which includes permanent and non-permanent, removable and non-removable media. Information storage can be achieved by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined in this article, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0063] These computer programs can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps of the functions specified in one or more blocks can be implemented by different modules corresponding to different steps.

[0064] Such a device or system is provided in this embodiment. The system is called the process industry manufacturing full-chain data integration and analysis system, including: a data acquisition module, used to obtain data from various business application systems and hardware equipment in the process industry, wherein each business application system is a system that provides software support for the process industry, and the hardware equipment is the production equipment in the process industry; a data integration module, used to classify and summarize the acquired data according to different subject businesses, wherein each type of classified and summarized data corresponds to a subject business; a data analysis module, used to receive data analysis requirements; according to the target of the data analysis requirements, the classified and summarized data required for the data analysis requirements are obtained, and the acquired classified and summarized data are analyzed to obtain the target results.

[0065] The system or device is used to implement the functions of the method in the above-mentioned embodiment. Each module in the system or device corresponds to each step in the method, which has been explained in the method and will not be repeated here.

[0066] For example, the data acquisition module is used to obtain real-time data in the process industry and historical data in the process industry. Optionally, the data acquisition module is used to obtain historical data from an enterprise relational database and / or log in the process industry.

[0067] For another example, the target result includes at least one of the following of products in the process industry: quality, energy consumption, material consumption, price, supply and demand. Optionally, the data analysis module is used to: select a corresponding machine learning model based on a neural network according to the target result to be analyzed, wherein the machine learning model is trained based on the acquired classified summary data, and the machine learning model outputs the target result; call the machine learning model to obtain the target result according to the data to be analyzed carried in the data analysis requirement.

[0068] Through the above embodiments, a full-chain integration and analysis system for process industry manufacturing is designed, and a data warehouse is built based on open source distributed big data storage and computing tools, including: a data acquisition module, which extracts data from various business application systems and production equipment in process industry industrial clusters for unified storage; a data integration module, which designs ODS layer, DWD layer, DWS layer, and DWM layer in the data warehouse; a data analysis module, which extracts features and performs correlation analysis on data, and performs model training on issues such as intelligent prediction and intelligent perception in industrial intelligence, so as to provide decision-making for the upper level of the enterprise; a data interaction module, which uses three different types of Web servers in combination to provide more efficient application services for the upper level from the perspective of security and scalability; in addition, a complete task scheduling system is designed for real-time and offline collection, integration, analysis, and interaction of the data warehouse to ensure high-performance operation of the system.

[0069] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A method for data integration and analysis of the entire process industry manufacturing chain, characterized in that: include: Acquire data from various business application systems and hardware devices in the process industry, wherein the various business application systems are systems that provide software support for the process industry, and the hardware devices are production equipment in the process industry; The acquired data is classified and summarized according to different subject businesses, wherein each classified and summarized data corresponds to a subject business; Receive data analysis requirements; According to the target of the data analysis demand, the classified and summarized data required by the data analysis demand are obtained, and the obtained classified and summarized data are analyzed to obtain the target result; the target result includes at least one of the following products in the process industry: quality, energy consumption, material consumption, price, supply and demand; wherein, according to the target result to be analyzed, a corresponding machine learning model based on a neural network is selected, wherein the machine learning model is trained based on the obtained classified and summarized data, and the machine learning model outputs the target result; the machine learning model is called to obtain the target result according to the data to be analyzed carried in the data analysis demand; based on the deep neural network, the model is trained for the intelligent prediction and intelligent perception of product quality, energy consumption, material consumption, price and supply and demand in the industry; wherein, for indicators related to time series as input data, variational mode decomposition signal analysis is used The solution method decomposes the time series of input data into feature time series, including: trend time series, periodic time series, and fluctuation and noise time series; wherein, the trend time series is used to characterize the long-term change trend of the input data, the periodic time series is used to characterize the periodic change trend of the input data over time, and the fluctuation and noise time series is used to characterize the unpredictable fluctuations of the input data caused by various emergencies; learn the implicit features in different sequences; train the prediction model separately based on each set of feature sequences obtained by decomposition, and weight the model output to obtain the final prediction result of the input data; wherein, for indicators that directly affect corporate decisions as input data, first select the basic influencing factors that affect the input data, and then use a deep neural network to mine the deep-level features and change modes in the basic influencing factors to obtain an intelligent perception model to predict changes in the input data trend.

2. The method according to claim 1, characterized in that Acquiring data from various business application systems and hardware devices in the process industry, including: Acquire real-time data in the process industry; Obtain historical data for the process industry.

3. The method according to claim 2, characterized in that Obtaining historical data in the process industry includes: The historical data is obtained from an enterprise relational database and / or log in the process industry.

Images