An implementation method applied to a mechanical equipment big data analysis platform

Abstract

The application discloses an implementation method of a mechanical equipment big data analysis platform, which combines offline and real-time two kinds of data warehouse analysis modes, realizes data collection, synchronization and caching through tools such as Datax, Maxwell, Flume and Kafka, and utilizes Hive on Spark and Flink to analyze and calculate data. Finally, the visualization display of offline and real-time data is carried out, and comprehensive and effective support is provided for equipment operation and maintenance. The platform architecture is clear, the technical means are advanced, an efficient solution is provided for mechanical equipment big data analysis, and the problems of untimely data acquisition, low analysis accuracy and high labor cost existing in the traditional mechanical equipment monitoring and management mode are solved.

Description

Technical Field

[0001] This invention relates to the field of big data processing technology, and specifically to a method for implementing a big data analysis platform for mechanical equipment. Background Technology

[0002] With the continuous advancement of Industry 4.0 and the rise of intelligent manufacturing, machinery and equipment play a crucial role in the manufacturing sector. However, traditional methods of monitoring and managing machinery and equipment often rely on manual data collection and experience-based judgment, resulting in problems such as untimely data acquisition, low analytical accuracy, and high labor costs. With the development and widespread adoption of big data technology, utilizing machine learning, data mining, and artificial intelligence to deeply analyze and mine data from machinery and equipment has become an important means of improving equipment operating efficiency, reducing costs, and preventing malfunctions.

[0003] The big data analysis platform for mechanical equipment, as a system integrating data collection, storage, processing and analysis, was born to address the above challenges. The platform can realize real-time monitoring, anomaly warning and fault diagnosis of mechanical equipment operation data, and provide timely and effective support for equipment operation and maintenance. In the background technology of big data, several key technical points worth noting include: (1) Data collection and storage technology: The development of sensor technology, Internet of Things technology, etc. has made the collection of equipment operation data more convenient and efficient, while traditional relational databases and distributed storage systems such as Hadoop and Kafka provide reliable data storage support. (2) Data processing and analysis technology: Big data processing technologies such as Spark and Flink can process massive amounts of data and realize complex data calculation and analysis tasks, while data mining and machine learning algorithms can discover hidden patterns and rules from the data. (3) Real-time and visualization technology: With the increasing demand for real-time data processing and visualization in the industry, the application of real-time stream processing technologies such as Flink and Kafka has become increasingly important, while data visualization tools such as Superset and Spring Boot can present data to users in an intuitive form, helping users to quickly understand the data and make corresponding decisions. Summary of the Invention

[0004] 1. The technical problem to be solved: To address the aforementioned technical problems, this invention provides a method for implementing a big data analysis platform for mechanical equipment. This method effectively stores and manages industrial big data, using Hadoop and Spark for data storage and processing through offline data warehouse analysis. Real-time data warehouse analysis utilizes Flink for data processing, enabling real-time data visualization and monitoring of equipment operating parameters and status. The entire platform aims to achieve comprehensive analysis and visualization of mechanical equipment data, thereby improving data processing efficiency and accuracy.

[0005] 2. Technical Solution: A method for implementing a big data analysis platform for mechanical equipment includes the following steps: Step 1: Collect basic equipment data and its operational data; package the equipment data and its operational data by equipment and then import them into the MySQL database.

[0006] Step 2: In the MySQL database, use Datax to generate full synchronized data to Hadoop's HDFS; use Maxwell to generate incremental synchronized data to Kafka for data caching.

[0007] Step 3: When collecting logs from the device, the front-end collects front-end event tracking data and uses Flume to collect the front-end event tracking data to Kafka for data caching to generate front-end log collection data.

[0008] Step 4: Transfer the front-end log collection data and incremental synchronization data generated in Step 3 to the Hadoop HDFS distributed file system for data storage via Kafka; use Hive to build a data warehouse layered structure in HDFS, dividing the data into ODS operation data layer, DWD detailed data layer, DWS summary data layer, ADS application data layer, and DIM dimension data layer; store the constructed data warehouse layered structure in the HDFS distributed file system.

[0009] Step 5: After synchronizing the layered ADS data application layer of the data warehouse to the MySQL database via Datax, create data visualizations according to preset requirements and display the data visualizations offline via Superset.

[0010] Step Six: Retrieve data from Kafka using Flink, and use Kafka message queues to facilitate data flow between different layers of the data warehouse. After each layer of Flink completes its data computation, it rewrites the completed data into Kafka to facilitate computation in the next layer.

[0011] Step 7: Store the dimension data of the DIM dimension data layer in HBase for the association calculation between different dimensions, thus realizing the storage of dimension data.

[0012] Step 8: The calculated result data, which corresponds to the ADS layer data in the ADS application data layer, is processed and stored in ClickHouse as a real-time data stream.

[0013] Step 9: Visualize the data in real time through the Spring Boot data service interface.

[0014] Furthermore, the basic equipment data and its operational data include: equipment name, equipment model, specifications, manufacturer, production date, and equipment warranty period; the operational data include: equipment operating status, real-time operating status, equipment fault status, and equipment maintenance record data.

[0015] Furthermore, in step two, the device's operating data and basic data that were previously transferred to the MySQL database in step one are fully synchronized to the Hadoop HDFS distributed file system storage via Datax; Maxwell is used to incrementally synchronize the device's real-time operating data, device fault status, and device maintenance record data that were previously transferred to the MySQL database in step one to Kafka for data caching.

[0016] Furthermore, the offline data visualization display through Superset includes equipment status graphs, production output graphs, equipment failure rate graphs, and equipment utilization rate graphs.

[0017] Furthermore, in step six, data is retrieved from Kafka using Flink and divided into ODS operation data layer, DWD detailed data layer, DWS summary data layer, ADS application data layer, and DIM dimension data layer. Then, Kafka message queues are used to transfer data between the layers. After each layer completes its data calculation, the calculated data is rewritten to Kafka to facilitate the calculation of the next layer.

[0018] Furthermore, the big data analysis platform for mechanical equipment includes a data transmission layer, a data storage layer, a data analysis and calculation layer, and a data visualization layer; The data transmission layer collects sensor data and front-end event tracking data via Flume. Sensor data is first transmitted to MySQL, and then synchronized daily full data or daily incremental data according to different needs. Daily full data is directly synchronized to Hadoop's HDFS for storage using the Datax data synchronization tool, for subsequent offline data warehouse construction. After the offline data warehouse is built, in order to provide good support for the subsequent visualization interface, the data warehouse application layer data is synchronized back to the MySQL database via Datax. Daily incremental data is incrementally synchronized to Kafka via Maxwell for use in real-time data warehouse construction. The data storage layer stores data from the processing process according to its characteristics. The attributes of the corresponding device specifications and the data received by the sensors are stored in a MySQL database. All data in this layer is eventually cached in Kafka for later use, and finally stored in the HDFS distributed storage system. The offline data warehouse can build data warehouse partitions upon receiving data from HDFS, dividing it into ODS operation data layer, DWD detailed data layer, DWS summary data layer, ADS application data layer, and DIM dimension data layer according to different data needs. Finally, it is synchronized to the MySQL database for offline visualization. In the real-time data warehouse ODS layer, each piece of data read is placed in a Kafka topic, processed by Flink, and then rewritten into a Kafka topic as the DWD detailed data layer. Dimension table data is stored in HBase for subsequent association. Finally, the DWS summary data layer data from Kafka is consumed and aggregated with the dimension table data to generate the DWS summary data layer. The data in the DWS summary data layer is written to ClickHouse. The data analysis and computing layer is based on Spark and Flink, using Hive on Spark for offline analysis and computing and Flink for real-time analysis and computing. Hive on Spark is mainly used for offline data warehouse computing, while Flink mainly consumes Kafka data for real-time computing. After each task is submitted, Yarn requests resources from the cluster. Once resources are obtained, the platform will send the required data to the corresponding computing engine and start analysis and computing. After the computing is completed, the data is stored on disk according to the configuration of the data storage layer to complete the entire computing process. The data visualization layer is used to present the final data in a visual form; as the outermost application, it enables the platform and users to interact and provide the required data display; users can customize the historical and real-time data of interest to the big data platform for mechanical equipment to display in different forms and styles of charts and modules as required.

[0019] 3. Beneficial effects: (1) The method disclosed herein is an implementation method for a big data analysis platform for mechanical equipment. It adopts both offline and real-time data warehouse analysis methods, which can more comprehensively process and analyze equipment data.

[0020] (2) The method disclosed herein is an implementation method for a big data analysis platform for mechanical equipment. It uses Datax and Maxwell for data synchronization, Flume for front-end data acquisition, and Kafka for data transmission and caching. These advanced technologies make data processing more efficient.

[0021] (3) The method disclosed in this paper is an implementation method for a big data analysis platform for mechanical equipment. It also uses Hive and Spark for data storage and computation, thereby improving computational efficiency, saving resources, improving system stability, and improving data processing capabilities.

[0022] (4) This method discloses an implementation method for a big data analysis platform for mechanical equipment, which uses Superset for data visualization. These tools not only improve the efficiency of data processing, but also make the data analysis results easier to understand and use. This is relevant to real-time data warehouse analysis.

[0023] (5) The method disclosed herein is an implementation method for a big data analysis platform for mechanical equipment. It uses Flink for data acquisition and processing, Kafka for data flow, and HBase and Redis for dimensional data storage. The application of these technologies makes real-time data processing possible.

[0024] (6) The method disclosed herein is an implementation method for a big data analysis platform for mechanical equipment. The processed data is stored in ClickHouse and the data is visualized in real time through the SpringBoot data service interface. This allows the data analysis results to be presented to the user intuitively, thereby providing strong support for the operation and maintenance of the equipment. Attached Figure Description

[0025] Figure 1 This is a system framework diagram of a big data analysis platform for mechanical equipment according to the present invention; Figure 2 This is a flowchart illustrating a method for implementing a big data analysis platform for mechanical equipment according to the present invention; Detailed Implementation

[0026] The present invention will now be described in detail with reference to the accompanying drawings.

[0027] A method for implementing a big data analysis platform for mechanical equipment, characterized by the following steps: Step 1: Collect basic equipment data and its operational data; package the equipment data and its operational data by equipment and then import them into the MySQL database.

[0028] Step 2: In the MySQL database, use Datax to generate full synchronized data to Hadoop's HDFS; use Maxwell to generate incremental synchronized data to Kafka for data caching.

[0029] Step 3: When collecting logs from the device, the front-end collects front-end event tracking data and uses Flume to collect the front-end event tracking data to Kafka for data caching to generate front-end log collection data.

[0030] Step 4: Transfer the front-end log collection data and incremental synchronization data generated in Step 3 to the Hadoop HDFS distributed file system for data storage via Kafka; use Hive to build a data warehouse layered structure in HDFS, dividing the data into ODS operation data layer, DWD detailed data layer, DWS summary data layer, ADS application data layer, and DIM dimension data layer; store the constructed data warehouse layered structure in the HDFS distributed file system.

[0031] Step 5: After synchronizing the layered ADS data application layer of the data warehouse to the MySQL database via Datax, create data visualizations according to preset requirements and display the data visualizations offline via Superset.

[0032] Step Six: Retrieve data from Kafka using Flink, and use Kafka message queues to facilitate data flow between different layers of the data warehouse. After each layer of Flink completes its data computation, it rewrites the completed data into Kafka to facilitate computation in the next layer.

[0033] Step 7: Store the dimension data of the DIM dimension data layer in HBase for the association calculation between different dimensions, thus realizing the storage of dimension data.

[0034] Step 8: The calculated result data, which corresponds to the ADS layer data in the ADS application data layer, is processed and stored in ClickHouse as a real-time data stream.

[0035] Step 9: Visualize the data in real time through the Spring Boot data service interface.

[0036] Furthermore, the basic equipment data and its operational data include: equipment name, equipment model, specifications, manufacturer, production date, and equipment warranty period; the operational data include: equipment operating status, real-time operating status, equipment fault status, and equipment maintenance record data.

[0037] Furthermore, in step two, the device's operating data and basic data that were previously transferred to the MySQL database in step one are fully synchronized to the Hadoop HDFS distributed file system storage via Datax; Maxwell is used to incrementally synchronize the device's real-time operating data, device fault status, and device maintenance record data that were previously transferred to the MySQL database in step one to Kafka for data caching.

[0038] Furthermore, the offline data visualization display through Superset includes equipment status graphs, production output graphs, equipment failure rate graphs, and equipment utilization rate graphs.

[0039] Furthermore, in step six, data is retrieved from Kafka using Flink and divided into ODS operation data layer, DWD detailed data layer, DWS summary data layer, ADS application data layer, and DIM dimension data layer. Then, Kafka message queues are used to transfer data between the layers. After each layer completes its data calculation, the calculated data is rewritten to Kafka to facilitate the calculation of the next layer.

[0040] Furthermore, the big data analysis platform for mechanical equipment includes a data transmission layer, a data storage layer, a data analysis and calculation layer, and a data visualization layer; The data transmission layer collects sensor data and front-end event tracking data via Flume. Sensor data is first transmitted to MySQL, and then synchronized daily full data or daily incremental data according to different needs. Daily full data is directly synchronized to Hadoop's HDFS for storage using the Datax data synchronization tool, for subsequent offline data warehouse construction. After the offline data warehouse is built, in order to provide good support for the subsequent visualization interface, the data warehouse application layer data is synchronized back to the MySQL database via Datax. Daily incremental data is incrementally synchronized to Kafka via Maxwell for use in real-time data warehouse construction. The data storage layer stores data from the processing process according to its characteristics. The attributes of the corresponding device specifications and the data received by the sensors are stored in a MySQL database. All data in this layer is eventually cached in Kafka for later use, and finally stored in the HDFS distributed storage system. The offline data warehouse can build data warehouse partitions upon receiving data from HDFS, dividing it into ODS operation data layer, DWD detailed data layer, DWS summary data layer, ADS application data layer, and DIM dimension data layer according to different data needs. Finally, it is synchronized to the MySQL database for offline visualization. In the real-time data warehouse ODS layer, each piece of data read is placed in a Kafka topic, processed by Flink, and then rewritten into a Kafka topic as the DWD detailed data layer. Dimension table data is stored in HBase for subsequent association. Finally, the DWS summary data layer data from Kafka is consumed and aggregated with the dimension table data to generate the DWS summary data layer. The data in the DWS summary data layer is written to ClickHouse. The data analysis and computing layer is based on Spark and Flink, using Hive on Spark for offline analysis and computing and Flink for real-time analysis and computing. Hive on Spark is mainly used for offline data warehouse computing, while Flink mainly consumes Kafka data for real-time computing. After each task is submitted, Yarn requests resources from the cluster. Once resources are obtained, the platform will send the required data to the corresponding computing engine and start analysis and computing. After the computing is completed, the data is stored on disk according to the configuration of the data storage layer to complete the entire computing process. The data visualization layer is used to present the final data in a visual form; as the outermost application, it enables the platform and users to interact and provide the required data display; users can customize the historical and real-time data of interest to the big data platform for mechanical equipment to display in different forms and styles of charts and modules as required. Specific implementation examples: As attached Figure 2The diagram illustrates the workflow of the entire big data analytics platform for mechanical equipment. This platform is primarily divided into offline data warehouse analysis and real-time data warehouse analysis. The offline data warehouse collects equipment operation data, as well as equipment model, specifications, and other information, and imports it into a MySQL database. On one hand, it uses Datax to fully synchronize data to Hadoop's HDFS for storage; on the other hand, it uses Maxwell to incrementally synchronize data to Kafka for data caching. Then, it collects front-end operation data (such as data from device tracking points) via Flume and caches it in Kafka. Finally, it transmits front-end log data and incrementally synchronized data to HDFS via Kafka for data storage. Hive is used to layer the data in HDFS into ODS (Operational Data Layer), DWD (Detailed Data Layer), DWS (Summary Data Layer), ADS (Application Data Layer), and DIM (Dimensional Data Layer). The Spark computing engine replaces Hive's default MapReduce. Finally, the ADS data application layer is output and synchronized to the MySQL database via Datax, and Superset is used for offline data visualization. The real-time data warehouse retrieves data from Kafka via Flink and divides the data into ODS (Operational Data Layer), DWD (Detailed Data Layer), DWS (Summary Data Layer), ADS (Application Data Layer), and DIM (Dimensional Data Layer). Since this system primarily focuses on real-time data processing, data persistence to disk is not required. Data flow between layers is handled by Kafka message queues. After each layer completes its calculations, the processed data is rewritten to Kafka for the next layer's computation. The DIM layer, due to the infrequent changes in dimensional data, stores it in HBase for inter-dimensional joins. In this invention, the calculated results—the ADS layer data in the data processing module—are processed and stored in ClickHouse. Depending on the specific needs, the required data can be directly queried from ClickHouse or further processed and applied through table joins. Dimensional data storage is implemented using HBase and Redis. Finally, real-time data visualization is achieved through a Spring Boot data service interface.

[0042] As attached Figure 1 The diagram shows the architecture of a big data analysis platform for mechanical equipment according to the present invention. The platform's architecture uses the classic Hadoop big data platform architecture plus a configurable big data visualization interface, as shown in the figure. The platform can be divided into four layers: a data transmission layer, a data storage layer, a data computation and analysis layer, and a data visualization layer.

[0043] Although the present invention has been disclosed above with reference to preferred embodiments, these are not intended to limit the invention. Any person skilled in the art can make various changes or modifications without departing from the spirit and scope of the invention. Therefore, the scope of protection of the present invention should be defined by the scope of the claims of this application.

Claims

1. A method for implementing a big data analysis platform for mechanical equipment, characterized in that: Includes the following steps: Step 1: Collect basic equipment data and its operational data; package the equipment data and operational data by device and import them into the MySQL database; Step 2: In the MySQL database, use Datax to generate full synchronized data to Hadoop's HDFS; use Maxwell to generate incremental synchronized data to Kafka for data caching. Step 3: When collecting logs from the device, the front-end collects front-end event tracking data. Flume is used to collect the front-end event tracking data and cache it in Kafka to generate front-end log collection data. Step 4: The front-end log collection data generated in Step 3, along with incrementally synchronized data, is transmitted via Kafka to the Hadoop HDFS distributed file system for data storage. Hive is used to build a layered data warehouse in HDFS, dividing the data into ODS operation data layer, DWD detailed data layer, DWS summary data layer, ADS application data layer, and DIM dimension data layer. The constructed layered data warehouse is then stored in the HDFS distributed file system. Step 5: After synchronizing the layered ADS data application layer of the data warehouse to the MySQL database via Datax, create data visualizations according to preset requirements and display the offline data visualizations via Superset; Step 6: Retrieve data from Kafka using Flink, and use Kafka message queues to handle data flow between different layers of the data warehouse. After each layer of Flink completes its data computation, it rewrites the completed data into Kafka to facilitate computation in the next layer. Step 7: Store the dimension data of the DIM dimension data layer in HBase for the association calculation between different dimensions, thus realizing the storage of dimension data; Step 8: The calculated result data, which corresponds to the ADS layer data in the ADS application data layer, is processed and stored in ClickHouse as a real-time data stream; Step 9: Visualize the data in real time through the Spring Boot data service interface.

2. The implementation method of a big data analysis platform for mechanical equipment according to claim 1, characterized in that: The basic equipment data and its operational data include: equipment name, equipment model, specifications, manufacturer, production date, and equipment warranty period; the operational data includes: equipment operating status, real-time operating status, equipment fault status, and equipment maintenance records.

3. The implementation method of a big data analysis platform for mechanical equipment according to claim 2, characterized in that: In step two, the device's operating data and basic data that were previously transferred to the MySQL database in step one are fully synchronized to the Hadoop HDFS distributed file system storage via Datax; Maxwell is then used to incrementally synchronize the device's real-time operating data, device fault status, and device maintenance records that were previously transferred to the MySQL database in step one to Kafka for data caching.

4. The implementation method of a big data analysis platform for mechanical equipment according to claim 3, characterized in that: The offline data visualization display via Superset includes equipment status graphs, production output graphs, equipment failure rate graphs, and equipment utilization rate graphs.

5. The implementation method of a big data analysis platform for mechanical equipment according to claim 4, characterized in that: In step six, data is retrieved from Kafka using Flink and divided into four layers: ODS operation data layer, DWD detailed data layer, DWS summary data layer, ADS application data layer, and DIM dimension data layer. Then, Kafka message queues are used to transfer data between the layers. After each layer completes its data calculation, the calculated data is rewritten to Kafka to facilitate the calculation of the next layer.

6. The implementation method of a big data analysis platform for mechanical equipment according to claim 1, characterized in that: The big data analysis platform for mechanical equipment includes a data transmission layer, a data storage layer, a data analysis and calculation layer, and a data visualization layer. The data transmission layer collects sensor data and front-end event tracking data via Flume. Sensor data is first transmitted to MySQL, and then synchronized daily full data or daily incremental data according to different needs. Daily full data is directly synchronized to Hadoop's HDFS for storage using the Datax data synchronization tool, for subsequent offline data warehouse construction. After the offline data warehouse is built, in order to provide good support for the subsequent visualization interface, the data warehouse application layer data is synchronized back to the MySQL database via Datax. Daily incremental data is incrementally synchronized to Kafka via Maxwell for use in real-time data warehouse construction. The data storage layer stores data from the processing process according to its characteristics. The attributes of the corresponding device specifications and the data received by the sensors are stored in a MySQL database. All data in this layer is eventually cached in Kafka for later use, and finally stored in the HDFS distributed storage system. The offline data warehouse can build data warehouse partitions upon receiving data from HDFS, dividing it into ODS operation data layer, DWD detailed data layer, DWS summary data layer, ADS application data layer, and DIM dimension data layer according to different data needs. Finally, it is synchronized to the MySQL database for offline visualization. In the real-time data warehouse ODS layer, each piece of data read is placed in a Kafka topic, processed by Flink, and then rewritten into a Kafka topic as the DWD detailed data layer. Dimension table data is stored in HBase for subsequent association. Finally, the DWS summary data layer data from Kafka is consumed and aggregated with the dimension table data to generate the DWS summary data layer. The data in the DWS summary data layer is written to ClickHouse. The data analysis and computing layer is based on Spark and Flink, using Hive on Spark for offline analysis and computing and Flink for real-time analysis and computing. Hive on Spark is mainly used for offline data warehouse computing, while Flink mainly consumes Kafka data for real-time computing. After each task is submitted, Yarn requests resources from the cluster. Once resources are obtained, the platform will send the required data to the corresponding computing engine and start analysis and computing. After the computing is completed, the data is stored on disk according to the configuration of the data storage layer to complete the entire computing process. The data visualization layer is used to present the final data in a visual form. As the outermost application, it enables the platform and users to interact and provide the necessary data display; users can customize the historical and real-time data that the big data platform for mechanical equipment focuses on to create different forms and styles of charts and modules for display.

Images