Big data monitoring analysis method
By designing big data monitoring and analysis methods, including node gateway control, data preprocessing and long-term and short-term network model construction, the accuracy and inefficiency of equipment condition judgment and fault prediction are solved, and efficient fault prediction and early warning notification are achieved.
Patent Information
- Application Number
- CN202411183526.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-27
- Publication Date
- 2025-02-25
AI Technical Summary
The prior art lacks effective methods for equipment condition judgment and fault prediction, resulting in inaccuracy and inefficiency of fault prediction analysis.
A big data monitoring and analysis method is designed, including node gateway control, data preprocessing and transmission control, analysis control processing, comparison management and transmission display. This method collects data through the wireless sensor network layer, performs clustering processing and feature extraction, builds a long-term and short-term network model, and performs fault prediction and early warning notification.
Through this method, the working analysis efficiency of nodes and gateways can be effectively improved, the electrical data can be converted into real electrical parameter data, management efficiency can be improved, analysis and processing errors can be reduced, and the accuracy and efficiency of fault prediction can be improved.
Smart Images
Figure CN119521154A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of big data, and in particular to a monitoring and analysis method for big data. Background Art
[0002] With the rapid development of information technology and automation technology, the integration and complexity of modern industrial systems are becoming increasingly higher, and they are outstanding in the fields of aviation, aerospace, communications, semiconductors, high-speed rail transportation, metallurgy and chemical industry. As the components of artificial systems continue to increase, the mutual influence between the parts is becoming more and more complex, resulting in a gradual increase in the probability of system failure and functional failure. Equipment failure prediction is an important part of fault diagnosis technology. It means that by analyzing the historical and current equipment failure characteristic values, the future fault characteristic values are predicted, so as to predict the operating status of the equipment in the future period of time, predict the possible failure of the equipment, and judge the equipment failure level based on these characteristic values, grasp the development trend of equipment failure in advance, and provide a basis for early prevention and repair of failures. However, there is currently a lack of analytical data for judging the equipment status, which seriously affects the prediction and analysis of faults. Therefore, it is necessary to design a big data monitoring and analysis method with efficient management analysis and high accuracy. Summary of the invention
[0003] The purpose of the present invention is to provide a big data monitoring and analysis method to solve the problems raised in the above background technology.
[0004] In order to solve the above technical problems, the present invention provides the following technical solutions: a big data monitoring and analysis method, comprising: Conduct node gateway control of industrial monitoring and analysis systems; Carry out data preprocessing and transmission control for industrial monitoring and analysis systems; Conduct analysis and control processing of industrial monitoring and analysis systems; Conduct comparative management and transmission display of industrial monitoring and analysis systems; The node gateway control of the industrial monitoring and analysis system includes: When the wireless node in the industrial monitoring and analysis system starts working, it first performs a network access operation. The node initiates a network access request. When the gateway receives the network access request data packet from the wireless node, it will determine the type and subtype of the data packet. When it is determined from the type and subtype of the data packet that the data packet is a network access request data packet, the data packet will be sent to the server. When the server receives the network access request, it responds with a data packet to the gateway. The gateway sends the network access response packet to the destination wireless node based on the information in the data packet. When the wireless node receives the network access response packet, it sends an ACK data packet again to complete the network access process.
[0005] According to the above technical solution, the data preprocessing and transmission control of the industrial monitoring and analysis system includes: The original data collected by the data acquisition card in the sensor node in the wireless sensor network layer is made into a hexadecimal number with two bytes as the unit, and the data is controlled to be sent to the gateway at every set collection interval. The gateway aggregates the data of multiple nodes and sends them to the server, and the server processes the data; The device collection data saved by the wireless sensor network layer is stored in MySQL, and the persistent data part located in the wireless sensor network layer server is imported into HBase at set time periods through the Java API of HBase.
[0006] According to the above technical solution, the analysis and control processing of the industrial monitoring and analysis system includes: Clustering is performed based on the manually set clustering K value and manually labeled data (device shutdown state, device standby state, device slow operation state, device high-intensity operation state, and data burr point state), the data is extracted and the corresponding features are constructed, and the corresponding features are merged into a feature vector, so that the feature vector data that completes the feature merging forms a feature matrix.
[0007] According to the above technical solution, the analysis and control processing of the industrial monitoring and analysis system further includes: After receiving the data information that has completed the clustering processing, the numpy.reshape() method is used to convert it into the set format required by the model, that is, the two-dimensional data is converted into three-dimensional data, where the three dimensions are the number of samples, the step length of the time series in each sample, and the number of features at each time point. After further normalization, the long-term and short-term network models are constructed; After completing the construction of the long-term and short-term network models, the parameters for setting the model training are controlled and managed, wherein the number of training times nb_epoch is set to 100, and the data size batch_size in the batch training is set manually.
[0008] According to the above technical solution, the comparative management and transmission display of the industrial monitoring and analysis system includes: After the selected equipment operation status data information is input into the long-term and short-term network model, the output result data information is compared with the set threshold value. If the comparison result does not meet the set data range, indicating that the predicted equipment will fail, an early warning notification will be issued immediately; If the comparison result meets or is equal to the set data range, indicating that the predicted equipment will not fail, it will be transmitted to the corresponding staff for review and management.
[0009] According to the above technical solution, a big data monitoring and analysis system includes: Management and control module, used for optimizing management and control of industrial monitoring and analysis systems; Analysis and processing module, used for operation analysis and processing of industrial monitoring and analysis systems; The comparison and display module is used to compare and display the data information of the industrial monitoring and analysis system.
[0010] According to the above technical solution, the management control module includes: Node gateway module, used for node gateway control of industrial monitoring and analysis system; Data preprocessing module, used to preprocess data information of industrial monitoring and analysis system; The data transmission module is used for data transmission management of industrial monitoring and analysis systems.
[0011] According to the above technical solution, the analysis and processing module includes: Cluster analysis module, used for data cluster analysis of industrial monitoring and analysis systems; Model construction module, used for model construction control and management; The training processing module is used to set parameters for model training.
[0012] According to the above technical solution, the comparison display module includes: Data comparison module, used for comparison and management of data information; The transmission display module is used to display the transmission of data information.
[0013] Compared with the prior art, the beneficial effects achieved by the present invention are as follows: in the present invention, by providing a management control module, an analysis and processing module and a comparison and display module, the gateway only identifies data packets that meet the settings, and then processes and forwards them, thereby effectively improving the work analysis efficiency of nodes and gateways, and converting electrical data calculations into real electrical parameter data, thereby effectively improving its management efficiency and avoiding the generation of analysis and processing errors. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 This is a flow chart of a big data monitoring and analysis method provided in the first embodiment of the present invention; Figure 2 This is a module structure diagram of a big data monitoring and analysis system provided in Example 2 of the present invention. DETAILED DESCRIPTION
[0015] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0016] Embodiment 1: Figure 1 This is a flowchart of a big data monitoring and analysis method provided in Embodiment 1 of the present invention. This embodiment can be applied to an industrial monitoring and analysis system. The method can be executed by a big data monitoring and analysis system provided in an embodiment of the present invention. The system is composed of multiple software and hardware modules, such as Figure 1 As shown, the method specifically comprises the following steps: S101, performing node gateway control of the industrial monitoring and analysis system; Exemplarily, in an embodiment of the present invention, when a wireless node in an industrial monitoring and analysis system starts working, it first performs a network access operation. The node initiates a network access request. When the gateway receives the network access request data packet of the wireless node, it will determine the type and subtype of the data packet. When it is determined from the type and subtype of the data packet that the data packet is a network access request data packet, the data packet will be sent to the server. When the server receives the network access request, it responds with a data packet to the gateway. The gateway sends the network access response packet to the destination wireless node based on the information in the data packet. When the wireless node receives the network access response packet, it sends an ACK data packet again to complete the network access processing. In this step, the gateway in the wireless sensor network mainly performs data conversion between the wired network and the wireless network and realizes the interconnection between the wired network and the wireless network. After the data packet of the device sent by the wireless node arrives at the gateway, the gateway sends the data to the server via Ethernet. The server will determine the type and subtype of the data packet. When it is determined from the type and subtype of the data packet that the data packet is a device data packet, the corresponding processing is performed. In the design of the wireless sensor network, the gateway only identifies the data packets that meet the settings, and then processes and forwards them, effectively improving the work analysis efficiency of the nodes and gateways.
[0017] S102, performing data preprocessing and transmission control on the industrial monitoring and analysis system; Exemplarily, in an embodiment of the present invention, the original data collected by the data acquisition card in the sensor node in the wireless sensor network layer is a hexadecimal number with two bytes as the unit, and the data is controlled to be sent to the gateway at every set collection interval. The gateway aggregates the data of multiple nodes and sends them to the server, and the server processes the data. In this step, the server will determine the type and subtype of the data packet when processing the message thread, and comprehensively determine whether the data packet is device data based on the type and subtype of the data packet. The data packet is preliminarily unpacked and stored in the device structure (including: device address, device type, data length, original collected data, node address and the number of devices connected to the node). The data stored in the device data structure by the device data structure server is the data collected by the data acquisition card, and the electrical data is calculated and converted into real electrical parameter data, which effectively improves its management efficiency and avoids the generation of analysis and processing errors.
[0018] The device collection data saved by the wireless sensor network layer is stored in MySQL, and the persistent data part located in the wireless sensor network layer server is imported into HBase at set time periods through the Java API of HBase.
[0019] S103, performing analysis and control processing of the industrial monitoring and analysis system; Exemplarily, in an embodiment of the present invention, clustering processing is performed according to the manually set clustering K value and manually labeled data (device shutdown state, device standby state, device slow running state, device high-intensity running state, and data burr point state), data is extracted and corresponding features are constructed, and the corresponding features are merged into a feature vector, so that the feature vector data after the feature merging forms a feature matrix; in this step, the feature matrix is standardized using the pandas tool, and the KMeans algorithm function in the scikit-learn algorithm library is called, the feature matrix and the set K value are input to obtain the clustering result, and the label data obtained by clustering are summarized and input into the corresponding file, so that the subsequent prediction analysis is more efficient and accurate.
[0020] After receiving the data information that has completed the clustering processing, the numpy.reshape() method is used to convert it into the set format required by the model, that is, the two-dimensional data is converted into three-dimensional data, where the three dimensions are the number of samples, the step size of the time series in each sample, and the number of features at each time point, and further normalized to construct a long-term and short-term network model; in this step, the step size of the time series in each sample is set to 120, and the number of features at each time point is 1. Then the data is normalized, and the data value is divided by the set data range. Then the model.add() method in the algorithm framework is called to add an LSTM network layer, and the input and output data size is used as a parameter to construct a long-term and short-term network model.
[0021] After completing the construction of the long-term and short-term network models, the parameters for setting the model training are controlled and managed, where the number of training times nb_epoch is set to 100, and the data size batch_size in the batch training is set to be manually set and adjusted; setting this parameter to 100 achieves the best effect and takes the least training time, and the data size batch_size in the batch training is set to be manually set and adjusted, which can effectively meet the different needs of users and reflect the intelligence of the analysis and processing.
[0022] S104. Conduct comparative management and transmission display of industrial monitoring and analysis systems; Exemplarily, in an embodiment of the present invention, after the selected equipment operation status data information is input into the long-term and short-term network model, the output result data information is compared with the set threshold value. If the comparison result does not meet the set data range, indicating that the predicted equipment will fail, an early warning notification is immediately issued; If the comparison result meets or is equal to the set data range, indicating that the predicted equipment will not fail, it will be transmitted to the corresponding staff for review and management.
[0023] Embodiment 2: Embodiment 2 of the present invention provides a big data monitoring and analysis system. Figure 2 A schematic diagram of the module structure of a big data monitoring and analysis system provided in the second embodiment, such as Figure 2 As shown, the system includes: Management and control module, used for optimizing management and control of industrial monitoring and analysis systems; Analysis and processing module, used for operation analysis and processing of industrial monitoring and analysis systems; The comparison and display module is used to compare and display the data information of the industrial monitoring and analysis system.
[0024] In some embodiments of the present invention, the management control module includes: Node gateway module, used for node gateway control of industrial monitoring and analysis system; Data preprocessing module, used to preprocess data information of industrial monitoring and analysis system; The data transmission module is used for data transmission management of industrial monitoring and analysis systems.
[0025] In some embodiments of the present invention, the analysis and processing module includes: Cluster analysis module, used for data cluster analysis of industrial monitoring and analysis systems; Model construction module, used for model construction control and management; The training processing module is used to set parameters for model training.
[0026] In some embodiments of the present invention, the comparison display module includes: Data comparison module, used for comparison and management of data information; The transmission display module is used to display the transmission of data information.
[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0028] Finally, it should be noted that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims
1. A big data monitoring and analysis method, characterized in that: include: Conduct node gateway control of industrial monitoring and analysis systems; Carry out data preprocessing and transmission control for industrial monitoring and analysis systems; Conduct analysis and control processing of industrial monitoring and analysis systems; Conduct comparative management and transmission display of industrial monitoring and analysis systems; The node gateway control of the industrial monitoring and analysis system includes: When the wireless node in the industrial monitoring and analysis system starts working, it first performs a network access operation. The node initiates a network access request. When the gateway receives the network access request data packet from the wireless node, it will determine the type and subtype of the data packet. When it is determined from the type and subtype of the data packet that the data packet is a network access request data packet, the data packet will be sent to the server. When the server receives the network access request, it responds with a data packet to the gateway. The gateway sends the network access response packet to the destination wireless node based on the information in the data packet. When the wireless node receives the network access response packet, it sends an ACK data packet again to complete the network access process.
2. The big data monitoring and analysis method according to claim 1, characterized in that: The data preprocessing and transmission control of the industrial monitoring and analysis system includes: The original data collected by the data acquisition card in the sensor node in the wireless sensor network layer is made into a hexadecimal number with two bytes as the unit, and the data is controlled to be sent to the gateway at every set collection interval. The gateway aggregates the data of multiple nodes and sends them to the server, and the server processes the data; The device collection data saved by the wireless sensor network layer is stored in MySQL, and the persistent data part located in the wireless sensor network layer server is imported into HBase at set time periods through the Java API of HBase.
3. The big data monitoring and analysis method according to claim 1, characterized in that: The analysis and control processing of the industrial monitoring and analysis system includes: Clustering is performed based on the manually set clustering K value and manually labeled data (device shutdown state, device standby state, device slow operation state, device high-intensity operation state, and data burr point state), the data is extracted and the corresponding features are constructed, and the corresponding features are merged into a feature vector, so that the feature vector data that completes the feature merging forms a feature matrix.
4. The big data monitoring and analysis method according to claim 1, characterized in that: The analysis and control process of the industrial monitoring and analysis system further includes: After receiving the data information that has completed the clustering processing, the numpy.reshape() method is used to convert it into the set format required by the model, that is, the two-dimensional data is converted into three-dimensional data, where the three dimensions are the number of samples, the step length of the time series in each sample, and the number of features at each time point. After further normalization, the long-term and short-term network models are constructed; After completing the construction of the long-term and short-term network models, the parameters for setting the model training are controlled and managed, wherein the number of training times nb_epoch is set to 100, and the data size batch_size in the batch training is set manually.
5. The big data monitoring and analysis method according to claim 1, characterized in that: The comparative management and transmission display of the industrial monitoring and analysis system includes: After the selected equipment operation status data information is input into the long-term and short-term network model, the output result data information is compared with the set threshold value. If the comparison result does not meet the set data range, indicating that the predicted equipment will fail, an early warning notification will be issued immediately; If the comparison result meets or is equal to the set data range, indicating that the predicted equipment will not fail, it will be transmitted to the corresponding staff for review and management.
6. A big data monitoring and analysis system, characterized by: include: Management and control module, used for optimizing management and control of industrial monitoring and analysis systems; Analysis and processing module, used for operation analysis and processing of industrial monitoring and analysis systems; The comparison and display module is used to compare and display the data information of the industrial monitoring and analysis system.
7. A big data monitoring and analysis system according to claim 6, characterized in that: The management control module includes: Node gateway module, used for node gateway control of industrial monitoring and analysis system; Data preprocessing module, used to preprocess data information of industrial monitoring and analysis system; The data transmission module is used for data transmission management of industrial monitoring and analysis systems.
8. The big data monitoring and analysis system according to claim 6, characterized in that: The analysis and processing module comprises: Cluster analysis module, used for data cluster analysis of industrial monitoring and analysis systems; Model construction module, used for model construction control and management; The training processing module is used to set parameters for model training.
9. The big data monitoring and analysis system according to claim 6, characterized in that: The comparison display module includes: Data comparison module, used for comparison and management of data information; The transmission display module is used to display the transmission of data information.