A monitoring and alarm system and method for operation and maintenance management platform based on big data

Abstract

The present invention discloses a monitoring and alarm system and method for an operation and maintenance management platform based on big data, and relates to the field of data monitoring technology. The present invention includes the following steps: obtaining the historical alarm log of the operation and maintenance management platform; constructing a neural network model to train the historical alarm log and make an alarm recognition model; collecting data from the operation and maintenance management platform for processing, and inputting the processed data into the alarm recognition model; the alarm recognition model outputs a feature vector, and the feature vector is classified and processed; the system generates an alarm code according to the classification result and sends it to the operation and maintenance management platform; the operation and maintenance management platform sends the alarm code to the mobile terminal of the operation and maintenance personnel. The present invention constructs an alarm recognition model with the collected data input value, classifies the feature vector output by the model, generates an alarm code according to the classification result, and sends it to the mobile terminal with the operation and maintenance personnel, so that the operation and maintenance personnel can quickly find abnormal equipment and improve the accuracy of alarms and the efficiency of operation and maintenance.

Description

Technical Field

[0001] The present invention belongs to the technical field of data monitoring, and in particular relates to a monitoring and alarm system and method for an operation and maintenance management platform based on big data. Background Art

[0002] Based on the continuous deepening of operation and maintenance services, the connection with users is getting closer. Through continuous communication with customers, we found that basic operation and maintenance guarantee services can no longer meet the actual needs of users, and the demand for professional and customized service scenarios continues to emerge; based on the previous accumulation of operation and maintenance technology, the company's existing operation and maintenance tools are fully utilized in the design to achieve precise, professional, and thematic customized operation and maintenance service delivery capabilities in operation and maintenance, security, management, assets and other scenarios.

[0003] At present, all alarm information in the operation and maintenance management platform is pushed in chronological order. Although it is timely, it is impossible to make reasonable plans according to the operation and maintenance personnel. At the same time, there are many false alarms in the alarm information that the operation and maintenance personnel find to be normal, which wastes time costs and the operation and maintenance personnel's processing time for other alarms. Therefore, how to customize and push the quickly configured alarm information has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to provide an operation and maintenance management platform monitoring and alarm system and method based on big data. By constructing an alarm recognition model with the collected data input values, the feature vectors output by the model are classified and processed, and an alarm code is generated according to the classification result, thereby solving the problems of inaccurate alarms and low alarm efficiency in the existing system.

[0005] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0006] The present invention is a monitoring and alarm method for an operation and maintenance management platform based on big data, comprising the following steps:

[0007] Step S1: Obtain historical alarm logs of the operation and maintenance management platform;

[0008] Step S2: Construct a neural network model to train historical alarm logs and create an alarm recognition model;

[0009] Step S3: Collect data from the operation and maintenance management platform for processing, and input the processed data into the alarm recognition model;

[0010] Step S4: the alarm recognition model outputs a feature vector, and the feature vector is classified and processed;

[0011] Step S5: The system generates an alarm code based on the classification result and sends it to the operation and maintenance management platform;

[0012] Step S6: The operation and maintenance management platform sends the alarm code to the operation and maintenance personnel's mobile terminal.

[0013] As a preferred technical solution, in step S1, the historical alarm log needs to be preprocessed before the neural network model training. The specific preprocessing process is as follows:

[0014] Step S11: Collect logs to aggregate, analyze and clean up log data;

[0015] Step S12: inserting default values ​​into the missing values ​​in the log and discarding irrelevant entries;

[0016] Step S13: Use Spacy to delete stop words, perform word segmentation, and mark them;

[0017] Step S14: Segment the sentences using spaces and punctuation marks;

[0018] Step S15: Create an index for the pre-processed log.

[0019] As a preferred technical solution, in step S2, the alarm recognition model making process is as follows:

[0020] Step S21: converting the preprocessed log into structured data;

[0021] Step S22: selecting appropriate features according to the alarm information and data characteristics, specifically including text features and numerical features;

[0022] Step S23: combine multiple related features into new features and reduce feature dimensions, and divide them into a test set and a training set according to the proportion;

[0023] Step S24: Select a neural network for model training based on business requirements and data features;

[0024] Step S25: During model training, the input training set is calculated through each layer of the neural network to obtain the output result;

[0025] Step S26: Use the loss function to calculate the difference between the output result and the true label;

[0026] Step S27: Use the test set to test the trained model, and deploy the model to actual applications after the test passes.

[0027] As a preferred technical solution, in step S3, the data collected from the operation and maintenance management platform are subjected to denoising and normalization processing; the denoising processing uses a mean filtering algorithm to calculate the average value of the data, smooth the data and remove noise; the normalization processing scales the data according to data of different magnitudes or units so that all the data falls within the interval of [0, 1], and the specific conversion function is:

[0028]

[0029] In the formula, x is the sample data, x max is the maximum value of the sample data, x min is the minimum value of the sample data.

[0030] As a preferred technical solution, in step S3, feature extraction is required after normalization. The specific process is as follows: the normalized data is semantically embedded to obtain word vectors, the word vectors are input into the BiGRU model, the output word vectors are connected and input into TextCNN, TextCNN uses a convolution kernel of size [3, 4, 5] to convolve the input and output vectors, and the vectors obtained by different kernels are connected to generate a feature vector.

[0031] As a preferred technical solution, in step S4, the feature vector is input into a fully connected network for classification. The fully connected network uses a single linear layer to connect all outputs of the previous layer with all inputs of this layer, and uses the Softmax nonlinear activation function to perform matrix weight calculation on the input of this layer, and applies maximum pooling to obtain features corresponding to the alarm indicators.

[0032] As a preferred technical solution, in step S5, the generated alarm code includes an alarm number, a time series of the alarm code, an association rule and a confidence level; the alarm number represents the alarm code in the network service system, or all alarms corresponding to the alarm code; the time series of the alarm code collects the alarm frequency of each alarm code in the system within a certain time interval; the association rule represents the association relationship between two alarm codes; and the confidence level represents the probability under the association rule.

[0033] The present invention is a monitoring and alarm system for an operation and maintenance management platform based on big data, comprising a data acquisition unit, an integration unit, a model training unit, an alarm code generation unit, and an alarm sending unit;

[0034] The data collected by the data acquisition unit includes the front-end mechanical equipment status data and operation data collected by various sensors, PLC devices and other intelligent devices; the data of the operation and maintenance management platform is cleaned, aggregated and pre-processed when passing through the gateway;

[0035] The integrated unit processes the uploaded data. When the collected data reaches a dangerous value, the integrated unit directly selects alarm processing, directly stores the alarm and sends the alarm code to the mobile terminal of the operation and maintenance personnel.

[0036] The model training unit is used to train according to historical alarm logs to create an alarm recognition model;

[0037] The alarm code generating unit is used to generate an alarm code according to the recognition result of the alarm recognition module;

[0038] The alarm sending unit sends the fault code to the mobile terminal of the operation and maintenance personnel.

[0039] The present invention has the following beneficial effects:

[0040] The present invention constructs an alarm recognition model with the collected data input values, classifies the feature vectors output by the model, generates an alarm code according to the classification result, and sends it to the mobile terminal of the operation and maintenance personnel, so as to facilitate the operation and maintenance personnel to quickly find abnormal equipment and improve the alarm accuracy and operation and maintenance efficiency.

[0041] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0043] Figure 1 This is a flow chart of a monitoring and alarm method for an operation and maintenance management platform based on big data of the present invention;

[0044] Figure 2 The present invention is a schematic diagram of the structure of a monitoring and alarm system for an operation and maintenance management platform based on big data. DETAILED DESCRIPTION

[0045] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0046] In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0047] In order to make the purpose, technical solutions and advantages of this application more clear, the following Figure 1-2 It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0048] Embodiment 1

[0049] See also Figure 1 As shown, the present invention is a monitoring and alarm method for an operation and maintenance management platform based on big data, comprising the following steps:

[0050] Step S1: Obtain the historical alarm log of the operation and maintenance management platform; there are many ways to obtain the historical alarm log of the operation and maintenance management platform: 1. Query through the console, directly in the system software, click the view log option to display all logs of the current task execution, or you can view the log corresponding to each task according to the node where the task is executed; 2. Through the log collection tool, commonly used log collection tools include rsyslog (Linux system log), Syslog-ng (widely used system log collection tool), Logback and log4j (Java application log framework), etc.

[0051] Step S2: Construct a neural network model to train historical alarm logs and create an alarm recognition model;

[0052] Step S3: Collect data from the operation and maintenance management platform for processing, and input the processed data into the alarm recognition model;

[0053] Step S4: the alarm recognition model outputs a feature vector, and the feature vector is classified and processed;

[0054] Step S5: The system generates an alarm code based on the classification result and sends it to the operation and maintenance management platform;

[0055] Step S6: The operation and maintenance management platform sends the alarm code to the operation and maintenance personnel's mobile terminal.

[0056] In step S1, the historical alarm log needs to be preprocessed before the neural network model training. The specific preprocessing process is as follows:

[0057] Step S11: Collect logs to aggregate, analyze and clean up log data;

[0058] Log data summary includes:

[0059] (1) Collect log data: Log data needs to be collected from various sources, which may include server logs, application logs, system logs, etc.

[0060] (2) Unified format: Since log data may come from different systems and applications, it needs to be converted into a unified format for further processing.

[0061] (3) Merge log data: Merge log data from different sources into one data set to facilitate subsequent analysis.

[0062] Log data analysis includes:

[0063] (1) Identify key information: Log data usually contains a lot of information, but only a part of it is key. Therefore, it is necessary to identify and extract this key information, such as timestamp, event type, error code, etc.

[0064] (2) Convert data format: Convert the original log data into a format that is easier to process, such as CSV, JSON, etc.

[0065] (3) Apply regular expressions: Use regular expressions to match and extract specific patterns or information in logs.

[0066] Log data cleaning includes:

[0067] (1) Remove irrelevant information: Delete irrelevant information in the log, such as debugging information, comments, etc.

[0068] (2) Filling missing values: For missing values ​​in the log, you can choose to fill in default values ​​or delete records containing missing values.

[0069] (3) Standardize data: Convert data into a consistent format, such as converting all dates into the same format.

[0070] (4) Filter noisy data: Delete or correct obviously erroneous data, such as incorrectly formatted timestamps.

[0071] Step S12: inserting default values ​​into the missing values ​​in the log and discarding irrelevant entries;

[0072] Step S13: Use Spacy to remove stop words, perform word segmentation, and mark them; in natural language processing, stop words refer to those common words that usually have no practical significance for text analysis. Removing stop words can help improve the effect of text analysis; SpaCy is a popular Python library for advanced natural language processing tasks;

[0073] Step S14: Segment the sentences using spaces and punctuation marks;

[0074] Step S15: Create an index for the pre-processed log.

[0075] In step S2, the alarm recognition model making process is as follows:

[0076] Step S21: converting the preprocessed log into structured data;

[0077] Step S22: selecting appropriate features according to the alarm information and data characteristics, specifically including text features and numerical features;

[0078] Step S23: combine multiple related features into new features and reduce feature dimensions, and divide them into a test set and a training set according to the proportion;

[0079] Step S24: Select a neural network for model training based on business requirements and data features;

[0080] Step S25: During model training, the input training set is calculated through each layer of the neural network to obtain the output result;

[0081] Step S26: Use the loss function to calculate the difference between the output result and the true label;

[0082] Step S27: Use the test set to test the trained model, and deploy the model to actual applications after the test passes.

[0083] In step S3, the data collected from the operation and maintenance management platform are denoised and normalized; the denoising process uses a mean filter algorithm to calculate the average value of the data to smooth the data and remove noise; the normalization process scales the data according to different magnitudes or units so that all the data falls within the interval of [0, 1]. The specific conversion function is:

[0084]

[0085] In the formula, x is the sample data, x max is the maximum value of the sample data, x min is the minimum value of the sample data.

[0086] In step S3, feature extraction is required after normalization. The specific process is as follows: the normalized data is semantically embedded to obtain word vectors, the word vectors are input into the BiGRU model, the output word vectors are connected and input into TextCNN, TextCNN uses convolution kernels of size [3, 4, 5] to convolve the input and output vectors, and the vectors obtained by different kernels are connected to generate feature vectors.

[0087] In step S4, the feature vector is input into a fully connected network for classification. The fully connected network uses a single linear layer to connect all outputs of the previous layer with all inputs of this layer, uses the nonlinear activation function of Softmax to perform matrix weight calculation on the input of this layer, and applies maximum pooling to obtain features corresponding to the alarm indicator.

[0088] In step S5, the generated alarm code includes an alarm number, a time series of the alarm code, an association rule and a confidence level; the alarm number represents the alarm code in the network service system, or all the alarms corresponding to the alarm code; the time series of the alarm code collects the alarm frequency of each alarm code in the system within a certain time interval; the association rule represents the association relationship between two alarm codes; and the confidence level represents the probability under the association rule.

[0089] Alarm code: represents the alarm code in the monitoring alarm system, or all alarms corresponding to the alarm code. A single alarm code is represented by lowercase letters, such as alarm code i and j, and an alarm code set is represented by uppercase letters, such as alarm code set I and J.

[0090] Time series of alarm codes: collect the alarm frequency of each alarm code in the system at a certain time interval (such as every minute), and record the number of alarm codes i in the time range T as time series;

[0091] Association rules: Indicates the association between two alarm codes (sets). Association rules are divided into "causality" and "correlation". i→j means that alarm code i is the cause of alarm code j, and alarm code j is the result caused by alarm code i;

[0092] Confidence: It represents the probability of an association rule, expressed as P(i→j), which specifically refers to the confidence of a directed association relationship.

[0093] Embodiment 2

[0094] See also Figure 2 As shown, the present invention is a monitoring and alarm system for an operation and maintenance management platform based on big data, which can be used to execute the method content of Embodiment 1 of the present invention, including: a data acquisition unit, an integration unit, a model training unit, an alarm code generation unit, and an alarm sending unit;

[0095] The data collected by the data acquisition unit includes the front-end mechanical equipment status data and operation data collected by various sensors, PLC devices and other intelligent devices; the data of the operation and maintenance management platform is cleaned, aggregated and pre-processed when passing through the gateway;

[0096] The integrated unit processes the uploaded data. When the collected data reaches a dangerous value, it directly selects alarm processing, directly stores the alarm and sends the alarm code to the mobile terminal of the operation and maintenance personnel;

[0097] The model training unit is used to train based on historical alarm logs to create an alarm recognition model;

[0098] The alarm code generation unit is used to generate an alarm code according to the recognition result of the alarm recognition module;

[0099] The alarm sending unit sends the fault code to the mobile terminal of the operation and maintenance personnel.

[0100] It is worth noting that in the above system embodiment, the various units included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0101] In addition, those skilled in the art can understand that all or part of the steps in the above-mentioned embodiments can be completed by instructing related hardware through a program, and the corresponding program can be stored in a computer-readable storage medium.

[0102] The preferred embodiments of the present invention disclosed above are only used to help illustrate the present invention. The preferred embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A monitoring and alarm method for an operation and maintenance management platform based on big data, characterized in that: The steps include: Step S1: Obtain historical alarm logs of the operation and maintenance management platform; Step S2: Construct a neural network model to train historical alarm logs and create an alarm recognition model; Step S3: Collect data from the operation and maintenance management platform for processing, and input the processed data into the alarm recognition model; Step S4: the alarm recognition model outputs a feature vector, and the feature vector is classified and processed; Step S5: The system generates an alarm code based on the classification result and sends it to the operation and maintenance management platform; Step S6: The operation and maintenance management platform sends the alarm code to the operation and maintenance personnel's mobile terminal.

2. According to a big data-based operation and maintenance management platform monitoring and alarm method according to claim 1, it is characterized in that: In step S1, the historical alarm log needs to be preprocessed before the neural network model training. The specific preprocessing process is as follows: Step S11: Collect logs to aggregate, analyze and clean up log data; Step S12: inserting default values ​​into the missing values ​​in the log and discarding irrelevant entries; Step S13: Use Spacy to delete stop words, perform word segmentation, and mark them; Step S14: Segment the sentences using spaces and punctuation marks; Step S15: Create an index for the pre-processed log.

3. According to a big data-based operation and maintenance management platform monitoring and alarm method according to claim 1, it is characterized in that: In step S2, the alarm recognition model making process is as follows: Step S21: converting the preprocessed log into structured data; Step S22: selecting appropriate features according to the alarm information and data characteristics, specifically including text features and numerical features; Step S23: combine multiple related features into new features and reduce feature dimensions, and divide them into a test set and a training set according to the proportion; Step S24: Select a neural network for model training based on business requirements and data features; Step S25: During model training, the input training set is calculated through each layer of the neural network to obtain the output result; Step S26: Use the loss function to calculate the difference between the output result and the true label; Step S27: Use the test set to test the trained model, and deploy the model to actual applications after the test passes.

4. According to a big data-based operation and maintenance management platform monitoring and alarm method according to claim 1, it is characterized in that: In step S3, the data collected from the operation and maintenance management platform are subjected to denoising and normalization processing; the denoising processing uses a mean filtering algorithm to calculate the average value of the data, smooth the data and remove noise; the normalization processing scales the data according to different magnitudes or units so that all the data falls within the interval of [0, 1], and the specific conversion function is: In the formula, x is the sample data, x max is the maximum value of the sample data, x min is the minimum value of the sample data.

5. According to a big data-based operation and maintenance management platform monitoring and alarm method according to claim 1, it is characterized in that: In the step S3, feature extraction is required after normalization, and the specific process is as follows: the normalized data is semantically embedded to obtain word vectors, the word vectors are input into the BiGRU model, the output word vectors are connected and input into TextCNN, TextCNN uses a convolution kernel of size [3, 4, 5] to convolve the input and output vectors, and the vectors obtained by different kernels are connected to generate a feature vector.

6. The method for monitoring and warning an operation and maintenance management platform based on big data according to claim 1 is characterized in that: In step S4, the feature vector is input into a fully connected network for classification. The fully connected network uses a single linear layer to connect all outputs of the previous layer with all inputs of this layer, uses the nonlinear activation function of Softmax to perform matrix weight calculation on the input of this layer, and applies maximum pooling to obtain features corresponding to the alarm indicator.

7. The method for monitoring and warning an operation and maintenance management platform based on big data according to claim 1 is characterized in that: In step S5, the generated alarm code includes an alarm number, a time series of the alarm code, an association rule and a confidence level; the alarm number represents an alarm code in the network service system, or all alarms corresponding to the alarm code; the time series of the alarm code collects the alarm frequency of each alarm code in the system within a certain time interval; the association rule represents the association relationship between two alarm codes; and the confidence level represents the probability under the association rule.

8. A monitoring and alarm system for an operation and maintenance management platform based on big data, comprising a data acquisition unit, an integration unit, a model training unit, an alarm code generation unit, and an alarm sending unit; characterized in that ； The data collected by the data acquisition unit includes the front-end mechanical equipment status data and operation data collected by various sensors, PLC devices and other intelligent devices; the data of the operation and maintenance management platform is cleaned, aggregated and pre-processed when passing through the gateway; The integrated unit processes the uploaded data. When the collected data reaches a dangerous value, the integrated unit directly selects alarm processing, directly stores the alarm and sends the alarm code to the mobile terminal of the operation and maintenance personnel. The model training unit is used to train according to historical alarm logs to create an alarm recognition model; The alarm code generating unit is used to generate an alarm code according to the recognition result of the alarm recognition module; The alarm sending unit sends the fault code to the mobile terminal of the operation and maintenance personnel.

Images