Industrial log noise reduction processing method and system based on artificial intelligence algorithm

Abstract

The invention relates to the technical field of network security, and discloses an industrial log noise reduction processing method and system based on an artificial intelligence algorithm, and the method comprises the steps: collecting historical alarm data of a device / application, carrying out the preprocessing, and dividing the preprocessed data into a training set and a test set; constructing a convolutional neural network model, and training and testing the convolutional neural network model based on the training set and the test set to obtain an alarm log noise reduction model; operating parameters of the alarm log noise reduction model are configured and integrated with a dynamic sensing system; receiving real-time alarm data of the equipment / application, and inputting the real-time alarm data into the alarm log noise reduction model to obtain alarm features; classifying according to the alarm features, and determining the category of the real-time alarm data; and carrying out storage and visual analysis based on the alarm features and categories. According to the invention, repeated or irrelevant alarm information can be accurately identified from a large number of alarms, and noise reduction processing is carried out, so that the system burden is reduced.

Description

Technical Field

[0001] The present invention relates to the field of network security technology, and in particular to an industrial log noise reduction processing method and system based on an artificial intelligence algorithm. Background Art

[0002] In the industrial production process, the log data generated by the operation of equipment or applications often contains a lot of noise, which may come from various factors such as normal operation fluctuations of the equipment, environmental interference, sensor errors, etc. The presence of noise not only increases the difficulty of processing log data, but also may lead to inaccurate data analysis results, affecting the efficiency and accuracy of fault diagnosis and maintenance decisions, and may also cover up truly important alarm information. Therefore, how to effectively remove noise from industrial logs and extract valuable alarm features is a key technical issue to improve the level of industrial automation and intelligence. Traditional log denoising methods mainly rely on rules and experience, which are difficult to adapt to complex and changeable industrial environments, and have limitations in noise identification and filtering. Therefore, an alarm denoising function is needed to filter and reduce invalid alarms and improve the quality and accuracy of alarms. Summary of the invention

[0003] The purpose of the present invention is to provide an industrial log noise reduction processing method and system based on artificial intelligence algorithm, aiming to solve the above problems.

[0004] The present invention provides an industrial log noise reduction processing method based on an artificial intelligence algorithm, comprising: Collect historical alarm data of devices / applications in the dynamic sensing system, the historical alarm data including daily alarms and abnormal alarms; Preprocessing the historical alarm data, and dividing the preprocessed historical alarm data into a training set and a test set; Constructing a convolutional neural network model, training the convolutional neural network model based on the training set, and testing the trained convolutional neural network model based on the test set to obtain an alarm log denoising model; Configuring the operating parameters of the alarm log denoising model according to server requirements, and integrating the alarm log denoising model with the dynamic perception system; Receive real-time alarm data of devices / applications in the dynamic perception system, perform preliminary processing on the real-time alarm data, and input the preliminarily processed real-time alarm data into the alarm log denoising model to obtain alarm features of the real-time alarm data; Classifying the real-time alarm data according to the alarm feature to determine the category of the real-time alarm data; The real-time alarm data is stored and visually analyzed based on the alarm characteristics and categories.

[0005] Preferably, preprocessing the historical alarm data includes: The historical alarm data is labeled and divided into value alarms and noise alarms; The historical alarm data is cleaned, standardized and feature extracted.

[0006] Preferably, the trained convolutional neural network model is tested based on the test set to obtain an alarm log denoising model, including: Get the precision, recall and F1 score of the test set when testing the trained convolutional neural network model; Evaluating the trained convolutional neural network model according to the precision, recall and F1 score; If the precision, recall rate and F1 score all pass the evaluation, the trained convolutional neural network model is determined to be the alarm log denoising model; If the precision, recall and F1 score do not pass the evaluation, the convolutional neural network model is retrained and tested.

[0007] Preferably, the operating parameters of the alarm log denoising model are configured according to server requirements, including: configuring a threshold and a processing frequency of the alarm log denoising model.

[0008] Preferably, the server requirement is to adapt the server specifications and storage requirements.

[0009] Preferably, the real-time alarm data is preliminarily processed, including: Using keyword screening rules to preliminarily screen the real-time alarm data; The real-time alarm data is filtered using a regular expression to remove the real-time alarm data whose format does not meet the requirements.

[0010] Preferably, the alarm characteristics include: timestamp, IP address, alarm type, alarm source and severity.

[0011] Preferably, classifying the real-time alarm data according to the alarm feature to determine the category of the real-time alarm data includes: An alarm category comparison table is provided, wherein the alarm category comparison table includes a plurality of alarm categories, and each alarm category is provided with a corresponding alarm feature; The alarm category comparison table is screened according to the alarm characteristics of the real-time alarm data to determine the category of the real-time alarm data.

[0012] Preferably, storing and visually analyzing the real-time alarm data based on the alarm characteristics and categories includes: Storing the real-time alarm data according to the alarm characteristics and categories, and performing statistical analysis; Determine the visualized attack chain of the real-time alarm data, and associate it through time, assets, networks and intelligence based on the traceability diagram, select threat-related entities and relationships as points and edges, form a small threat map, and form a visualized attack story chain.

[0013] The present invention also discloses an industrial log noise reduction processing system based on an artificial intelligence algorithm, which is used to apply the industrial log noise reduction processing method based on an artificial intelligence algorithm, comprising: A data collection module is used to collect historical alarm data of devices / applications in the dynamic sensing system, wherein the historical alarm data includes daily alarms and abnormal alarms; A data partitioning module, used to pre-process the historical alarm data and divide the pre-processed historical alarm data into a training set and a test set; A model building module, used to build a convolutional neural network model, train the convolutional neural network model based on the training set, and test the trained convolutional neural network model based on the test set to obtain an alarm log denoising model; A model integration module, configured to configure the operating parameters of the alarm log denoising model according to server requirements, and integrate the alarm log denoising model with a dynamic perception system; An alarm processing module is used to receive real-time alarm data of devices / applications in the dynamic perception system, perform preliminary processing on the real-time alarm data, and input the preliminarily processed real-time alarm data into the alarm log denoising model to obtain alarm features of the real-time alarm data; A log classification module, used to classify the real-time alarm data according to the alarm characteristics and determine the category of the real-time alarm data; The log storage module is used to store and visually analyze the real-time alarm data based on the alarm characteristics and categories.

[0014] Compared with the prior art, the beneficial effect of the present invention is that the functions of the present invention include alarm identification, duplicate checking, noise reduction and filtering. Through the powerful feature extraction capability of the CNN model, the system can accurately identify repeated or irrelevant alarm information from a large number of alarms, and perform noise reduction processing to reduce the system burden. Ultimately, the processed alarm information will be screened and the content with practical value will be extracted, thereby improving the work efficiency of quality inspectors and improving the overall reliability and response efficiency of the situational awareness system. This solution not only optimizes the processing flow of alarm information, but also enhances the system's adaptability to complex and dynamic environments.

[0015] The present invention can effectively identify and filter out false alarms or irrelevant alarm information by using a convolutional neural network model for training and testing, thereby improving the accuracy of alarm data. This helps to reduce the workload of operation and maintenance personnel and improve the efficiency of problem solving. By preprocessing and classifying historical alarm data, different types of alarm modes can be better understood, thereby reducing the false alarm rate. This is of great significance for improving the reliability and stability of the system. After the alarm log denoising model is integrated with the dynamic perception system, rapid processing and analysis of real-time alarm data can be achieved. This enables the system to respond to potential problems in a short time, thereby improving the real-time response capability of the entire system. By storing and visually analyzing real-time alarm data, operation and maintenance personnel can be helped to understand the operating status and potential problems of the system more intuitively. This helps to improve the work efficiency of operation and maintenance personnel, and also provides strong data support for decision makers. By effectively denoising the alarm data, unnecessary human and material investment caused by false alarms can be reduced, thereby reducing the operating costs of the enterprise. At the same time, improving the accuracy of alarm data also helps to reduce potential losses caused by missed reports. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings in the following description are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0017] Figure 1 It is a flow chart of an industrial log noise reduction processing method based on an artificial intelligence algorithm of the present invention; Figure 2 It is a functional block diagram of an industrial log noise reduction processing system based on an artificial intelligence algorithm of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0019] like Figure 1 As shown, the present invention provides an industrial log noise reduction processing method based on an artificial intelligence algorithm, comprising: Collect historical alarm data of devices / applications in the dynamic perception system, wherein the historical alarm data includes daily alarms and abnormal alarms.

[0020] The historical alarm data is preprocessed, and the preprocessed historical alarm data is divided into a training set and a test set.

[0021] A convolutional neural network model is constructed, the convolutional neural network model is trained based on the training set, and the trained convolutional neural network model is tested based on the test set to obtain an alarm log denoising model.

[0022] The operating parameters of the alarm log denoising model are configured according to server requirements, and the alarm log denoising model is integrated with a dynamic perception system.

[0023] Receive real-time alarm data of devices / applications in the dynamic perception system, perform preliminary processing on the real-time alarm data, and input the preliminarily processed real-time alarm data into the alarm log denoising model to obtain alarm features of the real-time alarm data.

[0024] The real-time alarm data is classified according to the alarm feature to determine the category of the real-time alarm data.

[0025] The real-time alarm data is stored and visually analyzed based on the alarm characteristics and categories.

[0026] In some embodiments of the present application, the historical alarm data is preprocessed, including: labeling the historical alarm data and dividing it into value alarms and noise alarms; and performing data cleaning, standardization processing and feature extraction on the historical alarm data.

[0027] In this embodiment, the data needs to be preprocessed, and the model is trained with the preprocessed data to build a diverse training set. These data cover daily and abnormal alarm information. The convolutional neural network model is trained with these data to enable it to identify different types of alarms and their relevance. Through large-scale data training, the model can grasp complex alarm features and relationships, and has strong automatic learning and adaptation capabilities.

[0028] In some embodiments of the present application, the trained convolutional neural network model is tested based on the test set to obtain an alarm log denoising model, including: obtaining the accuracy, recall rate and F1 score when the test set is used to test the trained convolutional neural network model; evaluating the trained convolutional neural network model according to the accuracy, recall rate and F1 score; if the accuracy, recall rate and F1 score all pass the evaluation, determining that the trained convolutional neural network model is the alarm log denoising model; if the accuracy, recall rate and F1 score fail the evaluation, retraining the convolutional neural network model and testing it.

[0029] After completing the training of the convolutional neural network (CNN) model, you first need to use the test set to evaluate the performance of the model. During the testing process, the precision, recall, and F1 score of the model on the test set are calculated. Accuracy refers to the ratio of the number of samples correctly predicted by the model to the total number of samples, which reflects the frequency of correct predictions by the model. Recall or true positive rate refers to the ratio of the number of positive samples correctly identified by the model to the total number of actual positive samples, which measures the model's ability to identify positive classes. The F1 score is the harmonic mean of precision and recall, which comprehensively considers the accuracy and recall of the model and is an important indicator for evaluating model performance.

[0030] The model can be comprehensively evaluated by calculating the precision, recall rate and F1 score. If all three indicators reach the predetermined threshold or standard, it can be considered that the model performs well in processing the alarm log denoising task, so the trained convolutional neural network model can be determined as the alarm log denoising model. If any of the three indicators does not meet the expected standard, it means that the model is still insufficient in some aspects and needs further optimization.

[0031] If the model fails the evaluation, we will analyze the shortcomings of the model based on the test results and take corresponding measures to improve it. Possible improvement measures include but are not limited to adjusting the network structure, optimizing hyperparameters, using more complex data enhancement techniques, or trying different training strategies. After the improvement, we need to retrain the model and use the test set for performance evaluation again until the model's precision, recall, and F1 score meet the requirements.

[0032] It is understandable that when training the model, a convolutional neural network model (CNN) is used for deep learning. CNN is suitable for processing data with spatial structure, and can effectively extract local features and perform noise reduction. And the performance is relatively stable and the computing requirements are low. The model is trained with preprocessed data for classification, recognition and noise reduction tasks. The model will learn to recognize alarm patterns and features, so that it can extract effective alarm information from various noises. The performance of the model is evaluated using the test set, including the model's accuracy, recall rate and F1 score. Cross-validation and hyperparameter optimization are used to improve the model performance to ensure that it can effectively identify and filter.

[0033] Implementation process: A convolutional neural network (CNN) for identifying alarm denoised logs was built and trained using TensorFlow and Keras, and tested on the MNIST dataset.

[0034] 1. Import the library Import TensorFlow and Keras related modules for building and training models.

[0035] 2. Load the dataset mnist.load_data(): Loads the MNIST dataset, which contains 60,000 training alarm logs and 10,000 testing alarm logs.

[0036] 3. Data Preprocessing reshape((60000, 28, 28, 1)): reshapes the alarm log from (60000, 28, 28) to (60000, 28, 28, 1), adding a dimension to represent the value of the alarm (a value of 1 means high-value valid alarm data).

[0037] astype('float32') / 255: Converts an integer to a floating point number and normalizes it to the range [0, 1].

[0038] to_categorical: Convert labels to one-hot encoded form.

[0039] 4. Build the model Sequential(): Initializes a sequential model.

[0040] Conv2D(32, (3, 3), activation='relu', input_shape=(28, 28, 1)) : adds a convolutional layer with 32 filters of size 3x3, using ReLU activation function, and input shape 28x28x1.

[0041] MaxPooling2D((2, 2)) : Adds a maximum pooling layer with a pooling window size of 2x2 to reduce the dimensionality of the feature map.

[0042] Flatten(): Flattens the feature map into a one-dimensional array.

[0043] Dense(128, activation='relu'): Add a fully connected layer with 128 neurons and ReLU activation function.

[0044] Dense(10, activation='softmax'): Add an output layer with 10 neurons corresponding to 10 categories and use the softmax activation function to output probability distribution.

[0045] 5. Compile the model optimizer='adam': Use the Adam optimizer.

[0046] loss='categorical_crossentropy': For multi-classification problems, use the cross entropy loss function.

[0047] metrics=['accuracy']: The evaluation metric is accuracy.

[0048] 6. Training the Model model.fit: Train the model on the training data, set epochs to 5 (training cycles), batch_size to 64 (64 samples per batch), and validate on 10% of the training set.

[0049] 7. Save the model model.save('mnist_model.h5'): Save the trained model as an H5 file.

[0050] 8. Evaluate the model model.evaluate: Evaluate the model on the test data.

[0051] The convolutional layer extracts spatial features, the pooling layer reduces the feature dimension, and the fully connected layer performs classification. The ReLU activation function increases nonlinearity to prevent the model from being too simple. The Softmax function is used to output the probability distribution of the predicted category. The training goal of the entire model is to minimize the loss function and improve the accuracy on unseen data.

[0052] The trained model intelligent algorithm performs duplicate checking, noise reduction and filtering, removes redundant and irrelevant or daily log alarm information and identifies it as low-value alarms, and identifies logs with potential safety hazards and certain risks as high-value alarms, so as to meet the needs of different business scenarios. This process model can dynamically adapt to environmental changes, continuously improve recognition capabilities, and can process massive amounts of data to achieve real-time response.

[0053] Example: Assume that the system detects two alarms: Alert A: Multiple failed login attempts to a user account originate from an internal IP during normal working hours.

[0054] Alarm B: A large amount of abnormal traffic from external IP addresses is detected, which may be a DDoS attack during business peak hours.

[0055] According to the evaluation criteria: Alarm A may be assessed as low value because it is a user operation error and has a limited impact.

[0056] Alert B is assessed as high value because it involves a possible DDoS attack, has a wide impact range, and occurs at a critical moment.

[0057] In some embodiments of the present application, the operating parameters of the alarm log denoising model are configured according to server requirements, including: configuring a threshold and a processing frequency of the alarm log denoising model.

[0058] In some embodiments of the present application, the server requirement is to adapt the server specifications and storage requirements.

[0059] In this embodiment, the operating parameters of the model are configured according to actual needs, such as setting thresholds, processing frequencies, etc. Adapt computing resources and environments, such as server specifications and storage requirements. Finally, ensure that the model can receive alarm information from devices and applications in real time and process it immediately. Deploy the trained model to the production environment to ensure that it can process alarm data in real time. Integrate the model with the situational awareness system to realize real-time processing of alarms reported by devices and applications.

[0060] In some embodiments of the present application, the real-time alarm data is preliminarily processed, including: using keyword screening rules to preliminarily screen the real-time alarm data; using regular expressions to filter the real-time alarm data to eliminate real-time alarm data whose formats do not meet the requirements.

[0061] In this embodiment, real-time alarm data from a device or application is received and preliminarily processed, such as keyword screening: applying basic keyword screening rules to preliminarily screen out relevant alarm information. Regular expression filtering: using regular expressions to further filter the alarm data and remove alarm information that does not meet the format requirements.

[0062] Use keyword screening rules for preliminary screening: By setting a series of keywords or phrases related to alarms, the system can automatically identify and filter out alarm data containing these keywords. For example, you can set keywords such as "error", "abnormal", "fault", etc., so that when these words appear in the alarm information, the system will mark it as an important alarm and give priority to it. This method can quickly narrow the scope of alarm data and improve processing efficiency.

[0063] Filter using regular expressions: Regular expressions are a powerful text processing tool that can define complex text matching patterns. In real-time alarm data processing, regular expressions can be used to remove alarm data that does not meet the required format. For example, you can set regular expression rules to check whether the alarm information contains necessary fields, such as timestamp, alarm level, device identification, etc. If the alarm data lacks these key information, or the information format is incorrect, the system will automatically filter it out to avoid affecting subsequent analysis and processing.

[0064] Through the above two steps, the real-time alarm data can be effectively processed preliminarily to ensure data quality and provide accurate and reliable input for subsequent alarm analysis and response.

[0065] In some embodiments of the present application, the alarm features include: timestamp, IP address, alarm type, alarm source and severity.

[0066] In this embodiment, the preliminarily processed alarm data is input into the trained alarm log denoising model, and relevant features are extracted from the identified alarm data: timestamp (such as specific date, time period), IP address (source IP address, destination IP address), alarm type (such as security vulnerability, system failure, performance bottleneck, malicious attack behavior), alarm source (such as system, application, network), severity (high, medium, low), etc. Based on the extracted features, the alarms are divided into different categories, such as security events, system failures, network anomalies, application failures, etc.

[0067] For example, the following features can be extracted for network traffic anomaly alarms: Traffic peak: whether it exceeds the normal range.

[0068] Abnormal mode: whether abnormal data packet types appear.

[0069] Time: Whether the anomaly occurred during peak business hours or during non-business hours.

[0070] It can be understood that the alarm log denoising model identifies the received alarms, filters out the noise alarms, and extracts valuable information. A feedback mechanism is set up to collect alarm data and user feedback in the actual operation of the system, and continuously optimize the model.

[0071] In some embodiments of the present application, the real-time alarm data is classified according to the alarm characteristics to determine the category of the real-time alarm data, including: providing an alarm category comparison table, the alarm category comparison table includes several alarm categories, and each alarm category is provided with a corresponding alarm feature; the alarm category comparison table is screened according to the alarm characteristics of the real-time alarm data to determine the category of the real-time alarm data.

[0072] In this embodiment, when processing real-time alarm data, it is first necessary to classify the data for quick identification and response. The classification process involves the following steps: Prepare an alarm category comparison table: Create a detailed comparison table containing all possible alarm categories in the system. Each category has a specific set of features that are used to distinguish different types of alarms. Define alarm features: Define a set of clear features for each alarm category. These features can include timestamps, IP addresses, alarm types, alarm sources, and severity. Filter real-time alarm data: When real-time alarm data arrives, the alarm features in these data are automatically checked. By comparing the features in the alarm category comparison table, the system can identify the alarm category that matches the real-time alarm data. Determine the category of real-time alarm data: Based on the screening results, the real-time alarm data will be assigned to the corresponding alarm category. In this way, the operator or the automation system can take appropriate response measures based on the category.

[0073] In some embodiments of the present application, the real-time alarm data is stored and visualized for analysis based on the alarm features and categories, including: storing the real-time alarm data according to the alarm features and categories, and performing statistical analysis; determining a visualized attack chain of the real-time alarm data, and associating it through time, assets, networks, and intelligence based on a traceability graph, selecting entities and relationships related to threats as points and edges, forming a small threat graph, and forming a visualized attack story chain.

[0074] In this embodiment, the real-time alarm data is stored in a database or log storage system, such as Elasticsearch, Splunk, etc.

[0075] like Figure 2 As shown, the present invention also discloses an industrial log noise reduction processing system based on artificial intelligence algorithm, which is used to apply the above industrial log noise reduction processing method based on artificial intelligence algorithm, including: The data collection module is used to collect historical alarm data of devices / applications in the dynamic perception system, and the historical alarm data includes daily alarms and abnormal alarms.

[0076] The data partitioning module is used to preprocess the historical alarm data and divide the preprocessed historical alarm data into a training set and a test set.

[0077] The model building module is used to build a convolutional neural network model, train the convolutional neural network model based on the training set, and test the trained convolutional neural network model based on the test set to obtain an alarm log denoising model.

[0078] The model integration module is used to configure the operating parameters of the alarm log denoising model according to the server requirements and integrate the alarm log denoising model with the dynamic perception system.

[0079] The alarm processing module is used to receive real-time alarm data of devices / applications in the dynamic perception system, perform preliminary processing on the real-time alarm data, and input the preliminarily processed real-time alarm data into the alarm log denoising model to obtain the alarm characteristics of the real-time alarm data.

[0080] The log classification module is used to classify the real-time alarm data according to the alarm characteristics and determine the category of the real-time alarm data.

[0081] The log storage module is used to store and visually analyze the real-time alarm data based on the alarm characteristics and categories.

[0082] By using artificial intelligence algorithms to reduce the noise of industrial logs, irrelevant information can be effectively removed and key features can be retained, thereby improving the accuracy of alarm data. This helps to reduce false alarms and missed alarms and improve the reliability of the system. And by performing preliminary processing and noise reduction on real-time alarm data, alarm features can be obtained in a short time, thereby improving the real-time nature of alarm data. This is of great significance for timely detection and handling of abnormal situations. Classifying and storing alarm data can make the alarm data more organized and convenient for subsequent analysis and processing. At the same time, visual analysis can help users understand the situation of alarm data more intuitively and improve data processing efficiency. Integrating the alarm log noise reduction model with the dynamic perception system can realize real-time monitoring and intelligent alarm of equipment / applications. This helps to improve the intelligence level of the system and reduce the need for manual intervention. Noise reduction and classified storage of alarm data can reduce the number of invalid alarms, reduce the workload of operation and maintenance personnel, and thus reduce operation and maintenance costs. Noise reduction of alarm data can effectively prevent malicious attackers from using a large amount of invalid alarm data to interfere with and damage the system, thereby improving the security of the system.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that they can still modify or replace the technical solution of the present invention with equivalents, and these modifications or equivalent replacements cannot cause the modified technical solution to deviate from the spirit and scope of the technical solution of the present invention.

[0084] The system provided in the above embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the modules or steps in the embodiments of the present invention can be decomposed or combined. For example, the modules in the above embodiment can be combined into one module, or further divided into multiple sub-modules to complete all or part of the functions described above. The names of the modules and steps involved in the embodiments of the present invention are only for distinguishing the modules or steps, and are not regarded as improper limitations of the present invention.

[0085] Those skilled in the art should be able to appreciate that the modules and method steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software or a combination of the two, and the programs corresponding to the software modules and method steps can be placed in random access memory (RAM), internal memory, read-only memory (ROM), electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs or any other form of storage medium known in the technical field. In order to clearly illustrate the interchangeability of electronic hardware and software, the composition and steps of each example have been generally described in the above description according to the function. Whether these functions are performed in electronic hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present invention.

Claims

1. An industrial log noise reduction processing method based on artificial intelligence algorithm, characterized in that: include: Collect historical alarm data of devices / applications in the dynamic sensing system, the historical alarm data including daily alarms and abnormal alarms; Preprocessing the historical alarm data, and dividing the preprocessed historical alarm data into a training set and a test set; Constructing a convolutional neural network model, training the convolutional neural network model based on the training set, and testing the trained convolutional neural network model based on the test set to obtain an alarm log denoising model; Configuring the operating parameters of the alarm log denoising model according to server requirements, and integrating the alarm log denoising model with the dynamic perception system; Receive real-time alarm data of devices / applications in the dynamic perception system, perform preliminary processing on the real-time alarm data, and input the preliminarily processed real-time alarm data into the alarm log denoising model to obtain alarm features of the real-time alarm data; Classifying the real-time alarm data according to the alarm feature to determine the category of the real-time alarm data; The real-time alarm data is stored and visually analyzed based on the alarm characteristics and categories.

2. The industrial log noise reduction processing method based on artificial intelligence algorithm according to claim 1 is characterized in that: Preprocessing the historical alarm data includes: The historical alarm data is labeled and divided into value alarms and noise alarms; The historical alarm data is cleaned, standardized and feature extracted.

3. The industrial log noise reduction processing method based on artificial intelligence algorithm according to claim 1 is characterized in that: The trained convolutional neural network model is tested based on the test set to obtain an alarm log denoising model, including: Get the precision, recall and F1 score of the test set when testing the trained convolutional neural network model; Evaluating the trained convolutional neural network model according to the precision, recall and F1 score; If the precision, recall rate and F1 score all pass the evaluation, the trained convolutional neural network model is determined to be the alarm log denoising model; If the precision, recall and F1 score do not pass the evaluation, the convolutional neural network model is retrained and tested.

4. The industrial log noise reduction processing method based on artificial intelligence algorithm according to claim 1 is characterized in that: The operating parameters of the alarm log denoising model are configured according to server requirements, including: configuring a threshold and a processing frequency of the alarm log denoising model.

5. The industrial log noise reduction processing method based on artificial intelligence algorithm according to claim 1 is characterized in that: The server requirements are to adapt the server specifications and storage requirements.

6. The method for industrial log noise reduction based on artificial intelligence algorithm according to claim 1 is characterized in that: The real-time alarm data is preliminarily processed, including: Using keyword screening rules to preliminarily screen the real-time alarm data; The real-time alarm data is filtered using a regular expression to remove the real-time alarm data whose format does not meet the requirements.

7. The industrial log noise reduction processing method based on artificial intelligence algorithm according to claim 1 is characterized in that: The alarm characteristics include: timestamp, IP address, alarm type, alarm source and severity.

8. The method for industrial log noise reduction based on artificial intelligence algorithm according to claim 1 is characterized in that: Classifying the real-time alarm data according to the alarm feature to determine the category of the real-time alarm data includes: An alarm category comparison table is provided, wherein the alarm category comparison table includes a plurality of alarm categories, and each alarm category is provided with a corresponding alarm feature; The alarm category comparison table is screened according to the alarm characteristics of the real-time alarm data to determine the category of the real-time alarm data.

9. The method for industrial log noise reduction based on artificial intelligence algorithm according to claim 1 is characterized in that: Storing and visually analyzing the real-time alarm data based on the alarm characteristics and categories includes: Storing the real-time alarm data according to the alarm characteristics and categories, and performing statistical analysis; Determine the visualized attack chain of the real-time alarm data, and associate it through time, assets, networks and intelligence based on the traceability diagram, select threat-related entities and relationships as points and edges, form a small threat map, and form a visualized attack story chain.

10. An industrial log noise reduction processing system based on artificial intelligence algorithm, used for applying the industrial log noise reduction processing method based on artificial intelligence algorithm as described in any one of claims 1 to 9, characterized in that: include: A data collection module is used to collect historical alarm data of devices / applications in the dynamic sensing system, wherein the historical alarm data includes daily alarms and abnormal alarms; A data partitioning module, used to pre-process the historical alarm data and divide the pre-processed historical alarm data into a training set and a test set; A model building module, used to build a convolutional neural network model, train the convolutional neural network model based on the training set, and test the trained convolutional neural network model based on the test set to obtain an alarm log denoising model; A model integration module, configured to configure the operating parameters of the alarm log denoising model according to server requirements, and integrate the alarm log denoising model with a dynamic perception system; An alarm processing module is used to receive real-time alarm data of devices / applications in the dynamic perception system, perform preliminary processing on the real-time alarm data, and input the preliminarily processed real-time alarm data into the alarm log denoising model to obtain alarm features of the real-time alarm data; A log classification module, used to classify the real-time alarm data according to the alarm characteristics and determine the category of the real-time alarm data; The log storage module is used to store and visually analyze the real-time alarm data based on the alarm characteristics and categories.

Images