Data security anomaly detection system and method based on AI technology

Through the data security anomaly detection system based on AI technology, the log data anomaly detection model of recurrent neural network and attention mechanism is used to solve the problems of low detection efficiency, insufficient accuracy and lack of alarm mechanism in the existing technology, and achieve efficient and accurate The log data abnormal detection and intelligent alarm notification have improved the response speed and accident traceability of data security management.

CN119025311AInactive Publication Date: 2024-11-26HEFEI YUANLI ZHONGHE ENERGY TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411006736.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-25
Publication Date
2024-11-26
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing data abnormality detection technology has shortcomings in detection efficiency and accuracy, especially in the log files of IoT terminal devices, it is difficult to effectively process large amounts of data and provide timely alarm notifications, resulting in insufficient response speed and accident traceability of data security management.

Method used

Using a data security anomaly detection system based on AI technology, the original log data of terminal devices is monitored, collected and structured, and the abnormal detection model is used to perform abnormal detection, and the log is constructed in combination with recurrent neural network and attention mechanism. Data exception detection model realizes automatic collection, storage, analysis and alarm notification.

Benefits of technology

It significantly improves the efficiency and accuracy of log data processing, improves the accuracy of abnormal detection, enhances the response speed and accident traceability of data security management, and provides intelligent guarantees for data security.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119025311A_ABST
    Figure CN119025311A_ABST
Patent Text Reader

Abstract

The invention discloses a data security anomaly detection system and method based on an AI technology. The method comprises the following steps: monitoring, collecting and structuralizing original log data of terminal equipment to obtain structured log data, and carrying out storage management and data analysis on the structured log data to obtain a log event template; performing preprocessing and feature processing on the log event template to obtain a log sequence and a word vector space; performing anomaly detection on the word vector space by utilizing a pre-trained log data anomaly detection model to obtain an anomaly detection result; and performing alarm notification according to an abnormal detection result and recording an alarm process. The invention relates to the technical field of data security detection, and by combining an AI technology, the calculation process and the data updating step are reduced; and an alarm notification mechanism is introduced, so that the data security of the Internet of Things terminal equipment is enhanced, and the technical problems of low detection efficiency, insufficient detection precision and lack of an alarm mechanism in a data security anomaly detection system are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of data security and artificial intelligence, and relates to data anomaly detection technology, specifically a data security anomaly detection system and method based on AI technology. Background Art

[0002] Nowadays, Internet services have been deeply integrated into people's daily lives, and Internet technology is still developing continuously; among them, the development of cloud computing technology is particularly rapid. Based on the stronger data storage and computing capabilities provided by cloud services, the application scope and scale of Internet services will be further expanded. In the Chinese patent with application number 201910383583.3, a "data anomaly detection method and device" is disclosed. This method divides the data set by attributes and calculates the information entropy to obtain the attribute weight; then calculates the cosine distance and initial anomaly value between data objects; inserts these data into the sequence in time sequence, and updates the initial anomaly value according to the abnormal state of the adjacent data set; finally detects the anomaly in the data set, and determines it as abnormal if it exceeds the threshold. Another "data anomaly detection method and device" is disclosed in the Chinese patent with application number 201910242202.X. This method first obtains the time series data of the indicator to be detected, including multiple monitoring time points and their corresponding indicator values; then determines the problem indicator value through data clustering; finally, if the monitoring time point corresponding to the problem indicator value conforms to the time distribution law of the anomaly value, the indicator to be detected is determined to be abnormal, and the problem indicator value is used as the abnormal indicator value.

[0003] Existing technologies mainly rely on traditional data processing techniques, such as cosine distance and clustering methods, for data anomaly detection. Although these methods can achieve basic data anomaly detection, they have obvious limitations. For example, traditional data processing techniques cannot fully mine the complex patterns and deep features in the data, resulting in insufficient detection accuracy and limiting their effectiveness in identifying and predicting complex abnormal data. In addition, existing technologies involve cumbersome calculation processes and data update steps, resulting in low efficiency and heavy computational burden. In particular, in the detection of anomalies in log files of IoT terminal devices, existing methods do not take into account the problem that log data may be difficult to collect in full due to its large volume, and how to effectively screen and use key information in log data for anomaly detection. In addition, existing technologies also lack an alarm notification mechanism, which brings potential risks to the timely response of data security management. Summary of the invention

[0004] The present invention aims to solve at least one of the technical problems existing in the prior art; to this end, the present invention proposes a data security anomaly detection system and method based on AI technology, which is used to solve the technical problems of low detection efficiency, insufficient detection accuracy and lack of alarm mechanism in the data security anomaly detection system.

[0005] A first aspect of the present invention provides a data security anomaly detection method based on AI technology, comprising the following steps:

[0006] Monitoring, collecting and structured processing of original log data of terminal devices to obtain structured log data, performing storage management and data analysis on the structured log data to obtain log event templates;

[0007] Preprocessing and feature processing are performed on the log event template to obtain a log sequence and a word vector space;

[0008] Using a pre-trained log data anomaly detection model to perform anomaly detection on the word vector space to obtain an anomaly detection result;

[0009] Providing an alarm notification based on the abnormal detection result and recording the alarm process;

[0010] It should be noted that the function of monitoring is to start the collector to collect the original log data when the log file changes.

[0011] Based on the above method, AI technology is integrated to realize the automatic collection, storage, analysis and processing of terminal device log data, and pre-trained anomaly detection models are used to identify data security risks. This not only improves the detection efficiency of abnormal log data and ensures the consistency and accuracy of data processing, but also enhances the response speed and accident tracing capabilities of data security management through timely abnormal alarms and records, providing intelligent protection for data security.

[0012] Furthermore, the monitoring, collecting and structured processing of the original log data of the terminal device includes the following steps:

[0013] Monitor and collect the log files of the specified terminal device to obtain the original log data;

[0014] Performing time format unification, type conversion, and query classification operations on the original log data, and performing structured processing to obtain structured log data;

[0015] Furthermore, the storage management and data parsing of the log data include the following steps:

[0016] Storing the structured log data and establishing a data index;

[0017] Perform visual analysis on stored log data;

[0018] Parsing the stored log data into a sequence of string groups;

[0019] Compare the string group sequence with the log event template stored in the current LCSMap to find a log event template that matches the string group sequence;

[0020] It should be noted that several pre-set log event templates are saved in LCSMap.

[0021] The data processing efficiency is significantly improved through automatic monitoring, collection and structured processing of raw log data; the data is stored and managed by creating indexes, which enables fast data query and analysis, while the visual analysis of log data enhances the readability of the data. At the same time, the data parsing process also effectively normalizes the log data, providing accurate basic data for subsequent data analysis and anomaly detection.

[0022] Furthermore, the log event template is preprocessed and feature processed, including:

[0023] Preprocessing the log event template: performing a text cleaning operation on the log event template; performing word segmentation, stop word removal and spelling error correction operations on the cleaned log event template to obtain a preprocessed log event template; grouping the preprocessed log event template to obtain a log sequence;

[0024] Feature processing of the log event template: constructing a word vector space for the context of the log event template; and associating the word vector space with adjacent text information in the log event template.

[0025] Furthermore, the performing anomaly detection on the word vector space using the pre-trained log data anomaly detection model includes:

[0026] Design a log data anomaly detection model: Build a log data anomaly detection model based on a recurrent neural network and introduce an attention mechanism;

[0027] Constructing a data set: Using IoT data collection technology to obtain log data from terminal devices, processing the log data into word vector representation text, and constructing a log data training set for model training;

[0028] Training model: Use the log data training set to train the log data anomaly detection model, and calculate the model accuracy according to the model accuracy formula: Perform accuracy verification, optimize and adjust the model according to the accuracy verification results, and obtain the pre-trained log data anomaly detection model; PR represents the accuracy value of the log data anomaly detection model, TP represents the number of sample data detected correctly, and FP represents the number of sample data detected incorrectly.

[0029] Output model results: Use the pre-trained log data anomaly detection model to perform an anomaly detection operation on the word vector representation text and obtain the detection result.

[0030] Furthermore, the log data anomaly detection model includes:

[0031] An input layer is used to receive a word vector representation text corresponding to the log event template;

[0032] A Bi-GRU layer, used to capture the bidirectional semantic dependencies in the word vector representation text and the context information of the log sequence;

[0033] An attention layer, used to learn local features of the log sequence and assign different weights to log event templates in the sequence;

[0034] The output layer is used to output the detection results; wherein the detection results include abnormal detection results and normal detection results.

[0035] By designing a log data anomaly detection model based on recurrent neural networks and attention mechanisms, we can deeply analyze the complex patterns and bidirectional semantic dependencies in sequence data, effectively capture the temporal dynamic characteristics in log data, and allow the model to automatically focus on the information most relevant to anomaly detection, thereby improving the accuracy of data security anomaly detection.

[0036] Furthermore, issuing an alarm notification and recording the alarm process according to the abnormal detection result includes:

[0037] Recording the abnormality of the detection result; wherein the content of the abnormality record includes abnormal log content, abnormal context, record sequence number, timestamp and abnormal value;

[0038] Manage the alarm rules preset by the terminal equipment management personnel; wherein the alarm rules include the alarm rule name, filtering alarm level, record sequence number and alarm handler;

[0039] Matching the abnormal detection result with the alarm rule to obtain a matching result;

[0040] Alarm record, based on the matching result, sends an alarm notification to the corresponding alarm handler and records the alarm process.

[0041] A second aspect of the present invention provides a data security anomaly detection system based on AI technology, comprising: a log data acquisition unit, a log data processing unit, a log anomaly data detection unit and a system alarm unit.

[0042] Further, the log data acquisition unit includes: a log data acquisition module, which is used to monitor, collect and perform structured processing on the original log data of the terminal device to obtain structured log data;

[0043] A log data storage module, used for storing the structured log data;

[0044] The log data parsing module is used to perform parsing operations on the stored structured log data.

[0045] Further, the log data processing unit includes: a log data preprocessing module, which is used to perform preprocessing operations on the parsed and stored log event templates;

[0046] The log data feature processing module is used to perform digital symbolization operations on the pre-processed log event template to obtain word vectors to represent the text.

[0047] Furthermore, the log abnormal data detection unit includes: an abnormal detection model design module, which is used to construct a log data abnormality detection model based on a recurrent neural network and introducing an attention mechanism;

[0048] The data set construction module is used to process the log data obtained from the terminal device into word vector representation text and construct a log data training set for model training;

[0049] A model training module is used to perform model training on the log data anomaly detection model using the log data training set, and perform accuracy verification according to the model accuracy calculation formula, and optimize and adjust the model according to the accuracy verification result to obtain a pre-trained log data anomaly detection model;

[0050] The model output result acquisition module is used to use the pre-trained log data anomaly detection model to perform anomaly detection operations on the word vector representation text and obtain the detection results.

[0051] The system alarm unit includes: an abnormality recording module, which is used to record the abnormality of the detection result; wherein the abnormality record content includes abnormal log content, abnormal context, record sequence number, timestamp and abnormal value;

[0052] The alarm rule management module is used to manage the alarm rules preset by the terminal equipment management personnel; wherein the alarm rules include the alarm rule name, the filtering alarm level, the record sequence number and the alarm handler;

[0053] An alarm rule matching module is used to perform an alarm rule matching operation on the abnormal value in the detection result and the alarm rule to obtain a matching result;

[0054] The alarm recording module is used to send an alarm notification to the corresponding alarm handler based on the matching result and record the alarm process.

[0055] Compared with the prior art, the present invention has the following beneficial effects:

[0056] Through automated data collection, efficient data storage and intelligent data analysis, the efficiency and accuracy of log data processing are significantly improved. The log data anomaly detection model based on a recurrent neural network and the introduction of an attention mechanism improves the accuracy of anomaly detection by deeply analyzing the complex patterns and bidirectional semantic dependencies in the data. In addition, the present invention also realizes timely alarm notification and recording of abnormal values ​​in the test results, enhancing the response speed and accident tracing capabilities of data security management. In general, the present invention provides an intelligent, efficient and reliable means of ensuring data security, which is particularly suitable for terminal equipment environments that need to process large amounts of log data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0058] Figure 1 A schematic flow chart of the data security anomaly detection method based on AI technology provided by the present invention;

[0059] Figure 2 A schematic flow chart of the preprocessing and feature processing steps of the log event template provided by the present invention;

[0060] Figure 3 A schematic flow chart of the log data anomaly detection steps provided by the present invention;

[0061] Figure 4 A schematic flow chart of the alarm notification and recording steps provided by the present invention;

[0062] Figure 5 A schematic block diagram of a data security anomaly detection system based on AI technology provided by the present invention;

[0063] Figure 6 A schematic block diagram of a log data acquisition unit provided by the present invention;

[0064] Figure 7 A schematic block diagram of a log data processing unit provided by the present invention;

[0065] Figure 8 A schematic block diagram of a log abnormal data detection unit provided by the present invention;

[0066] Fig. 9 A schematic block diagram of a system alarm unit provided by the present invention;

[0067] In the figure: 100, log data acquisition unit; 101, log data collection module; 102, log data storage module; 103, log data parsing module; 200, log data processing unit; 201, log data preprocessing module; 202, log data feature processing module; 300, log abnormal data detection unit; 301, abnormal detection model design module; 302, data set construction module; 303, model training module; 304, model output result acquisition module; 400, system alarm unit; 401, log data abnormality recording module; 402, alarm rule management module; 403, alarm rule matching module; 404, alarm recording module. DETAILED DESCRIPTION

[0068] The technical solution of the present invention will be clearly and completely described below in conjunction with the embodiments. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0069] See also Figure 1-Figure 4 The first embodiment of the present invention provides a data security anomaly detection method based on AI technology, comprising the following steps:

[0070] Monitor, collect and structure the original log data of the terminal device to obtain structured log data, store and manage the structured log data and parse the data to obtain log event templates;

[0071] Preprocessing and feature processing are performed on the log event template to obtain a processed log event template;

[0072] Use the pre-trained log data anomaly detection model to perform anomaly detection on the processed log event template to obtain anomaly detection results;

[0073] Issue alarm notifications based on abnormal detection results and record the alarm process.

[0074] In this embodiment, taking the security anomaly detection method for the log data of the IoT terminal device as an example, the original log data of the terminal device is monitored, collected and structured, and Filebeat can be used to monitor the log file of the specified IoT terminal device; when the log file changes, the collector is started to collect the original log data; then the plug-in in Filebeat is called to unify the time format, convert the type, and query and classify the original log data to obtain processed data, and then the processed data is converted into structured log data through Grok expression syntax;

[0075] In order to provide accurate basic data, the structured log data is stored, managed and parsed, including the following steps:

[0076] Use Elasticsearch to store structured log data and create indexes for different data items to implement conditional queries, statistics, and aggregation operations;

[0077] Use Kibana to visualize and analyze stored log data;

[0078] Based on the specific log event templates stored in the LCSMap, a set of predefined delimiters are used to parse the stored log data into a sequence of string groups;

[0079] Compare the string group sequence with the log event templates saved in the current LCSMap to find the log event template that matches the string group sequence.

[0080] For example, an IoT terminal device generates the following raw log data lines:

[0081] ```

[0082] {"timestamp":"2023-09-15T10:30:00","device_id":"sensor_001","sensor_data":{"temperature":24.5,"humidity":60},"battery_level":75}

[0083] {"timestamp":"2023-09-15T10:31:00","device_id":"sensor_002","sensor_data":{"temperature":26.0,"humidity":58},"battery_level":80}

[0084] {"timestamp":"2023-09-15T10:32:00","device_id":"sensor_001","sensor_data":{"temperature":24.7,"humidity":61},"network_status":"offline"}

[0085] ```

[0086] For the original log data, the Filebeat tool is used to monitor and collect it, and it is converted into a structured form through Grok expressions. Then, Elasticsearch is used to store these structured log data and index different data items (such as device ID, sensor data, battery power, etc.) for fast query, statistics, and aggregation operations.

[0087] Use Kibana to visualize and analyze the data stored in Elasticsearch. For example, you can create a chart to show the battery level changes of different devices, or analyze the temperature and humidity data of a specific device. Then parse the stored log data into a sequence of string groups. Assume that there are the following predefined delimiters: `["","\t",""]`. Applying these delimiters, each log data can be split into multiple strings, for example:

[0088] ```

[0089] ["timestamp:2023-09-15T10:30:00","device_id:sensor_001","sensor_data:{"temperature":24.5,"humidity":60}","battery_level:75"]

[0090] ```

[0091] At this point, assume that the following log event template exists in LCSMap:

[0092] ```

[0093] [{"template_id":1,"pattern":["timestamp","device_id","sensor_data","battery_level"],"index":1},

[0094] {"template_id":2,"pattern":["timestamp","device_id","network_status"],"index":2}]

[0095] ```

[0096] Calculate the longest common subsequence (LCS) length of each string group sequence and the log event template. In this example, the first log event has an LCS length of 4 with template ID 1, which is a complete match, while the LCS length with template ID 2 is 2, and only "timestamp" and "device_id" match. Therefore, template ID 1 is selected as the matching result.

[0097] If there are multiple templates with the same maximum LCS length, the template with the smallest index value is selected. If no matching template is found, a new log event template is created in the LCSMap based on the string group sequence and used as the log event template for the string group sequence.

[0098] Through the above data storage and parsing process, not only can the log data of IoT terminal devices be effectively stored and managed, but the log data can also be parsed into corresponding log event templates through matching algorithms, thereby providing an accurate data basis for subsequent anomaly detection tasks.

[0099] In this embodiment, in the process of preprocessing and feature processing the log event template, in order to capture the sequence information in the log data, firstly, in the preprocessing stage, the log event template is cleaned to remove identifiers, punctuation marks and unconventional characters to remove noise in the data, ensure the neatness of the text data, and avoid introducing unnecessary errors in subsequent processing;

[0100] Then, in order to better extract valuable keywords and reduce data dimensions, the cleaned text needs to be segmented, stop words removed, and spelling corrected;

[0101] Then the session window partitioning method is used to group the log event templates to form a series of log sequences;

[0102] In the feature processing stage, FastText can effectively capture the semantic information of words and build a word vector space for the context of the log event template.

[0103] Finally, N-gram features are used to associate adjacent text information in log event templates, thereby optimizing the training quality of word vectors and expanding the vocabulary to build a larger word vector space. N-gram is a model that takes into account the context of text sequences. It can capture local grammatical and lexical co-occurrence information in the text, which helps improve the model's ability to understand log data.

[0104] After processing the original log data into word vectors to represent text, the pre-trained log data anomaly detection model is used to perform anomaly detection on the processed log data.

[0105] Specifically, in the data anomaly detection stage, a pre-trained log data anomaly detection model is used to identify anomalies in the processed log data. The model is an advanced network structure based on a recurrent neural network (RNN) and an integrated attention mechanism. The recurrent neural network is suitable for processing sequence data, while the attention mechanism can highlight important information, thereby improving the accuracy of detection.

[0106] In order to train this model, we first need to obtain the log data of IoT terminal devices through IoT data collection, and then convert the original log data into word vector representation text according to the above data analysis, data preprocessing and feature processing processes to build a log data training set;

[0107] During the model training phase, the model is learned using the constructed log data training set, and then the accuracy of the model is verified by calculating the formula Evaluate the performance of the model; PR represents the precision value of the log data anomaly detection model, TP represents the number of sample data detected correctly, and FP represents the number of sample data detected incorrectly;

[0108] When the accuracy verification result is greater than 0.95, the accuracy of the model is considered acceptable; if it is lower than 0.95, it indicates that the model needs further optimization and adjustment, and finally a pre-trained log data anomaly detection model is obtained.

[0109] In this embodiment, the log data anomaly detection model mainly consists of the following parts:

[0110] Input layer: receives the word vector representation text corresponding to the log event template;

[0111] Bi-GRU (Bidirectional Gated Recurrent Unit) layer: used to capture bidirectional semantic dependencies in the word vector representation text and take into account the contextual information in the log sequence;

[0112] Attention layer: used to learn the local features of log sequences and assign different weights to different log event templates, helping the model focus more on the information most relevant to anomaly detection;

[0113] Output layer: responsible for outputting the final detection results and identifying which data are abnormal.

[0114] The log data anomaly detection model is built based on a recurrent neural network and introduces an attention mechanism. It comprehensively utilizes the advantages of deep learning in sequence data processing and the ability of the attention mechanism in importance weighting, making anomaly detection both efficient and accurate.

[0115] Issue an alarm notification based on the abnormal detection results and record the alarm process, including:

[0116] Record the abnormality of the detection results, including abnormal log content, abnormal context, record sequence number, timestamp and abnormal value;

[0117] Manage the alarm rules preset by the terminal equipment management personnel; the alarm rules include the alarm rule name, filtering alarm level, record sequence number and alarm handler;

[0118] Perform alarm rule matching operations on abnormal detection results and alarm rules to obtain matching results;

[0119] Alarm record, based on the matching results, sends alarm notification to the corresponding alarm handler and records the alarm process.

[0120] See also Figure 5-Figure 9 The second aspect of the present invention provides a data security anomaly detection system based on AI technology, including: a log data acquisition unit 100, a log data processing unit 200, a log anomaly data detection unit 300 and a system alarm unit 400.

[0121] In this embodiment, the log data acquisition unit 100 is used to monitor, collect and convert data of log files, and specifically includes: a log data collection module 101, which is used to monitor, collect and perform structured processing on the original log data of the terminal device to obtain structured log data;

[0122] The log data storage module 102 is used to store the collected data in a structured form;

[0123] The log data parsing module 103 is used to parse the stored data.

[0124] The log data processing unit 200 is used to further process the stored and parsed log data, including: a log data preprocessing module 201, which is used to perform necessary preprocessing operations on the log event template;

[0125] The log data feature processing module 202 converts these log event templates into digital symbolic forms to facilitate subsequent analysis and detection operations.

[0126] The log data anomaly detection unit 300 is used to detect anomalies in data, including: an anomaly detection model design module 301, which builds a log data anomaly detection model based on a recurrent neural network and an integrated attention mechanism;

[0127] A data set construction module 302 is used to generate the processed log data into word vector representation text to create a data set for training the model;

[0128] A model training module 303 is used to train an anomaly detection model using these data sets and evaluate the performance of the model through accuracy verification;

[0129] The model output result acquisition module 304 is used to use the trained model to perform anomaly detection on the data and generate detection results.

[0130] The system alarm unit 400 is used to perform an alarm operation according to the detection result, including: an abnormality recording module 401, which is used to record detailed information of the abnormality detection result, such as abnormality log content, abnormality context, record sequence number, timestamp and abnormal value;

[0131] The alarm rule management module 402 is used to manage preset alarm rules, including alarm rule name, filter alarm level, record sequence number and alarm handler;

[0132] The alarm rule matching module 403 is used to match the detection result with the alarm rule to determine whether an alarm is required;

[0133] The alarm recording module 404 is used to send an alarm notification to a corresponding processor according to the matching result and record the alarm process.

[0134] Based on the above modules, the system can efficiently detect and respond to data security issues and improve the intelligence level of data management.

[0135] Part of the data in the above formula is calculated by removing the dimension and taking its numerical value. The formula is a formula closest to the actual situation obtained by software simulation of a large amount of collected data; the preset parameters and preset thresholds in the formula are set by technical personnel in this field according to actual conditions or obtained through simulation of a large amount of data.

[0136] Working principle of the present invention:

[0137] S1: log data is collected by the log data collection module, structured log data is stored by the log data storage module, stored log data is parsed by the log data parsing module, processed log data is preprocessed by the log data preprocessing module, and preprocessed log event templates are digitally symbolized by the log data feature processing module;

[0138] S2: Build a log data anomaly detection model through the anomaly detection model design module, build a log data training set for log data anomaly detection model training through the data set construction module, train the log data anomaly detection model and verify its accuracy through the model training module, perform anomaly detection operations through the model output result acquisition module, and obtain the detection results;

[0139] S3: The detection results obtained in the model output result acquisition module are recorded through the log data anomaly recording module, the alarm rules preset by the IoT terminal equipment management personnel are managed through the alarm rule management module, the alarm rule matching operation is performed based on the anomaly detection results and the alarm rules through the alarm rule matching module, and the alarm notification is sent to the alarm handler of the corresponding alarm rule based on the alarm rule matching result through the alarm recording module, and the alarm process is recorded.

[0140] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. Data security anomaly detection method based on AI technology, It is characterized in that The following steps are involved: Monitoring, collecting and structured processing of original log data of terminal devices to obtain structured log data, performing storage management and data analysis on the structured log data to obtain log event templates; Preprocessing and feature processing are performed on the log event template to obtain a log sequence and a word vector space; Using a pre-trained log data anomaly detection model to perform anomaly detection on the word vector space to obtain an anomaly detection result; An alarm notification is issued based on the abnormal detection result and the alarm process is recorded.

2. According to the data security anomaly detection method based on AI technology according to claim 1, It is characterized in that The structured log data is stored and managed and analyzed, including the following steps: Storing the structured log data and establishing a data index; Perform visual analysis on stored log data; Parsing the stored log data into a sequence of string groups; The string group sequence is compared with the log event templates stored in the current LCSMap to find a log event template that matches the string group sequence.

3. According to claim 1, the data security anomaly detection method based on AI technology, It is characterized in that The using a pre-trained log data anomaly detection model to perform anomaly detection on the word vector space includes: Design a log data anomaly detection model: Build a log data anomaly detection model based on a recurrent neural network and introduce an attention mechanism; Constructing a data set: Using IoT data collection technology to obtain log data from terminal devices, processing the log data into word vector representation text, and constructing a log data training set for model training; Training model: Use the log data training set to train the log data anomaly detection model, and calculate the model accuracy according to the model accuracy formula: Perform accuracy verification, optimize and adjust the model according to the accuracy verification results, and obtain the pre-trained log data anomaly detection model; PR represents the accuracy value of the log data anomaly detection model, TP represents the number of sample data detected correctly, and FP represents the number of sample data detected incorrectly; Output model results: Use the pre-trained log data anomaly detection model to perform an anomaly detection operation on the word vector representation text and obtain the detection result.

4. According to claim 1, the data security anomaly detection method based on AI technology, It is characterized in that The log data anomaly detection model includes: An input layer is used to receive a word vector representation text corresponding to the log event template; A Bi-GRU layer, used to capture the bidirectional semantic dependency of the word vector representation text and the context information of the log sequence; An attention layer, used to learn local features of the log sequence and assign different weights to log event templates in the sequence; The output layer is used to output the detection results; wherein the detection results include abnormal detection results and normal detection results.

5. According to claim 1, the data security anomaly detection method based on AI technology, It is characterized in that Providing an alarm notification based on the abnormal detection result and recording the alarm process include: Recording the abnormality of the detection result; wherein the content of the abnormality record includes abnormal log content, abnormal context, record sequence number, timestamp and abnormal value; Manage the alarm rules preset by the terminal equipment management personnel; wherein the alarm rules include the alarm rule name, filtering alarm level, record sequence number and alarm handler; Matching the abnormal detection result with the alarm rule to obtain a matching result; Alarm record, based on the matching result, sends an alarm notification to the corresponding alarm handler and records the alarm process.

6. Data security anomaly detection system based on AI technology, It is characterized in that It comprises a log data acquisition unit (100), a log data processing unit (200), a log abnormal data detection unit (300) and a system alarm unit (400); The log data acquisition unit (100) comprises: a log data collection module (101), a log data storage module (102) and a log data parsing module (103); The log data processing unit (200) comprises: a log data preprocessing module (201) and a log data feature processing module (202); The log abnormal data detection unit (300) comprises: an abnormal detection model design module (301), a data set construction module (302), a model training module (303) and a model output result acquisition module (304); The system alarm unit (400) comprises: an abnormality recording module (401), an alarm rule management module (402), an alarm rule matching module (403) and an alarm recording module (404).

7. The data security anomaly detection system based on AI technology according to claim 6, It is characterized in that The log data collection module (101) is used to monitor, collect and perform structured processing on the original log data of the terminal device to obtain structured log data; The log data storage module (102) is used to store the structured log data; The log data parsing module (103) is used to perform a parsing operation on the stored structured log data to obtain a log event template.

8. The data security anomaly detection system based on AI technology according to claim 7, It is characterized in that The log data preprocessing module (201) is used to perform preprocessing operations on the parsed log event template; The log data feature processing module (202) is used to perform digital symbolization operations on the pre-processed log event template to obtain a word vector representation text.

9. The data security anomaly detection system based on AI technology according to claim 6, It is characterized in that The anomaly detection model design module (301) is used to construct a log data anomaly detection model based on a recurrent neural network and introducing an attention mechanism; The data set construction module (302) is used to process the log data obtained from the terminal device into a word vector representation text, and to construct a log data training set for model training; The model training module (303) is used to perform model training on the log data anomaly detection model using the log data training set, and perform accuracy verification according to the model accuracy calculation formula, and optimize and adjust the model according to the accuracy verification result to obtain a pre-trained log data anomaly detection model; The model output result acquisition module (304) is used to use the pre-trained log data anomaly detection model to perform an anomaly detection operation on the word vector representation text and obtain the detection result.

10. The data security anomaly detection system based on AI technology according to claim 9, It is characterized in that The abnormality recording module (401) is used to record the abnormality of the detection result; wherein the abnormality record content includes abnormal log content, abnormal context, record sequence number, timestamp and abnormal value; An alarm rule management module (402) is used to manage the alarm rules preset by the terminal equipment management personnel; wherein the alarm rules include the alarm rule name, the filtering alarm level, the record sequence number and the alarm handler; An alarm rule matching module (403) is used to perform an alarm rule matching operation on the abnormal value in the detection result and the alarm rule to obtain a matching result; The alarm recording module (404) is used to send an alarm notification to the corresponding alarm handler based on the matching result and record the alarm process.