Network anomaly detection method, device, equipment and medium based on deep learning

Abstract

The present invention discloses a network anomaly detection method, device, equipment and medium based on deep learning, the method comprising: in response to the received initial log data, extracting features from the initial log data according to a preset constant event template to generate a template symbol sequence, segmenting the template symbol sequence by setting a sliding rule according to a preset sliding window to generate a target log sequence and a label template symbol, detecting the target log sequence according to a preset anomaly detection model to generate a target template symbol corresponding to the target log sequence, and determining that the network behavior corresponding to the initial log data is abnormal when the target template symbol does not match the label template symbol. Thus, by parsing and extracting features from the log data, determining the log data abnormality according to the anomaly detection model, and then detecting the network abnormal behavior caused by the network attack, the efficiency of network anomaly detection is improved.

Description

Technical Field

[0001] The present invention relates to the field of data detection technology, and specifically to a network anomaly detection method, device, equipment and medium based on deep learning. Background Art

[0002] In the related art, the detection of network behavior anomalies is to parse the log data and determine the anomalies of the network behavior based on the anomalies of the log data. Network behavior anomaly detection based on runtime logs is one of the key building blocks to ensure system security and post-incident investigation. However, due to the large number of types of user network behaviors in the network system, the richness of log data is high, and the data required for anomaly detection is rich, resulting in low detection efficiency of network behavior anomaly detection based on log data on existing network systems. Summary of the invention

[0003] In view of the technical problem that the efficiency of network anomaly detection based on deep learning in the prior art is low, the present invention provides a network anomaly detection method, device, equipment and medium based on deep learning.

[0004] To achieve the above objectives, the present invention is implemented through the following technical solutions:

[0005] According to a first aspect of an embodiment of the present invention, a network anomaly detection method based on deep learning is provided, the method comprising:

[0006] In response to the received initial log data, extracting rules from the initial log data according to a preset constant event template to generate a template symbol sequence;

[0007] The template sequence is segmented by setting a sliding rule according to a preset sliding window to generate a target log sequence and a label template, wherein the label template is the next template after the target log sequence in the template sequence;

[0008] Detecting the target log sequence according to a preset anomaly detection model to generate a target template corresponding to the target log sequence;

[0009] In the case that the target template character does not match the label template character, it is determined that the network behavior corresponding to the initial log data is abnormal.

[0010] Optionally, the preset anomaly detection model is generated by the following steps:

[0011] Acquire a template symbol training sequence of the preset anomaly detection model, wherein the template symbol training sequence includes a plurality of template symbols;

[0012] According to the preset sliding window and the set sliding rule, the template symbol training sequence is segmented to generate a training log sequence and a training label template symbol, wherein the training label template symbol is the next template symbol after the training log sequence in the template symbol training sequence;

[0013] Performing feature extraction on the training log sequence to generate a sequence vector and a count vector corresponding to the training log sequence;

[0014] Model training is performed on the preset anomaly detection model according to the sequence vector, the count vector and the training label template to generate the trained preset anomaly detection model.

[0015] Optionally, the performing model training on the preset anomaly detection model according to the sequence vector, the count vector and the training label template to generate the trained preset anomaly detection model includes:

[0016] generating a first target vector based on the sequence vector, and generating a second target vector based on the count vector;

[0017] Inputting the first target vector and the training label template into the temporal convolutional neural network module of the preset anomaly detection model to generate a first hidden vector;

[0018] Inputting the second target vector and the training label template into the ring convolutional neural network module of the preset anomaly detection model to generate a second hidden vector;

[0019] Model training is performed on the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate the trained preset anomaly detection model.

[0020] Optionally, performing model training on the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate the trained preset anomaly detection model includes:

[0021] Inputting the first hidden vector and the second hidden vector into a fully connected neural network module of the preset anomaly detection model to generate a neural network output result;

[0022] The output result of the neural network is transmitted to the output layer of the preset anomaly detection model to generate the trained preset anomaly detection model.

[0023] Optionally, the response to the received initial log data is to extract rules from the initial log data according to a preset constant event template to generate a template symbol sequence, including:

[0024] According to the preset constant event template, extracting rules from the initial log data to generate an event database corresponding to the initial log data;

[0025] The template sequence is generated according to the sequence of the event database in the initial log data.

[0026] Optionally, the segmenting the template sequence by setting a sliding rule according to a preset sliding window to generate a target log sequence and a label template includes:

[0027] Obtaining the window length and sliding step size of the preset sliding window;

[0028] The template sequence is segmented according to the window length and the sliding step size to generate the target log sequence and the label template.

[0029] Optionally, the detecting the target log sequence according to a preset anomaly detection model to generate a target template symbol corresponding to the target log sequence includes:

[0030] Detect the target log sequence according to the preset anomaly detection model to determine a plurality of initial candidate template symbols;

[0031] A plurality of candidate template symbols with the highest probability are determined from the plurality of initial candidate template symbols as the target template symbols.

[0032] According to a second aspect of an embodiment of the present invention, a network anomaly detection device based on deep learning is provided, the device comprising:

[0033] A determination module, configured to respond to the received initial log data and extract rules from the initial log data according to a preset constant event template to generate a template symbol sequence;

[0034] A first generating module is used to segment the template sequence by setting a sliding rule according to a preset sliding window to generate a target log sequence and a label template, wherein the label template is the next template after the target log sequence in the template sequence;

[0035] A second generating module detects the target log sequence according to a preset anomaly detection model to generate a target template symbol corresponding to the target log sequence;

[0036] The execution module is used to determine that the network behavior corresponding to the initial log data is abnormal when the target template symbol does not match the label template symbol.

[0037] According to a third aspect of an embodiment of the present invention, there is provided an electronic device, including:

[0038] a memory having a computer program stored thereon;

[0039] A processor is used to execute the computer program in the memory to implement the steps of the deep learning-based network anomaly detection method in any one of the first aspects of the present disclosure.

[0040] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a terminal application control program is stored. When the terminal application control program is executed by a processor, the steps of the deep learning-based network anomaly detection method described in any one of the first aspects of the present disclosure are implemented.

[0041] The present invention provides a network anomaly detection method, device, electronic device and storage medium based on deep learning. Compared with the prior art, it has the following beneficial effects:

[0042] In the above manner, in response to the received initial log data, according to the preset constant event template, the initial log data is subjected to extraction rules to generate a template symbol sequence, and the template symbol sequence is segmented by setting sliding rules according to the preset sliding window to generate a target log sequence and a label template symbol, wherein the label template symbol is the next template symbol after the target log sequence in the template symbol sequence, and the target log sequence is detected according to the preset anomaly detection model to generate a target template symbol corresponding to the target log sequence, and when the target template symbol does not match the label template symbol, the network behavior corresponding to the initial log data is determined to be abnormal. Thus, by parsing and extracting features from the log data, the log data anomaly is determined according to the anomaly detection model, and then the abnormal network behavior caused by the network attack is detected, thereby improving the efficiency of network anomaly detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 The present invention provides a flowchart of a network anomaly detection method based on deep learning.

[0044] Figure 2 The present invention provides another flowchart of a network anomaly detection method based on deep learning.

[0045] Figure 3 A block diagram of a network anomaly detection device based on deep learning provided according to the present invention. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0047] Figure 1 For a flowchart of a network anomaly detection method based on deep learning provided by the present invention, please refer to Figure 1 The present invention provides a network anomaly detection method based on deep learning, and the method comprises the following steps.

[0048] In step S11, in response to the received initial log data, the initial log data is subjected to extraction rules according to a preset constant event template to generate a template symbol sequence.

[0049] For example, this embodiment is applied to a terminal device, and the initial log data is obtained by collecting the log data generated by the user device. The log data is an unstructured text, including two parts: a constant part named event template and a change part named log parameter. The constant part indicates the amount that will not change in the recorded log data, and the log parameter part indicates the amount that will change in the recorded log data based on the difference of data on each day. For example, the log message "081109 203615 148INFO dfs data node packet responder: packet responder 1 terminated for block blk_38865049064139660" records the event "packet responder 1 terminated for block blk_3886509906413966", wherein "packet responder", "for", "block", "blk_", and "terminate" are constant event templates; "1" and "38865049064139660" are log parameters. The terminal device can determine the event template in the initial log data by comparing the initial log data.

[0050] By way of example, in this embodiment, a preset constant event template extraction rule is set in the terminal device, which includes a mapping relationship between multiple event templates and multiple template symbols. The template symbol in the event template is extracted through the mapping relationship, thereby determining the template symbol corresponding to the event template, and generating a template symbol sequence corresponding to the event template according to the arrangement order of the template symbol in the event template. It is worth mentioning that in this embodiment, when extracting the template symbol, according to the order of the event template in the initial log data, feature extraction is performed from the event template in sequence, thereby generating a template symbol sequence, wherein the template symbol sequence may include multiple template symbols, and the order of each template symbol is consistent with the order of the event template in the initial log data. In the above manner, unstructured log data is converted into structured data. By way of example, an event template database can be generated according to the event template, and a mapping relationship between the event template and the template symbol can be established in the template database.

[0051] Optionally, the above step S11 includes:

[0052] According to the preset constant event template, extracting rules from the initial log data to generate an event database corresponding to the initial log data;

[0053] The template sequence is generated according to the sequence of the event database in the initial log data.

[0054] For example, the goal of log parsing in this embodiment is to extract event templates and log parameters from each log data, and determine multiple event templates in the initial log data based on the preset constant event template, and generate an event database. And according to the order of each event template in the initial log data, a template symbol sequence corresponding to the initial log data is generated.

[0055] In step S12, the template sequence is segmented by setting a sliding rule according to a preset sliding window to generate a target log sequence and a label template.

[0056] For example, in this embodiment, the template sequence is divided into a preset sliding window, and the template sequence is divided according to the preset sliding window to generate a target log sequence and a label template. Among them, the label template is the next template after the target log sequence in the template sequence. The preset sliding window is an active sliding window, which includes a preset length and a unit step size for each slide. After the template sequence is divided into the preset sliding window, the template sequence is divided according to the preset length and unit step size of the window, and based on the sliding order of the preset sliding window, multiple target log sequences and multiple label templates corresponding to each other can be generated. It is worth mentioning that in this embodiment, when the template sequence is divided by the preset sliding window, the template sequence can be divided into multiple log sequences according to the length of the template sequence, the window length and the unit sliding step. For example, when the number of log information corresponding to the template sequence is 6, they are: e1, e2, e3, e4, e2, e3; when the window length corresponding to the preset sliding window is 2 and the unit sliding step is 1, the template sequence is divided according to the preset sliding window to generate 5 target log sequences, which are: e1e2, e2e3, e3e4, e4e2, e2e3, and the label templates corresponding to the 5 target log sequences are determined to be: e2, e3, e4, e2, e3. The window length and unit sliding step of the preset sliding window can be set according to actual needs, and there is no limitation on this in this embodiment.

[0057] Optionally, the above step S12 includes:

[0058] Obtaining the window length and sliding step size of the preset sliding window set;

[0059] The template sequence is segmented according to the window length and the sliding step size to generate the target log sequence and the label template.

[0060] For example, in this embodiment, the window length and sliding step of the preset sliding window are determined, and the template sequence is divided according to the window length and sliding step until the preset sliding window slides to the last template of the template sequence, thereby dividing the template sequence into multiple target log sequences and multiple label templates. It can be understood that when dividing the template sequence according to the preset sliding window, it is necessary to divide the template sequence into multiple target log sequences according to the window length and sliding step of the preset sliding window, and record the next template of the target log sequence in the template sequence as the label template corresponding to the target log sequence. Among them, the number of target log sequences is related to the window length, sliding step of the preset sliding window and the length of the template sequence.

[0061] In step S13, the target log sequence is detected according to a preset anomaly detection model to generate a target template symbol corresponding to the target log sequence.

[0062] For example, in this embodiment, after the target log sequence is generated through the above steps, the next template symbol of the target log sequence in the template symbol sequence is used as the label template symbol, and the target log sequence is detected by the preset anomaly detection model, thereby generating a target template symbol corresponding to the target log sequence. Among them, the target template symbol is the next template symbol predicted after the target log sequence in the template symbol sequence after the target log sequence is detected by the preset anomaly detection model. Among them, when there are multiple target log sequences, the preset anomaly detection model will predict the next template symbol of multiple target log sequences, thereby generating target template symbols corresponding to multiple target log sequences one by one.

[0063] Optionally, in one implementation, the preset anomaly detection model may be generated by the following steps, including:

[0064] Acquire a template symbol training sequence of the preset anomaly detection model, wherein the template symbol training sequence includes a plurality of template symbols;

[0065] According to the preset sliding window and the set sliding rule, the template symbol training sequence is segmented to generate a training log sequence and a training label template symbol, wherein the training label template symbol is the next template symbol after the training log sequence in the template symbol training sequence;

[0066] Performing feature extraction on the training log sequence to generate a sequence vector and a count vector corresponding to the training log sequence;

[0067] Model training is performed on the preset anomaly detection model according to the sequence vector, the count vector and the training label template to generate the trained preset anomaly detection model.

[0068] It is worth mentioning that after obtaining the template symbol training sequence of the preset anomaly detection model in this embodiment, the preset anomaly detection model is trained according to the template symbol training sequence, wherein the template symbol training sequence is generated after processing the training log data, and the processing method of the training log data can refer to the processing method of the initial log data in the above scheme, which will not be repeated in this embodiment.

[0069] Referring to the segmentation method of the above-mentioned template symbol sequence, the template symbol training sequence is segmented by a preset sliding window and a set sliding rule to generate a training log sequence and a training label template symbol, wherein the segmentation method of the template symbol training sequence can refer to the segmentation method of the template symbol sequence in the above-mentioned scheme, which will not be repeated in this embodiment. After the template symbol training sequence is segmented into multiple training log sequences and multiple one-to-one corresponding training label template symbols by a preset sliding window, feature extraction is performed on the training log sequence, and a sequence vector and a count vector corresponding to the training log sequence are calculated and generated, wherein the sequence vector i identifies the sequence of each template symbol in the training log sequence, and the count vector identifies the number of times each different type of template symbol appears in the training log sequence. The sequence vector, the count vector and the training label template symbol are input into a preset anomaly detection model, and the preset anomaly detection model is trained. Through a deep learning method, a corresponding relationship between the training log sequence and the training label template symbol is established, thereby generating a trained preset anomaly detection model, and the network abnormal behavior detection is performed on the initial log data according to the trained preset anomaly detection model.

[0070] Optionally, in another implementation, the above step performs model training on the preset anomaly detection model according to the sequence vector, the count vector and the training label template to generate the trained preset anomaly detection model, including:

[0071] generating a first target vector based on the sequence vector, and generating a second target vector based on the count vector;

[0072] Inputting the first target vector and the training label template into the temporal convolutional neural network module of the preset anomaly detection model to generate a first hidden vector;

[0073] Inputting the second target vector and the training label template into the ring convolutional neural network module of the preset anomaly detection model to generate a second hidden vector;

[0074] Model training is performed on the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate the trained preset anomaly detection model.

[0075] For example, in this embodiment, when the preset sliding window is swiped to generate a training log sequence, the next template symbol of the training log sequence in the template symbol training sequence is recorded and used as a label to train the preset anomaly detection model, that is, the preset anomaly detection model is trained by the training log sequence and the training label template symbol. After the corresponding sequence vector and count vector are generated according to the training log sequence through the above steps, the sequence vector is converted into a first target vector according to the training log sequence:

[0076] s t =[m t-h , m t-h+1 , ..., m t-1 ]

[0077] Among them, m i represents the value of the template symbol at timestamp i, and h is the window length of the preset sliding window;

[0078] The set of n different template symbols for each training log sequence is identified by the following method (where m i ∈E n ):

[0079] E n ={e1, e2, ..., e n}

[0080] According to each template symbol e in each training log sequence n The number of times it appears in the training log sequence generates a count vector corresponding to the training log sequence, where the count vector can be expressed as:

[0081] p t =[count(e1),count(e2),...,count(e n )]

[0082] Among them, count(e i ) represents the template symbol e in the training log sequence i The number of occurrences, for example, given a training log sequence of [1, 2, 3], by comparing the corresponding relationship of the set template symbol, the sequence vector of the training log sequence can be determined as s according to the training log sequence. t =[e1, e2, e1], according to each template symbol e in the sequence vector i The number of occurrences determines the corresponding counting vector: p t =[2, 1, 0, 0, 0], where n = 5.

[0083] The sequence vector s t and the count vector p t They are respectively input into the TCN module and ACNN module of the preset anomaly detection model to learn hidden representations, and the TCN module and ACNN module are connected. The output results of the TCN module and ACNN module are provided to the MLP module of the preset anomaly detection model. The MLP module consists of two fully connected TCN modules and ACNN modules. For example:

[0084] h1=TCN(s t )

[0085] h2=ACNN(pt )

[0086] τ=MLP(h1;h2)

[0087] Among them, h1 is the first hidden vector and h2 is the second hidden vector, τ represents the output of the MLP module, and then τ is forwarded to the output layer of the preset anomaly detection model, and the preset anomaly detection model is trained based on the τ of the output layer to generate a trained preset anomaly detection model. It should be noted that the preset anomaly detection model generated after training in this embodiment is used to detect the label template symbol after the log sequence in the template symbol sequence according to the log sequence, and compare the generated target template symbol with the label template symbol after the log sequence in the template symbol sequence. When the target template symbol matches the label template symbol, it is determined that there is no network abnormal behavior in the initial log data. When the target template symbol does not match the label template symbol, it is determined that there is network abnormal behavior in the initial log data.

[0088] Optionally, in another implementation, the above step performs model training on the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate the trained preset anomaly detection model, including:

[0089] Inputting the first hidden vector and the second hidden vector into a fully connected neural network module of the preset anomaly detection model to generate a neural network output result;

[0090] The neural network output result is transmitted to the output layer of the preset anomaly detection model, and the preset anomaly detection model is trained based on the output layer to generate the trained preset anomaly detection model.

[0091] For example, in this embodiment, after the first latent vector h1 and the second latent vector h2 are generated through the above steps, h1 and h2 are fused through the fully connected neural network module of the preset anomaly detection model, that is, the MLP module, to generate a neural network output result τ, and then the τ is forwarded to the output layer of the preset anomaly detection model, and the preset anomaly detection model is trained based on the τ of the output layer, thereby establishing a corresponding relationship between the training log sequence and the training label template symbol, and generating a trained preset anomaly detection model.

[0092] Optionally, the above step S13 includes:

[0093] Detect the target log sequence according to the preset anomaly detection model to determine a plurality of initial candidate template symbols;

[0094] A plurality of candidate template symbols with the highest probability are determined from the plurality of initial candidate template symbols as the target template symbols.

[0095] By way of example, in this embodiment, the target log sequence is detected by a preset anomaly detection model to determine the multiple initial candidate template symbols corresponding to the target log sequence, wherein the multiple initial candidate template symbols are label template symbols that may appear after the target log sequence in the template symbol sequence. According to the number of times the multiple initial candidate template symbols appear, the multiple initial candidate template symbols are sorted, wherein the more times the template symbols appear, the higher the corresponding probability, and the higher the corresponding sorting, and the multiple candidate template symbols that appear the most times in the sequence are used as target template symbols. By way of example, the number of candidate template symbols to be extracted is determined to be 3, and after obtaining the probability order of the candidate template symbols according to the sorting, the first three candidate template symbols with the highest probability of occurrence are extracted as target template symbols.

[0096] In step S14, when the target template symbol does not match the label template symbol, it is determined that the network behavior corresponding to the initial log data is abnormal.

[0097] For example, in this embodiment, the target template symbol generated by the preset anomaly detection model is compared with the next template symbol (label template symbol) corresponding to the target log sequence in the template symbol sequence. When it is determined that the label template symbol and the target template symbol do not match, that is, the target template symbol generated by the preset anomaly detection model is not the next template symbol corresponding to the target log sequence in the template symbol sequence, then the network behavior corresponding to the initial log data is determined to be abnormal.

[0098] Through the above scheme, in response to the received initial log data, the initial log data is parsed to determine the event template in the initial log data, and the event template is extracted according to the preset constant event template extraction rule to generate a template symbol sequence, and the template symbol sequence is segmented according to the preset sliding window by setting the sliding rule to generate a target log sequence and a label template symbol, wherein the label template symbol is the next template symbol after the target log sequence in the template symbol sequence, and the target log sequence is detected according to the preset anomaly detection model to generate a target template symbol corresponding to the target log sequence, and when the target template symbol does not match the label template symbol, the network behavior corresponding to the initial log data is determined to be abnormal. Thus, by parsing and extracting features from the log data, the log data anomaly is determined according to the anomaly detection model, and then the network abnormal behavior caused by the network attack is detected, thereby improving the efficiency of network anomaly detection.

[0099] Figure 2 Flow chart of another network anomaly detection method based on deep learning provided according to the present invention. Figure 2 As shown, the network anomaly detection method based on deep learning includes:

[0100] (1) Log parsing: Log messages are unstructured text, consisting of two parts: a constant part called event template and a variable part called log parameters. The goal of log parsing is to extract event templates and log parameters from each log message and create a dictionary that maps each event template to a unique template symbol. In this way, unstructured log messages will be converted into structured data. Figure 2 An example of parsing HDFS logs is shown. For example, the log message "081109 203615 148INFO dfs datanode packet responder: packet responder 1 for block blk_38865049064139660 terminated" records the event "packet responder 1 for block blk_3886509906413966 terminated", where the words "packet responder", "for", "block", "blk_", "terminated" are constant event templates extracted as "packet responder <*> for block blk_<*> terminated". Note that <*> represents the log parameters, that is, "1" and "38865049064139660".

[0101] (2) Feature extraction: The purpose of feature extraction is to convert structured log data into numerical vectors. First, the log data is divided into a set of sliding windows. In this set of sliding windows, the log messages located in the same window form a log sequence. Figure 2 As shown in the figure, if the log message series is [1,2,3,4,5], the window size is set to 3, and the step size is set to 1, we can get three log sequences, [1,2,3], [,3,4], and [,4,5]. Further, two vectors are calculated based on each derived log sequence. The sequence vector represents the sequence of template symbols in the log sequence, and the count vector represents the number of times each template symbol appears in the log sequence. In addition, the next template symbol of each window is recorded as a label to train the anomaly detection model. Specifically, given a log sequence, the sequence vector is represented as:

[0102] s t =[m t-h , m t-h+1 , ..., m t-1 ]

[0103] Among them, m i represents the value of the template symbol at timestamp i, and h is the window size of the preset sliding window set;

[0104] The set of n different templates in the log data set is identified by the following method (where m i ∈E n ):

[0105] En ={e1, e2, ..., e n}

[0106] The count vector is represented as:

[0107] p t =[count(e1),count(e2),...,count(e n )]

[0108] Among them, count(e i ) represents the number of times each template symbol appears in the log sequence. For example, given a log sequence [1, 2, 3], we can determine s based on the log sequence. t =[e1, e2, e3] and p t =[2, 1, 0, 0, 0], where n = 5.

[0109] The sequence vector S t and the count vector p t They are respectively input into the TCN module and ACNN module of the preset anomaly detection model to learn hidden representations, and the TCN module and ACNN module are connected. The output results of the TCN module and ACNN module are provided to the MLP module of the preset anomaly detection model. The MLP module consists of two fully connected TCN modules and ACNN modules. For example:

[0110] h1=TCN(s t )

[0111] h2=ACNN(p t )

[0112] τ=MLP(h1;h2)

[0113] Among them, h1 and h2 are hidden vectors, τ represents the output of the MLP module, and then τ is forwarded to the output layer of the preset anomaly detection model. Given the past window W, we get m t The probability distribution of , and through:

[0114] P(m t =e i |w)=softmax(τ)

[0115] During training, P(m t =e i |w) and the labels obtained from the feature extraction part. SGD is used to train the parameters of the TCN-ACNN model. During inference, P(m t =e i |w) to sort and select e iThe first c template symbols with the highest probability are used as multiple candidate template symbols.

[0116] If the template of the incoming log message is not one of these candidates, then The log sequence containing this log message is classified as an anomaly.

[0117] (3) Anomaly Detection: Anomaly detection focuses on identifying data patterns that are significantly different from normal observations. The goal of log-based anomaly detection is to reveal collective anomalies, i.e., abnormal log events that appear in a sequence. Predict log messages relative to historical log sequences. Specifically, if the template of the incoming log message is different from the predicted template, the log sequence containing the log message will be considered anomalous.

[0118] Figure 3 A block diagram of a network anomaly detection device based on deep learning provided according to the present invention is shown in FIG. Figure 3 As shown, the device 100 includes: a determination module 110, a first generation module 120, a second generation module 130 and an execution module 140.

[0119] The determination module 110 is used to respond to the received initial log data and extract rules from the initial log data according to a preset constant event template to generate a template symbol sequence;

[0120] A first generating module 120 is used to segment the template sequence by setting a sliding rule according to a preset sliding window to generate a target log sequence and a label template, wherein the label template is the next template after the target log sequence in the template sequence;

[0121] A second generating module 130 is used to detect the target log sequence according to a preset anomaly detection model to generate a target template symbol corresponding to the target log sequence;

[0122] The execution module 140 is used to determine that the network behavior corresponding to the initial log data is abnormal when the target template symbol does not match the label template symbol.

[0123] Optionally, the preset anomaly detection model is generated by the following steps:

[0124] Acquire a template symbol training sequence of the preset anomaly detection model, wherein the template symbol training sequence includes a plurality of template symbols;

[0125] According to the preset sliding window and the set sliding rule, the template symbol training sequence is segmented to generate a training log sequence and a training label template symbol, wherein the training label template symbol is the next template symbol after the training log sequence in the template symbol training sequence;

[0126] Performing feature extraction on the training log sequence to generate a sequence vector and a count vector corresponding to the training log sequence;

[0127] Model training is performed on the preset anomaly detection model according to the sequence vector, the count vector and the training label template to generate the trained preset anomaly detection model.

[0128] Optionally, the performing model training on the preset anomaly detection model according to the sequence vector, the count vector and the training label template to generate the trained preset anomaly detection model includes:

[0129] generating a first target vector based on the sequence vector, and generating a second target vector based on the count vector;

[0130] Inputting the first target vector and the training label template into the temporal convolutional neural network module of the preset anomaly detection model to generate a first hidden vector;

[0131] Inputting the second target vector and the training label template into the ring convolutional neural network module of the preset anomaly detection model to generate a second hidden vector;

[0132] Model training is performed on the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate the trained preset anomaly detection model.

[0133] Optionally, performing model training on the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate the trained preset anomaly detection model includes:

[0134] Inputting the first hidden vector and the second hidden vector into a fully connected neural network module of the preset anomaly detection model to generate a neural network output result;

[0135] The neural network output result is transmitted to the output layer of the preset anomaly detection model, and the preset anomaly detection model is trained based on the output layer to generate the trained preset anomaly detection model.

[0136] Optionally, the determining module 110 is configured to:

[0137] Generate an event database according to the event template;

[0138] According to the preset constant event template extraction rule, the template symbols in the event database are extracted to generate the template symbol sequence.

[0139] Optionally, the first generating module 120 is used to:

[0140] Obtaining the window length and sliding step size of the preset sliding window;

[0141] The template sequence is segmented according to the window length and the sliding step size to generate the target log sequence and the label template.

[0142] Optionally, the second generating module 130 is used to:

[0143] Detect the target log sequence according to the preset anomaly detection model to determine a plurality of initial candidate template symbols;

[0144] A plurality of candidate template symbols with the highest probability are determined from the plurality of initial candidate template symbols as the target template symbols.

[0145] In the above manner, in response to the received initial log data, according to the preset constant event template, the initial log data is subjected to extraction rules to generate a template symbol sequence, and the template symbol sequence is segmented by setting sliding rules according to the preset sliding window to generate a target log sequence and a label template symbol, wherein the label template symbol is the next template symbol after the target log sequence in the template symbol sequence, and the target log sequence is detected according to the preset anomaly detection model to generate a target template symbol corresponding to the target log sequence, and when the target template symbol does not match the label template symbol, the network behavior corresponding to the initial log data is determined to be abnormal. Thus, by parsing and extracting features from the log data in the terminal device, and determining the anomaly of the log data according to the anomaly detection model, the efficiency of data anomaly detection is improved.

[0146] In another exemplary embodiment, a computer program product is also provided, which includes a computer program that can be executed by a programmable device, and the computer program has a code portion for executing the above-mentioned deep learning-based network anomaly detection method when executed by the programmable device.

[0147] In another exemplary embodiment, an electronic device is also provided, including:

[0148] a memory having a computer program stored thereon;

[0149] A processor is used to execute the computer program in the memory to perform the steps of the above-mentioned deep learning-based network anomaly detection method.

[0150] Based on the above ideal embodiments of this application, the relevant staff can make various changes and modifications without departing from the technical concept of this application through the above description. The technical scope of this application is not limited to the content in the specification, and its technical scope must be determined according to the scope of the claims.

[0151] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A network anomaly detection method based on deep learning, characterized in that: The method comprises: In response to the received initial log data, extracting rules from the initial log data according to a preset constant event template to generate a template symbol sequence; The template symbol sequence is segmented by setting a sliding rule according to a preset sliding window to generate a target log sequence and a label template symbol, wherein the label template symbol is the next template symbol after the target log sequence in the template symbol sequence; Detecting the target log sequence according to a preset anomaly detection model to generate a target template corresponding to the target log sequence; In the case that the target template character does not match the label template character, it is determined that the network behavior corresponding to the initial log data is abnormal.

2. The method according to claim 1, characterized in that The preset anomaly detection model is generated by the following steps: Acquire a template symbol training sequence of the preset anomaly detection model, wherein the template symbol training sequence includes a plurality of template symbols; According to the preset sliding window and the set sliding rule, the template symbol training sequence is segmented to generate a training log sequence and a training label template symbol, wherein the training label template symbol is the next template symbol after the training log sequence in the template symbol training sequence; Performing feature extraction on the training log sequence to generate a sequence vector and a count vector corresponding to the training log sequence; Model training is performed on the preset anomaly detection model according to the sequence vector, the count vector and the training label template to generate the trained preset anomaly detection model.

3. The method according to claim 2, characterized in that The performing model training on the preset anomaly detection model according to the sequence vector, the count vector and the training label template to generate the trained preset anomaly detection model includes: generating a first target vector based on the sequence vector, and generating a second target vector based on the count vector; Inputting the first target vector and the training label template into the temporal convolutional neural network module of the preset anomaly detection model to generate a first hidden vector; Inputting the second target vector and the training label template into the ring convolutional neural network module of the preset anomaly detection model to generate a second hidden vector; Model training is performed on the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate the trained preset anomaly detection model.

4. The method according to claim 3, characterized in that The performing model training on the preset anomaly detection model according to the first hidden vector and the second hidden vector to generate the trained preset anomaly detection model includes: Inputting the first hidden vector and the second hidden vector into a fully connected neural network module of the preset anomaly detection model to generate a neural network output result; The output result of the neural network is transmitted to the output layer of the preset anomaly detection model to generate the trained preset anomaly detection model.

5. The method according to claim 1, characterized in that The response to the received initial log data is to extract rules from the initial log data according to a preset constant event template to generate a template symbol sequence, including: According to the preset constant event template, extracting rules from the initial log data to generate an event database corresponding to the initial log data; The template sequence is generated according to the sequence of the event database in the initial log data.

6. The method according to claim 1, characterized in that The step of segmenting the template sequence by setting a sliding rule according to the preset sliding window to generate a target log sequence and a label template comprises: Obtaining the window length and sliding step size of the preset sliding window; The template sequence is segmented according to the window length and the sliding step size to generate the target log sequence and the label template.

7. The method according to claim 1, characterized in that The detecting the target log sequence according to the preset anomaly detection model to generate a target template symbol corresponding to the target log sequence includes: Detect the target log sequence according to the preset anomaly detection model to determine a plurality of initial candidate template symbols; A plurality of candidate template symbols with the highest probability are determined from the plurality of initial candidate template symbols as the target template symbols.

8. A network anomaly detection device based on deep learning, characterized in that: The device comprises: A determination module, configured to respond to the received initial log data and extract rules from the initial log data according to a preset constant event template to generate a template symbol sequence; A first generating module is used to segment the template sequence by setting a sliding rule according to a preset sliding window to generate a target log sequence and a label template, wherein the label template is the next template after the target log sequence in the template sequence; A second generating module detects the target log sequence according to a preset anomaly detection model to generate a target template symbol corresponding to the target log sequence; The execution module is used to determine that the network behavior corresponding to the initial log data is abnormal when the target template symbol does not match the label template symbol.

9. An electronic device, characterized in that: include: a memory having a computer program stored thereon; A processor, configured to execute the computer program in the memory to implement the steps of the deep learning-based network anomaly detection method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a terminal application control program, and when the terminal application control program is executed by the processor, the steps of the deep learning-based network anomaly detection method described in any one of claims 1 to 7 are implemented.

Images