Camouflage intrusion detection method and system based on contour hidden horse model

Abstract

The invention provides a camouflage intrusion detection method and system based on a contour hidden horse model, and the method comprises the steps: carrying out the standard formatting processing of a Linux operation command in log data in a data loading and extraction process; performing numerical transcoding mapping on the command stream block in an encoder self-encoding mode to generate the command stream block; the method comprises the following steps of: constructing a user command execution profile (Profile) parameter matrix by using a multi-sequence alignment mode; vectorization or data representation is carried out in a word embedding mode by utilizing a feature engineering module; and setting estimation of the HMM model, outputting model parameters, and performing evaluation, prediction and learning by using the HMM model to obtain a camouflage intrusion detection result. The invention solves the technical problems of single detection mode, lack of flexibility, and low robustness, ductility and host intrusion detection precision of a detection system caused by dependence on a large amount of data annotation.

Description

Technical Field

[0001] The invention relates to the field of network security, and in particular to a camouflage intrusion detection method and system based on a profile hidden horse model. Background Art

[0002] With the complexity and diversity of network attacks, attackers may launch a large number of varied and difficult-to-detect intrusion behaviors at any time with the target host server of the enterprise as the target. The intelligent and concealed nature of the attack methods make the host intrusion detection system (HIDS) currently known to be used face huge challenges. The existing intrusion detection system can be divided into signature-based detection and abnormal behavior-based detection. The signature-based intrusion detection is generally matched according to known rule features. For example, the existing invention patent application document "A System for Automatic Detection of Industrial Network Attacks" with publication number CN111181971A, the existing system includes a data packet capture module, a network traffic profile generation module, a signature-based intrusion detection module, an abnormality detection module and an alarm generation and processing module, which is used to detect network attacks on ICS. From the specific implementation content of the prior art, it can be seen that the signature-based intrusion detection module adopted by the existing solution matches the content of the data packet captured by the data packet capture module with the known attack signature. If any match is found, the corresponding data packet will be regarded as an intrusion. The aforementioned intrusion detection methods have a single detection method and cannot detect unknown abnormal attacks and variants of known attacks, and lack flexibility; intrusion detection based on abnormal behavior uses methods such as behavioral baseline deviation, machine learning and deep learning models. At present, deep neural networks seem to be a positive direction for intrusion detection. For example, the existing invention patent application document "A Network Intrusion Detection Method and System Based on Deep Learning Optimization" with publication number CN117278338A, the existing method includes: preprocessing the original data set collected from the honeypot system to form an experimental data set; dividing the training set and the test set based on the experimental data set to construct a benchmark ELM model; combining the AE autoencoder with the benchmark ELM model to construct an AE-ELM algorithm model; reconstructing the inertia weight of the standard particle swarm algorithm to form a particle swarm optimization algorithm, and using the particle swarm optimization algorithm to optimize the learning parameters of the AE-ELM algorithm model to construct a PSO-AE-ELM algorithm model as an experimental model; comparing the experimental model with the mainstream algorithm model to verify the reliability of the experimental model in network intrusion detection, and using the experimental model for network intrusion detection. However, in the aforementioned existing technologies, deep neural networks still rely on a sufficient number of annotations. On the one hand, the annotation of a large number of data samples requires a lot of manpower and material resources. For small sample training data, the false alarm rate in detection is relatively high and the explanatory power is not strong. On the other hand, the robustness of the model needs to be verified, and the model detection often lacks scalability. For new intrusion behaviors, the original model cannot recognize them and a new model needs to be trained.

[0003] In summary, the existing technology has technical problems such as a single detection method, lack of flexibility, and reliance on a large amount of data annotation, which leads to low robustness, scalability and host intrusion detection accuracy of the detection system. Summary of the invention

[0004] The technical problem to be solved by the present invention is: how to solve the technical problems in the prior art that the detection method is single and lacks flexibility, and relies on a large amount of data annotation, resulting in low robustness, scalability and host intrusion detection accuracy of the detection system.

[0005] The present invention adopts the following technical solutions to solve the above technical problems: a camouflage intrusion detection method based on a contour hidden horse model comprises:

[0006] S1. During the data loading and extraction process, the server's own audit log is obtained to obtain log data, and the Linux operation commands in the log data are formatted in a standard manner to complete the data loading and extraction operations and obtain standardized command data;

[0007] S2, using the encoder self-encoding method to perform command stream block numerical transcoding mapping on the standardized command data, and generate an execution command stream block;

[0008] S3. Compare the global, semi-global and local contents of the pre-coded sequence through multiple sequence alignment to generate comparison scores and similarity scores, take the preset test segment in the standardized command data, align it with the user signature, and build the user command execution profile parameter matrix based on it, and obtain the intrusion indicator data in the preset test segment;

[0009] S4. Using the feature engineering module, in the case of discrete data, input the command stream data of the encoded annotated command to perform a word embedding operation, so as to identify the unit importance and unit weight of each input unit by using a preset tagging method, wherein the preset tagging method includes: Bag-of-words model, Term Frequency-Inverse Document Frequency model TF-IDF and N-gram model;

[0010] S5. Use the hidden Markov model HMM to perform model state transitions and observable event operations under specific states to obtain command flow feature data for setting estimates, training and setting model parameters of the hidden Markov model, and use the hidden Markov model HMM to perform evaluation operations, prediction operations, and learning operations to complete observable event operations under specific states, obtain the command flow feature performance probability matrix, and obtain the camouflage intrusion detection results based on it.

[0011] The present invention uses Profile Hidden Markov Models (PHMM) to perform host intrusion detection. The present invention directly uses operation commands, such as Linux commands, for further analysis. The location information of command execution is considered to perform host disguise intrusion detection, thereby improving the accuracy and precision of host intrusion detection. At the same time, the present invention is different in feature extraction. This application provides a word embedding method based on natural language processing, a word sliding window, a word frequency and an inverse text frequency statistical method, and further considers the command execution order dependency and word frequency issues.

[0012] In a more specific technical solution, step S2 includes:

[0013] S21, input the standardized command data into the autoencoder to generate the corresponding ID;

[0014] S22, encoding and mapping the numerical value in the standardized command according to a preset threshold;

[0015] S23, using a value less than or equal to a preset threshold as the beginning and end of each standardized command and the start and end flag of a preset session;

[0016] S24. Sort the pre-coded sequence according to the execution order of the preset session, and generate execution command stream blocks corresponding to each account in blocks.

[0017] In a more specific technical solution, step S3 includes:

[0018] S31, preparing historical operation command records of each account of the current host, using them as a reference comparison sequence, and converting the historical operation command records into an encoded historical sequence;

[0019] S32, performing an alignment operation on the reference comparison sequence and the encoded historical sequence, comparing the reference comparison sequence and the encoded historical sequence in a dictionary manner to obtain a matching pattern score, wherein the alignment operation methods include: padding completion and truncate truncation methods.

[0020] S33, calculating a score matrix according to the matching pattern score, and obtaining local abnormal intrusion degree data of the sequence string of the standardized command data;

[0021] S34, initialize a floating matrix to store the alignment operation scores, calculate the best score matrix based on it, add an index mark to the best score matrix to generate a contour characterization matrix as a model fitting parameter factor, and perform model training based on it.

[0022] The present invention adds an index mark to the best score matrix to generate the required contour characterization matrix, which is added and used in model training as a parameter factor of model fitting to achieve better detection effect.

[0023] In a more specific technical solution, step S33 further includes:

[0024] S331, using the Smith-Waterman local sequence alignment algorithm, finding the best local alignment modification parameters relative to a preset scoring system;

[0025] S332, calculating the sequence semi-global alignment data of the standardized command data to obtain a scoring matrix, wherein the preset scoring system includes: a substitution matrix and a gap scoring scheme.

[0026] In a more specific technical solution, step S34 further includes:

[0027] S341, performing diagonal processing on the floating matrix to obtain instruction alignment penalties and rewards of the historical operation command set his_data and the input data set val_data;

[0028] S342, processing the floating matrix from top to bottom to introduce a gap penalty in the historical operation command set his_data;

[0029] S343 , the floating matrix is ​​processed left and right to introduce a gap penalty in the input data set val_data.

[0030] In a more specific technical solution, step S4 includes:

[0031] S41. Using the Bag-of-words model, the unit commands in all command stream data are counted in descending order of word frequency, and each section of command stream data is vectorized and identified to obtain a vector value of the number of occurrences of a specific element, so as to obtain the difference command attention;

[0032] S42, using the term frequency-inverse document frequency model TF-IDF to perform weighted operations, based on which intrusion features are determined and acquired, and local assessment weight values ​​and global assessment weight values ​​of the standardized commands corresponding to the command stream data are processed to obtain a similar weighted matrix;

[0033] S43. Perform a sliding window operation using the N-gram model to obtain a related feature matrix.

[0034] The present invention uses the tape model to find the command words or command operations frequently used by the account, and adds different attention levels to different commands, which can further improve the actual scalability of the model. At the same time, the N-gram model is used to perform the sliding window operation to maintain the execution order and dependency.

[0035] In a more specific technical solution, in step S42, the following logic is used to obtain the local assessment weight value and the global assessment weight value:

[0036]

[0037] TF-IDF=TF*IDF.

[0038] In a more specific technical solution, in step S5, the following logic is used to express the hidden Markov model HMM:

[0039] λ=[π,A,B]

[0040] In the formula, λ is the model parameter, π is the initial state probability, A is the state transition probability, and B is the output observation probability.

[0041] In a more specific technical solution, step S5 includes:

[0042] S51, performing an evaluation operation using a forward-backward algorithm according to the model parameters to obtain the occurrence probability of a given observation sequence;

[0043] S52, performing a prediction operation using the Viterbi algorithm according to the model parameters, determining a hidden state column in a given observation sequence, so as to obtain disguised intruder operation data;

[0044] S53. Utilize the EM expectation maximization algorithm to solve, train and learn to obtain a suitable hidden Markov model, wherein the Baum-Welch algorithm is used for parameter estimation, a position score matrix is ​​added to the preset network structure to construct a profile hidden Markov model, and the profile hidden Markov model is trained and verified to obtain a suitable hidden Markov model.

[0045] The hidden Markov model used in the present invention adds a profile matrix, which can be understood as a multiple sequence alignment score matrix, which is equivalent to adding a weighted item during feature processing, which takes into account different degrees of host command execution and execution order.

[0046] The profile hidden Markov model adopted by the present invention takes into account the command position and context execution order dependency. The model obtained by training with a large amount of data has higher accuracy and faster running speed, better model scalability, and can detect unknown intrusion behaviors in an unsupervised manner.

[0047] In a more specific technical solution, the camouflage intrusion detection system based on the profile hidden horse model includes:

[0048] The data loading and extraction module is used to obtain the server's own audit log during the data loading and extraction process, obtain log data, and perform standard formatting on the Linux operation commands in the log data to complete the data loading and extraction operations and obtain standardized command data;

[0049] The transcoding mapping module is used to perform command stream block numerical transcoding mapping on the standardized command data by using the encoder self-encoding method to generate an execution command stream block. The transcoding mapping module is connected to the data loading and extraction module;

[0050] The comparison and alignment module is used to compare the global, semi-global and local contents of the pre-coded sequence through multiple sequence alignment to generate comparison scores and similarity scores, take the preset test segment in the standardized command data, align it with the user signature, and build the user command execution profile parameter matrix based on it, and obtain the intrusion indicator data in the preset test segment, the comparison and alignment module and the data loading and extraction module;

[0051] The feature engineering module is used to input the command stream data of the encoded annotated commands in the case of discrete data to perform word embedding operations, so as to identify the unit importance and unit weight of each input unit by using a preset tagging method, wherein the preset tagging methods include: Bag-of-words model, Term Frequency-Inverse Document Frequency model TF-IDF and N-gram model. The feature engineering module is connected with the transcoding mapping module and the data loading and extraction module;

[0052] The HMM model intrusion detection module is used to use the hidden Markov model HMM to perform model state conversion and observable event operations under specific states to obtain command flow feature data for setting estimation, training and setting model parameters of the hidden Markov model, and using the hidden Markov model HMM to perform evaluation operations, prediction operations and learning operations to complete observable event operations under specific states, and obtain the command flow feature performance probability matrix, based on which the camouflage intrusion detection results are obtained. The HMM model intrusion detection module is connected to the feature engineering module.

[0053] Compared with the prior art, the present invention has the following advantages:

[0054] The present invention uses Profile Hidden Markov Models (PHMM) to perform host intrusion detection. The present invention directly uses operation commands, such as Linux commands, for further analysis. The location information of command execution is considered to perform host camouflage intrusion detection, thereby improving the accuracy and precision of host intrusion detection. At the same time, the present invention is different in feature extraction. This application provides a word embedding method based on natural language processing, a word sliding window, a word frequency and an inverse text frequency statistical method, and further considers the command execution order dependency and word frequency issues.

[0055] The present invention adds an index mark to the best score matrix to generate the required contour characterization matrix, which is added and used in model training as a parameter factor of model fitting to achieve better detection effect.

[0056] The present invention uses the tape model to find the command words or command operations frequently used by the account, and adds different attention levels to different commands, which can further improve the actual scalability of the model. At the same time, the N-gram model is used to perform the sliding window operation to maintain the execution order and dependency.

[0057] The hidden Markov model used in the present invention adds a profile matrix, which can be understood as a multiple sequence alignment score matrix, which is equivalent to adding a weighted item during feature processing, which takes into account different degrees of host command execution and execution order.

[0058] The profile hidden Markov model adopted by the present invention takes into account the command position and context execution order dependency. The model obtained by training with a large amount of data has higher accuracy and faster running speed, better model scalability, and can detect unknown intrusion behaviors in an unsupervised manner.

[0059] The present invention solves the technical problems existing in the prior art of single detection method, lack of flexibility, and reliance on a large amount of data annotation resulting in low robustness, scalability of the detection system and low host intrusion detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 Schematic diagram of the basic steps of the camouflage intrusion detection method based on the contour hidden Markov model according to Embodiment 1 of the present invention;

[0061] Figure 2 This is a schematic diagram of specific steps for constructing a user command execution profile parameter matrix according to Embodiment 1 of the present invention;

[0062] Figure 3 This is a specific schematic diagram of calculating the optimal score matrix in Example 1 of the present invention;

[0063] Figure 4This is a schematic diagram of specific steps for position parameter sequence comparison score processing according to Example 1 of the present invention;

[0064] Figure 5 This is a schematic diagram of word embedding feature engineering data flow processing according to Embodiment 1 of the present invention;

[0065] Figure 6 This is a schematic diagram of specific steps for parameter estimation using the Baum-Welch algorithm in Example 1 of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in combination with the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Example 1

[0068] like Figure 1 As shown, the camouflage intrusion detection method based on the profile hidden Markov model provided by the present invention includes the following basic steps:

[0069] S1. During the data loading and extraction process, standard formatting is performed on the Linux operation commands in the log data;

[0070] In this embodiment, the Linux host operation log is used, and the server's own audit log can be used to provide account operation command data. Other capabilities of using traffic monitoring equipment can also obtain more friendly and easy-to-parse host operation logs;

[0071] In this embodiment, the Linux command executed by the user can directly reflect the content of the user's operation behavior. The operation command includes but is not limited to: command words, command parameters and operation objects; in the data extraction operation process of this embodiment, the aforementioned operation command is pre-processed by standard formatting, and the unified operation command format is: command word command parameter object field. In this embodiment, since the object field cannot be limited and changes too much, a fixed identifier is used. Instead, i is the number of file paths, IP domain names, etc. that appear after the command parameter. For example, the operation command: tail -f log.log is output as tail -f after format standardization. <1> ;cp-r file1 file2, after the format is standardized, the output is cp-r <2> ;

[0072] S2, the encoder self-encodes the command stream block to convert the numerical value into a map and generates a command stream block;

[0073] In this embodiment, all the command contents after de-reformatting standards appearing within the extraction time range are input into the autoencoder to generate the corresponding id; in this embodiment, a numerical value greater than 110 is used for encoding mapping, and each command word, command parameter and object field corresponds to a unique id, wherein the numerical value below 110 is used as a flag for the beginning and end of each command and the beginning and end of each session;

[0074] In this embodiment, the encoded sequence is divided into blocks according to the session execution order to generate execution command flow blocks corresponding to each account; in this embodiment, each small command flow block has a start and end execution flag, and each complete session command flow is a set of encoding sequences with prefixes and suffixes;

[0075] S3. Use multiple sequence alignment to construct a user command execution profile parameter matrix;

[0076] In this embodiment, the sequence alignment method is used to align each sequence in global, semi-global or local content; the aforementioned sequence alignment operation generates a score, which indicates the degree of sequence alignment and the degree of similarity between the sequences; by aligning a short segment of commands with the user signature, the alignment score can be used, for example, as an indicator of whether there is an intrusion in the segment being tested in this embodiment;

[0077] like Figure 2 As shown, in this embodiment, step S3 of constructing a user command execution profile parameter matrix further includes the following specific steps:

[0078] S31, historical reference comparison sequence preparation;

[0079] In this embodiment, a record of the historical operation commands of each account of the host is prepared as a reference comparison sequence, and the historical data also performs the aforementioned steps S31 and S32 in sequence to be converted into an encoded historical sequence; in this embodiment, in order to facilitate the description of the following embodiment, the historical operation command set is named his_data, and the input data set command is val_data;

[0080] S32, align two sequences;

[0081] In this embodiment, the data of two sequences are aligned. In this embodiment, for example, padding and truncation methods can be used to obtain sequences of the same length dimension, and the strings are compared in a dictionary manner to find matching patterns and score them;

[0082] S33, calculating a score matrix;

[0083] In this embodiment, a scoring matrix is ​​calculated, and such a scoring matrix can represent the degree of local invasion anomaly of the sequence string input in step S1; in this embodiment, a modification of the Smith-Waterman local sequence alignment algorithm (a dynamic programming algorithm that can ensure that the best local alignment relative to the scoring system used is found) is used to calculate the semi-global alignment of the command sequence. In this embodiment, the Smith-Waterman local sequence alignment algorithm is a dynamic programming algorithm for finding the best local alignment relative to the scoring system used. In this embodiment, the scoring system includes but is not limited to: a substitution matrix, a gap scoring scheme;

[0084] S34, obtaining the best score matrix;

[0085] In this embodiment, a floating matrix is ​​initialized to store scores throughout the alignment process; in this embodiment, each position (i, j) in the floating matrix corresponds to the optimal score of the alignment ending at his_data and val_data; the optimal score matrix is ​​calculated by starting from the upper left corner of the floating matrix, such as point (0,0); recursively taking one step to produce the maximum value from the following three options;

[0086] like Figure 3 As shown, in this embodiment, the operation of calculating the best score matrix also includes the following specific steps:

[0087] S341, diagonal processing;

[0088] In this example, the score ending up at position (i-1, j-1) is added with matchScore(his_data i-1, val_data j-1), which is the penalty or reward for aligning the command at his_data at i-1 with the instruction at val_data at j-1;

[0089] S342, top-down processing;

[0090] In this embodiment, the score ending at position (i, j-1) is added with a penalty value, which is the penalty for introducing a gap in his_data;

[0091] S343, left and right processing;

[0092] In this embodiment, the score ending at position (i-1, j) is added with a penalty value, which is the penalty for introducing a gap in val_data;

[0093] In this embodiment, the index mark is added to the best score matrix to generate the required contour characterization matrix, which is added and used in model training as a parameter factor of model fitting to achieve better detection effect;

[0094] S4, feature engineering module, using word embedding to vectorize or represent data;

[0095] In this embodiment, the feature module performs word embedding or word segmentation. For discrete data, in this embodiment, the importance and weight of each input unit can be identified by using methods such as Bag-of-words model, TF-IDF (Term Frequency-Inverse Document Frequency) and N-gram.

[0096] like Figure 4 As shown, in this embodiment, the command stream data after input encoding is segmented from different dimensions and focuses using the bag-of-words model, TF-IDF and N-gram methods.

[0097] In this embodiment, the bag-of-words model temporarily does not consider the order of each command operation in the command stream, but counts all unit commands in all input command streams in descending order of word frequency; each segment of the command stream is vectorized and identified, and each vector value corresponds to the number of times the element appears in the command; in this embodiment, for example, a segment of the operation command stream is represented as [1,2,1,1,1,0,0,0,1,1], where 2 means that the command cat appears twice in the segment of the command stream. The purpose of the above method is to find the command words or command operations frequently used by an account, and adding different attention levels to different commands can further improve the actual scalability of the model;

[0098] In this embodiment, TF-IDF is a commonly used weighting technology for information retrieval and data mining. TF means term frequency, and IDF means inverse document frequency. In this embodiment, the importance of a single command increases in direct proportion to the number of times the command appears in a single user's operation command stream, and decreases in inverse proportion to the frequency of the single command appearing in all user operation commands of the local machine. That is, it can be considered that the more times a command appears in the current statistical time range of the user, and the fewer times it appears in all local operation command data, the more it can represent a different operation sequence, and the more obvious the feature of being judged as an intrusion is.

[0099] In this embodiment, the calculation formulas of term frequency TF and inverse text frequency index IDF are as follows:

[0100]

[0101] TF-IDF=TF*IDF

[0102] That is, TF-IDF is equal to TF value multiplied by IDF value. TF-IDF value calculates the local and global assessment weights of the command in this session and in all commands, which performs well in practical application fitting, thus obtaining a similar weighted matrix;

[0103] In this embodiment, the N-gram model performs a sliding window operation in order to maintain the execution order and dependency, where n represents the size of the sliding window. For example, the 2-gram model treats two words as a group, then moves back the length of a word to form another group of words, and generates a dictionary with the dictionary value being the word after each sliding two words. In the aforementioned step S1, each command is formatted into three parts. The continuity of the operation commands can be considered, and the direct context dependency of different commands can also be paid attention to. For example, for two consecutive commands: cat / etc / sudoers, sudo-i account, they are represented in a 2-gram manner as [[cat, <1> ],[ <1> sudo],[sudo-i]] and [[sudo],[-i],[ <1> ]], of course, these commands are expressed after numerical coding. The above methods can obtain the relevant feature matrix;

[0104] S5, HMM model setting estimation, output model parameters;

[0105] The HMM model used in this embodiment is a double random process, including: the state transition of the model and the randomness of observable events in a specific state;

[0106] like Figure 5 As shown, in this embodiment, X represents the hidden state sequence, which represents two types in this embodiment, [this user, intruder user], and S represents the observation state, which represents the command stream data after characterization processing in this embodiment; the value of the state sequence at time t is only related to the value of the state sequence at time t-1, and the value of the observation sequence at time t is only related to the value of the observation sequence at time t. According to the known parameters, a hidden Markov model is generally recorded as:

[0107] λ=[π,A,B]

[0108] Among them, the initial state probability π is the probability of each state of the model appearing at the initial moment, that is, the possible operation ratio of the normal user account and the possible operation ratio of the intruder user account;

[0109] State transition probability A: The probability of the model switching between various states, that is, the probability of switching from normal account use to intrusion account command operation, should be a 2x2 probability matrix;

[0110] Output observation probability B: The model obtains the probability matrix of each observation value according to the current state, that is, the characteristic performance of each command flow;

[0111] In this embodiment, the HMM model is used for evaluation, prediction and learning operations; in the evaluation operation of this embodiment, the model parameters λ = [π, A, B] and the observation sequence Q are known, and the probability of the observation sequence occurring is calculated. After training a model or obtaining a model parameter for existing data, the probability of a given observation sequence occurring can be calculated. The forward-backward algorithm is generally used to solve this problem;

[0112] In the prediction operation of this embodiment, that is, the decoding operation, the model parameters λ = [π, A, B] and the observation sequence Q are known, and the most likely state sequence corresponding to the observation sequence is calculated; after a model is trained or a model parameter is obtained for existing data, given an observation sequence, we are more concerned about the hidden state column, whether it is a normal user operation or an operation disguised as an intruder. The more commonly used method to solve this prediction problem is the Viterbi algorithm, which is a dynamic programming algorithm, and can also be understood as a pruning method of the search space, which can ensure that the global optimal path is found;

[0113] In the learning operation of this embodiment, the parameter estimation problem, the model parameter λ = [π, A, B] is unknown, given the observation sequence, the model parameter is inferred; when there is no ready-made identification data, it is necessary to train and learn a convergent and satisfactory model from a large number of execution command streams, and this model can be the HMM model required for evaluation and prediction problems. Generally, the EM expectation maximization algorithm is used for solving;

[0114] In this embodiment, the Baum-Welch algorithm is used for parameter estimation to construct a profile hidden Markov model, which is a characterization of the position score matrix added to the structure; in this embodiment, according to the data preprocessing and feature representation in the above steps, the required observation matrix is ​​constructed, input into the constructed model, and training verification is started until it reaches the set convergence threshold, and the model is output;

[0115] like Figure 6 As shown, in this embodiment, the process of using the Baum-Welch algorithm to perform parameter estimation in this embodiment also includes the following specific steps:

[0116] S51, initialization;

[0117] In this embodiment, initial assignment is performed by initialization, and model parameters are randomly selected:

[0118] λ=[π,A,B], when n=0.

[0119] S52, traverse parameter update, iteratively update parameters at n=1, 2, 3... moments;

[0120] S53, until the model parameters converge, stop iterating;

[0121] In this embodiment, for example, the hmmlearn library can be used to directly estimate parameters. In this embodiment, it can be seen that the evaluated parameters have a very low camouflage intrusion rate. Further model testing is performed using log data with historical intrusion traces. It is found that the HMM model with the position parameter added has better results.

[0122] In summary, the present invention uses Profile Hidden Markov Models (PHMM) to perform host intrusion detection. The present invention directly uses operation commands, such as Linux commands, for further analysis. The location information where the command execution occurs is considered to perform host camouflage intrusion detection, thereby improving the accuracy and precision of host intrusion detection. At the same time, the present invention is different in feature extraction. This application provides a word embedding method based on natural language processing, a word sliding window, a word frequency, and an inverse text frequency statistical method, and further considers the command execution order dependency and word frequency issues.

[0123] The present invention adds an index mark to the best score matrix to generate the required contour characterization matrix, which is added and used in model training as a parameter factor of model fitting to achieve better detection effect.

[0124] The present invention uses the tape model to find the command words or command operations frequently used by the account, and adds different attention levels to different commands, which can further improve the actual scalability of the model. At the same time, the N-gram model is used to perform the sliding window operation to maintain the execution order and dependency.

[0125] The hidden Markov model used in the present invention adds a profile matrix, which can be understood as a multiple sequence alignment score matrix, which is equivalent to adding a weighted item during feature processing, which takes into account different degrees of host command execution and execution order.

[0126] The profile hidden Markov model adopted by the present invention takes into account the command position and context execution order dependency. The model obtained by training with a large amount of data has higher accuracy and faster running speed, better model scalability, and can detect unknown intrusion behaviors in an unsupervised manner.

[0127] The present invention solves the technical problems existing in the prior art of single detection method, lack of flexibility, and reliance on a large amount of data annotation resulting in low robustness, scalability of the detection system and low host intrusion detection accuracy.

[0128] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A camouflage intrusion detection method based on a contour hidden Markov model, characterized in that: The method comprises: S1. During the data loading and extraction process, the server's own audit log is obtained to obtain log data, and the Linux operation commands in the log data are formatted in a standard manner to complete the data loading and extraction operations and obtain standardized command data; S2, using an encoder self-encoding method to perform command stream block numerical transcoding mapping on the standardized command data to generate an execution command stream block; S3. By multiple sequence alignment, the global, semi-global and local contents of the pre-coded sequence are aligned to generate a comparison degree score and a similarity degree score, a preset test segment in the standardized command data is taken, and aligned with the user signature to construct a user command execution profile parameter matrix, and obtain the intrusion indicator data in the preset test segment; S4. Using a feature engineering module, in a discrete data situation, input the encoded command stream data of the annotated command to perform the word embedding operation, so as to identify the unit importance and unit weight of each input unit using a preset tagging method, wherein the preset tagging method includes: a Bag-of-words model, a Term Frequency-Inverse Document Frequency model TF-IDF, and an N-gram model; S5. Use the hidden Markov model HMM to perform model state transition and observable event operations under specific states to obtain command stream feature data for setting estimation, training and setting model parameters of the hidden Markov model, and use the hidden Markov model HMM to perform evaluation operations, prediction operations and learning operations to complete the observable event operations under the specific state, obtain the command stream feature performance probability matrix, and obtain the camouflage intrusion detection results accordingly.

2. The camouflage intrusion detection method based on contour hidden Markov model according to claim 1 is characterized in that: The step S2 comprises: S21, inputting the standardized command data into the autoencoder to generate a corresponding ID; S22, encoding and mapping the numerical value in the standardized command according to a preset threshold; S23, using the value less than or equal to the preset threshold as the beginning and end of each standardized command and the start and end flag of the preset session; S24. Sort the pre-coded sequence according to the execution order of the preset session, and generate the execution command stream block corresponding to each account in blocks.

3. The camouflage intrusion detection method based on contour hidden Markov model according to claim 1 is characterized in that: The step S3 comprises: S31, preparing historical operation command records of each account of the current host, using them as a reference comparison sequence, and converting the historical operation command records into an encoded historical sequence; S32, performing an alignment operation on the reference comparison sequence and the encoded historical sequence, comparing the reference comparison sequence and the encoded historical sequence in a dictionary manner to obtain a matching pattern score, wherein the alignment operation includes padding completion and truncation. S33, calculating a score matrix according to the matching pattern score, and obtaining local abnormal intrusion degree data of the sequence string of the standardized command data; S34, initialize a floating matrix to store the alignment operation scores, calculate the best score matrix based on it, add an index mark to the best score matrix to generate a contour characterization matrix as a model fitting parameter factor, and perform model training based on it.

4. The camouflage intrusion detection method based on contour hidden Markov model according to claim 3 is characterized in that: The step S33 further comprises: S331, using the Smith-Waterman local sequence alignment algorithm, finding the best local alignment modification parameters relative to a preset scoring system; S332, calculating the sequence semi-global alignment data of the standardized command data to obtain the score matrix, wherein the preset scoring system includes: a substitution matrix and a gap scoring scheme.

5. The camouflage intrusion detection method based on contour hidden Markov model according to claim 3 is characterized in that: The step S34 further comprises: S341, performing diagonal processing on the floating matrix to obtain instruction alignment penalties and rewards of a historical operation command set his_data and an input data set val_data; S342, processing the floating matrix from top to bottom to introduce a gap penalty into the historical operation command set his_data; S343 . Process the floating matrix in left and right directions to introduce a gap penalty in the input data set val_data.

6. The camouflage intrusion detection method based on contour hidden Markov model according to claim 1 is characterized in that: The step S4 comprises: S41, using the Bag-of-words model, performing word frequency descending statistics on all unit commands in the command stream data, and vectorizing and marking each section of the command stream data to obtain a vector value of the number of occurrences of a specific element, so as to obtain a difference in command attention; S42, using the term frequency-inverse document frequency model TF-IDF to perform a weighted operation, based on which the intrusion feature is determined and obtained, and the local assessment weight value and the global assessment weight value of the standardized command corresponding to the command stream data are processed to obtain a similar weighted matrix; S43. Perform a sliding window operation using the N-gram model to obtain a relevant feature matrix.

7. The camouflage intrusion detection method based on contour hidden Markov model according to claim 5 is characterized in that: In step S42, the local assessment weight value and the global assessment weight value are obtained using the following logic: TF-IDF=TF*IDF.

8. The camouflage intrusion detection method based on contour hidden Markov model according to claim 1 is characterized in that: In step S5, the hidden Markov model HMM is expressed using the following logic: λ=[π,A,B] Where π is the initial state probability, A is the state transition probability, and B is the output observation probability.

9. The camouflage intrusion detection method based on contour hidden Markov model according to claim 1 is characterized in that: The step S5 comprises: S51, performing the evaluation operation using a forward-backward algorithm according to the model parameters to obtain the occurrence probability of a given observation sequence; S52, performing the prediction operation using the Viterbi algorithm according to the model parameters, determining the hidden state column in the given observation sequence, so as to obtain the disguised intruder operation data; S53, using the EM expectation maximization algorithm to solve, train and learn to obtain a suitable hidden Markov model, wherein the Baum-Welch algorithm is used for parameter estimation, and a position score matrix is ​​added to the preset network structure to construct a profile hidden Markov model, and the profile hidden Markov model is trained and verified to obtain the suitable hidden Markov model.

10. A camouflage intrusion detection system based on a contour hidden Markov model, characterized in that: The system comprises: The data loading and extraction module is used to obtain the server's own audit log during the data loading and extraction process, obtain log data based on it, and perform standard formatting on the Linux operation commands in the log data to complete the data loading and extraction operations and obtain standardized command data; A transcoding mapping module, used for performing command stream block numerical transcoding mapping on the standardized command data by using an encoder self-encoding method to generate an execution command stream block, wherein the transcoding mapping module is connected to the data loading and extraction module; A comparison and alignment module, used to compare the global, semi-global and local contents of the pre-coded sequence through multiple sequence alignment to generate a comparison degree score and a similarity degree score, take the preset test segment in the standardized command data, align it with the user signature, and construct a user command execution profile parameter matrix based on it, and obtain the intrusion indicator data in the preset test segment. The comparison and alignment module is connected to the data loading and extraction module; A feature engineering module, for inputting the encoded command stream data of the annotated command in the case of discrete data to perform the word embedding operation, so as to identify the unit importance and unit weight of each input unit by using a preset tagging method, wherein the preset tagging method includes: Bag-of-words model, term frequency-inverse document frequency model TF-IDF and N-gram model, and the feature engineering module is connected with the transcoding mapping module and the data loading and extraction module; The HMM model intrusion detection module is used to use the hidden Markov model HMM to perform model state conversion and observable event operations under specific states to obtain command flow feature data for setting estimation, training and setting model parameters of the hidden Markov model, and use the hidden Markov model HMM to perform evaluation operations, prediction operations and learning operations to complete the observable event operations under the specific state, obtain the command flow feature performance probability matrix, and obtain the camouflage intrusion detection results based on it. The HMM model intrusion detection module is connected to the feature engineering module.

Images