Camouflage intrusion detection method and system based on profile hidden horse model

Abstract

This invention provides a method and system for detecting spoofed intrusions based on a Hidden Markov Model (HMM). The method includes: during the data loading and extraction process, standard formatting of Linux operation commands in log data; generating command stream blocks by numerical transcoding using an encoder; constructing a user command execution profile parameter matrix using multiple sequence alignment; vectorizing or representing data using word embeddings via a feature engineering module; setting and estimating the HMM model, outputting model parameters, and using the HMM model for evaluation, prediction, and learning to obtain spoofed intrusion detection results. This invention solves the technical problems of single detection methods, lack of flexibility, and low robustness, scalability, and host intrusion detection accuracy due to reliance on large amounts of data annotation.

Description

Technical Field

[0001] This invention relates to the field of network security, and specifically to a method and system for detecting camouflaged intrusions based on a silhouette cryptic model. Background Technology

[0002] With the increasing complexity and diversity of cyberattacks, attackers targeting enterprise host servers can launch numerous varied and difficult-to-detect intrusion attacks at any time. The intelligence and stealth of attack methods pose a significant challenge to currently used Host Intrusion Detection Systems (HIDS). Existing intrusion detection systems can be divided into signature-based detection and anomaly detection-based detection. Signature-based intrusion detection generally matches known rule features. For example, the existing invention patent application document CN111181971A, entitled "A System for Automatically Detecting Industrial Network Attacks," includes a packet capture module, a network traffic profile generation module, a signature-based intrusion detection module, an anomaly detection module, and an alarm generation and processing module for detecting ICS network attacks. From the specific implementation of this prior art, it can be seen that the signature-based intrusion detection module in this existing solution matches the content of the packets captured by the packet capture module with known attack signatures. If any match is found, the corresponding packet is considered an intrusion. The aforementioned intrusion detection methods are limited in their detection approaches and cannot detect unknown abnormal attacks or variants of known attacks, lacking flexibility. Intrusion detection based on abnormal behavior utilizes methods such as behavioral baseline deviation, machine learning, and deep learning models. Currently, deep neural networks appear to be a promising direction for intrusion detection. For example, the existing invention patent application document CN117278338A, entitled "A Network Intrusion Detection Method and System Based on Deep Learning Optimization," includes the following steps: preprocessing the raw dataset collected from the honeypot system to form an experimental dataset; dividing the experimental dataset into training and test sets to construct a baseline ELM model; combining the AE autoencoder with the baseline ELM model to construct the AE-ELM algorithm model; modifying the inertial weights of the standard particle swarm optimization algorithm to form a particle swarm optimization algorithm; using the particle swarm optimization algorithm to optimize the learning parameters of the AE-ELM algorithm model to construct the PSO-AE-ELM algorithm model as an experimental model; comparing the experimental model with mainstream algorithm models to verify the reliability of the experimental model in network intrusion detection; and using the experimental model for network intrusion detection. However, in the aforementioned existing technologies, deep neural networks still rely on a sufficient number of annotations. On the one hand, the annotation of a large number of data samples requires a lot of manpower and resources. For training data with a small sample size, the false alarm rate in detection is high and the interpretability is not strong. On the other hand, the robustness of the model needs to be verified. Often, the model detection lacks scalability. For new intrusion behaviors, the original model cannot be identified, and a new model needs to be trained.

[0003] In summary, existing technologies suffer from problems such as limited detection methods, lack of flexibility, reliance on large amounts of data annotation leading to low robustness, scalability, and host intrusion detection accuracy. Summary of the Invention

[0004] The technical problem to be solved by this invention is: how to solve the problems of the existing technology, such as the single detection method, lack of flexibility, and low robustness, scalability and host intrusion detection accuracy due to the reliance on a large amount of data annotation.

[0005] This invention solves the above-mentioned technical problems by employing the following technical solution: a camouflage intrusion detection method based on a contour cryptic model, comprising:

[0006] S1. During the data loading and extraction process, the server's built-in audit logs are obtained to acquire log data. The Linux operation commands in the log data are then formatted in a standard way to complete the data loading and extraction operation and obtain standardized command data.

[0007] S2. Use the encoder's self-encoding method to perform numerical transcoding mapping on the standardized command data to generate the execution command stream block;

[0008] S3. By comparing the global, semi-global and local contents of the precoded sequence through multiple sequence alignment, a comparison score and a similarity score are generated. A pre-set test segment in the standardized command data is taken and aligned with the user signature to construct the user command execution profile parameter matrix and obtain intrusion indicator data within the pre-set test segment.

[0009] S4. Using the feature engineering module, in the case of discrete data, input the command stream data of the encoded annotated commands to perform word embedding operations, and use pre-set labeling methods to identify the importance and weight of each input unit. The pre-set labeling methods include: Bag-of-words model, TF-IDF (Term Frequency-Inverse Document Frequency) model and N-gram model.

[0010] S5. Utilize Hidden Markov Model (HMM) to perform model state transitions and observable event operations under specific states to obtain command flow feature data for setting estimation. Train and set the model parameters of the Hidden Markov Model. Use the Hidden Markov Model (HMM) to perform evaluation, prediction, and learning operations to complete observable event operations under specific states, obtain the command flow feature performance probability matrix, and obtain the spoofing intrusion detection results.

[0011] This invention utilizes Profile Hidden Markov Models (PHMM) for host intrusion detection. It directly uses operational commands, such as Linux commands, for further analysis. It considers the location information of command execution to perform host spoofing intrusion detection, improving the accuracy and precision of host intrusion detection. Furthermore, this invention differs in its feature extraction methods. This application presents methods borrowing from natural language processing techniques, such as word embedding, word sliding windows, word frequency, and inverse text frequency statistics, further considering command execution order dependencies and word frequency issues.

[0012] In a more specific technical solution, step S2 includes:

[0013] S21. Input the standardized command data into the autoencoder to generate the corresponding ID;

[0014] S22. Encode and map the values ​​in the standardized commands according to the preset threshold.

[0015] S23. Use values ​​less than or equal to a preset threshold as the beginning and end flags of each standardized command and the start and end flags of a preset session.

[0016] S24. Sort the pre-encoded sequence according to the execution of the preset session, and generate the execution command stream block corresponding to each account in blocks.

[0017] In a more specific technical solution, step S3 includes:

[0018] S31. Prepare the historical operation command records of each account on the current host, and use them as a reference comparison sequence. Convert the historical operation command records into an encoded historical sequence.

[0019] S32. Perform alignment operations on the reference comparison sequence and the encoded history sequence. Compare the reference comparison sequence and the encoded history sequence in a dictionary manner to obtain a matching pattern score. The alignment operations include padding and truncate.

[0020] S33. Calculate the score matrix based on the matching pattern score to obtain the local anomaly intrusion degree data of the sequence string of standardized command data.

[0021] S34. Initialize the floating matrix to store the alignment operation scores, calculate the best score matrix based on it, add an index label to the best score matrix to generate the contour characterization matrix, which serves as the model fitting parameter factor for model training.

[0022] This invention adds an index identifier to the optimal score matrix to generate the required contour characterization matrix, which is then added as a parameter factor in model training to achieve better detection results.

[0023] In a more specific technical solution, step S33 also includes:

[0024] S331. Use the Smith-Waterman local sequence alignment algorithm to find the optimal local alignment modification parameters relative to the preset scoring system;

[0025] S332. Calculate the semi-global alignment data of the standardized command data to obtain the score matrix, wherein the preset scoring system includes: replacement matrix and gap scoring scheme.

[0026] In a more specific technical solution, step S34 also includes:

[0027] S341. Perform diagonal processing on the floating matrix to obtain the instruction alignment penalty and reward of the historical operation command set his_data and the input dataset val_data;

[0028] S342. Process the floating matrix from top to bottom to introduce gap penalty in the historical operation command set his_data;

[0029] S343. Perform left and right processing on the floating matrix to introduce a penalty for gaps in the input dataset val_data.

[0030] In a more specific technical solution, step S4 includes:

[0031] S41. Using the Bag-of-words model, perform word frequency statistics in descending order for unit commands in all command stream data, and vectorize each segment of command stream data to obtain the vector value of the number of occurrences of specific elements, so as to obtain the attention of differentiated commands.

[0032] S42. Use the term frequency-inverse document frequency model TF-IDF to perform weighting operations, thereby determining and obtaining intrusion features. Process the data to obtain the local assessment weight value and global assessment weight value of the standardized commands corresponding to the command stream data, and then process them to obtain a similar weighting matrix.

[0033] S43. Perform a sliding window operation using an N-gram model to obtain the relevant feature matrix.

[0034] This invention utilizes a magnetic tape model to identify frequently used command words or operations by an account, assigning different levels of attention to different commands to further improve the model's practical scalability. Simultaneously, it employs an N-gram model to execute sliding window operations, maintaining the execution order and dependencies.

[0035] In a more specific technical solution, step S42 uses the following logic to calculate the local assessment weight value and the global assessment weight value:

[0036]

[0037] TF-IDF = TF*IDF.

[0038] In a more specific technical solution, step S5 uses the following logic to express the Hidden Markov Model (HMM):

[0039] λ = [π, A, B]

[0040] In the formula, λ is the model parameter, π is the initial state probability, A is the state transition probability, and B is the output observation probability.

[0041] In a more specific technical solution, step S5 includes:

[0042] S51. Based on the model parameters, use the forward-backward algorithm to perform an evaluation operation to obtain the probability of occurrence of a given observation sequence;

[0043] S52. Based on the model parameters, use the Viterbi algorithm to perform prediction operations, determine the hidden state column in the given observation sequence, and process the data of the disguised intruder operation.

[0044] S53. Solve using the EM expectation-maximization algorithm, train and learn to obtain a suitable Hidden Markov Model, wherein the Baum-Welch algorithm is used for parameter estimation, and a position score matrix is ​​added to the pre-set network structure to construct a contour Hidden Markov Model, train and validate the contour Hidden Markov Model, and obtain a suitable Hidden Markov Model.

[0045] The Hidden Markov Model used in this invention incorporates a contour matrix, which can be understood as a multi-sequence alignment score matrix. This is equivalent to adding a weighted term during feature processing, taking into account different levels of host command execution and execution order.

[0046] The contour hidden map model used in this invention considers the dependency of command position and context execution order. The model trained with a large amount of data has higher accuracy and faster running speed, and better model extensibility. It can detect unknown intrusion behavior in an unsupervised manner.

[0047] In more specific technical solutions, camouflage intrusion detection systems based on contour hidden phasing models include:

[0048] The data loading and extraction module is used to obtain the server's built-in audit logs during the data loading and extraction process, thereby obtaining log data. The Linux operation commands in the log data are then formatted in a standard way to complete the data loading and extraction operation and obtain standardized command data.

[0049] The transcoding and mapping module is used to perform numerical transcoding and mapping of standardized command data using the encoder's self-encoding method to generate execution command stream blocks. The transcoding and mapping module is connected to the data loading and extraction module.

[0050] The alignment module is used to compare the global, semi-global, and local contents of the precoded sequence through multi-sequence alignment to generate comparison score and similarity score. It takes the preset test segment from the standardized command data, aligns it with the user signature, constructs the user command execution profile parameter matrix, and obtains intrusion indicator data within the preset test segment. The alignment module and the data loading and extraction module are also included.

[0051] The feature engineering module is used to take the input encoded and annotated command stream data in the case of discrete data and perform word embedding operation. It uses preset labeling methods to identify the importance and weight of each input unit. The preset labeling methods include: Bag-of-words model, TF-IDF (Term Frequency-Inverse Document Frequency) model and N-gram model. The feature engineering module is connected to the transcoding and mapping module and the data loading and extraction module.

[0052] The HMM model intrusion detection module utilizes Hidden Markov Models (HMMs) to perform model state transitions and observable event operations under specific states to obtain command flow feature data for setting estimation, training and setting the model parameters of the HMM, and using the HMM to perform evaluation, prediction and learning operations to complete observable event operations under specific states, thereby obtaining the command flow feature performance probability matrix to obtain the disguised intrusion detection results. The HMM model intrusion detection module is connected to the feature engineering module.

[0053] The present invention has the following advantages over the prior art:

[0054] This invention utilizes Profile Hidden Markov Models (PHMM) for host intrusion detection. It directly uses operational commands, such as Linux commands, for further analysis. It considers the location information of command execution to perform host spoofing intrusion detection, improving the accuracy and precision of host intrusion detection. Furthermore, this invention differs in its feature extraction methods. This application presents methods borrowing from natural language processing techniques, such as word embedding, word sliding windows, word frequency, and inverse text frequency statistics, further considering command execution order dependencies and word frequency issues.

[0055] This invention adds an index identifier to the optimal score matrix to generate the required contour characterization matrix, which is then added as a parameter factor in model training to achieve better detection results.

[0056] This invention utilizes a magnetic tape model to identify frequently used command words or operations by an account, assigning different levels of attention to different commands to further improve the model's practical scalability. Simultaneously, it employs an N-gram model to execute sliding window operations, maintaining the execution order and dependencies.

[0057] The Hidden Markov Model used in this invention incorporates a contour matrix, which can be understood as a multi-sequence alignment score matrix. This is equivalent to adding a weighted term during feature processing, taking into account different levels of host command execution and execution order.

[0058] The contour hidden map model used in this invention considers the dependency of command position and context execution order. The model trained with a large amount of data has higher accuracy and faster running speed, and better model extensibility. It can detect unknown intrusion behavior in an unsupervised manner.

[0059] This invention solves the technical problems of existing technologies, such as the single detection method, lack of flexibility, and low robustness, scalability, and host intrusion detection accuracy due to reliance on a large amount of data annotation. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the basic steps of the camouflage intrusion detection method based on the contour cryptanalysis model in Embodiment 1 of the present invention;

[0061] Figure 2 This is a schematic diagram illustrating the specific steps of constructing the user command execution profile parameter matrix in Embodiment 1 of the present invention;

[0062] Figure 3 This is a schematic diagram illustrating the calculation of the optimal score matrix in Embodiment 1 of the present invention;

[0063] Figure 4This is a schematic diagram illustrating the specific steps of the position parameter sequence comparison and scoring process in Embodiment 1 of the present invention;

[0064] Figure 5 This is a schematic diagram of the word embedding feature engineering data flow processing in Embodiment 1 of the present invention;

[0065] Figure 6 This is a schematic diagram illustrating the specific steps of parameter estimation using the Baum-Welch algorithm in Embodiment 1 of the present invention. Detailed Implementation

[0066] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0067] Example 1

[0068] like Figure 1 As shown, the camouflage intrusion detection method based on a contour cryptic model provided by this invention includes the following basic steps:

[0069] S1. During the data loading and extraction process, the Linux operation commands in the log data are processed using standard formatting.

[0070] In this embodiment, Linux host operation logs are used. The server's built-in audit logs can provide account operation command data. Other traffic monitoring devices can also obtain more user-friendly and easily parsed host operation logs.

[0071] In this embodiment, the Linux commands executed by the user directly reflect the user's operation behavior. These commands include, but are not limited to, command words, command parameters, and operation objects. During the data extraction process in this embodiment, the aforementioned operation commands undergo standard formatting preprocessing, unifying the command format to: command word, command parameters, and object field. In this embodiment, since the object field is not easily defined and varies greatly, a fixed identifier is used. The substitution is where 'i' represents the number of file paths, IP addresses, domain names, etc., that appear after the command parameters. For example, the command `tail -f log.log` will output `tail -f` after format standardization. <1> ;cp -r file1 file2, after format standardization, the output is cp -r <2> ;

[0072] S2, the encoder's self-encoding method performs numerical transcoding mapping on the command stream block to generate the command stream block;

[0073] In this embodiment, all de-formatted command contents appearing within the extraction time range are input into the autoencoder to generate corresponding IDs. In this embodiment, values ​​greater than 110 are used for encoding mapping, and each command word, command parameter, and object field corresponds to a unique ID. Values ​​below 110 are reserved as the beginning and end flags of each command and the start and end flags of each session.

[0074] In this embodiment, the encoded sequence is divided into blocks according to the session execution order to generate execution command flow blocks corresponding to each account; in this embodiment, each small command flow block has a start execution and end execution flag, and each complete session command flow is a set of encoded sequences with prefixes and suffixes;

[0075] S3. Construct a user command execution profile parameter matrix using multiple sequence alignment.

[0076] In this embodiment, the sequence alignment method is used to compare each sequence in global, semi-global, or local content; the aforementioned sequence alignment operation generates a score, which represents the degree of sequence alignment and the degree of similarity between sequences; by aligning a small segment of commands with the user signature, for example, the alignment score can be used as an indicator of whether there is an intrusion within the segment being tested in this embodiment;

[0077] like Figure 2 As shown, in this embodiment, step S3, which constructs the user command execution profile parameter matrix, further includes the following specific steps:

[0078] S31. Preparation of historical reference comparison sequences;

[0079] In this embodiment, a record of historical operation commands for each account on the host is prepared as a reference comparison sequence. The historical data is also converted into an encoded historical sequence by sequentially executing the aforementioned steps S31 and S32. In this embodiment, for the convenience of the following description, the historical operation command set is named his_data and the input dataset command is val_data.

[0080] S32. Align the two sequences;

[0081] In this embodiment, the data of two sequences are aligned. For example, padding and truncate can be used to obtain sequences of the same length and dimension. The strings are compared in a dictionary manner to find matching patterns and score them.

[0082] S33. Calculate the score matrix;

[0083] In this embodiment, a score matrix is ​​calculated, which represents the degree of local intrusion anomaly of the sequence string input in step S1. In this embodiment, a semi-global alignment of the command sequence is calculated using the Smith-Waterman Local Sequence Alignment Algorithm (a dynamic programming algorithm that guarantees finding the optimal local alignment relative to the used scoring system). In this embodiment, the Smith-Waterman Local Sequence Alignment Algorithm is a dynamic programming algorithm used to find the optimal local alignment relative to the used scoring system, which includes, but is not limited to, substitution matrices and gap scoring schemes.

[0084] S34. Obtain the optimal score matrix;

[0085] In this embodiment, a floating matrix is ​​initialized to store scores throughout the alignment process; in this embodiment, each position (i, j) in the floating matrix corresponds to the optimal score of the alignment ending at his_data and val_data; the optimal score matrix is ​​calculated by recursively performing a step from the top left corner of the floating matrix, for example, point (0,0), to generate the maximum value from the following three options;

[0086] like Figure 3 As shown, in this embodiment, the operation of calculating the optimal score matrix further includes the following specific steps:

[0087] S341, Diagonal treatment;

[0088] In this embodiment, the score ending at position (i-1, j-1) is added to matchScore(his_data i-1, val_data j-1), which is a penalty or reward for aligning the command of his_data at i-1 with the instruction of val_data at j-1.

[0089] S342, Top-down processing;

[0090] In this embodiment, a penalty value is added to the score ending at position (i, j-1), which is the penalty for introducing gaps in his_data;

[0091] S343, Left and right processing;

[0092] In this embodiment, a penalty value is added to the score ending at position (i-1, j), which is the penalty for introducing a gap in val_data;

[0093] In this embodiment, adding an index identifier to the best score matrix generates the required contour characterization matrix, which is added as a parameter factor in model fitting during model training to achieve better detection results.

[0094] S4, Feature Engineering Module, uses word embedding to vectorize or represent data;

[0095] In this embodiment, the feature module performs word embedding or word segmentation. For discrete data, this embodiment can use, for example, bag-of-words model, TF-IDF (term frequency – inverse document frequency), and N-gram to identify the importance and weight of each input unit.

[0096] like Figure 4 As shown, in this embodiment, the input encoded command stream data is segmented using the bag-of-words model, TF-IDF, and N-gram methods from different dimensions and focuses.

[0097] In this embodiment, the bag-of-words model temporarily disregards the order of each command operation in the command stream. Instead, it performs word frequency ranking on all unit commands in all input command streams. Each segment of the command stream is vectorized, with each vector value corresponding to the number of times the element appears in the command. In this embodiment, for example, a segment of the command stream is represented as [1,2,1,1,1,0,0,0,1,1], where 2 indicates that the command "cat" appears twice in this segment. The purpose of the aforementioned method is to find frequently used command words or command operations for an account. By assigning different levels of attention to different commands, the actual scalability of the model can be further improved.

[0098] In this embodiment, TF-IDF is a commonly used weighting technique for information retrieval and data mining. TF stands for Term Frequency, and IDF stands for Inverse Document Frequency. In this embodiment, the importance of a single command increases proportionally to the number of times the command appears in a single user operation command stream, and decreases inversely proportionally to the frequency of the aforementioned single command appearing in all user operation commands on the local machine. That is, it can be considered that the more times a command appears in the user's current statistical time range, and the fewer times it appears in all local operation command data, the more it can represent a different operation sequence, and the more obvious the characteristics of intrusion are.

[0099] In this embodiment, the formulas for calculating the term frequency (TF) and the inverse text frequency index (IDF) are as follows:

[0100]

[0101] TF-IDF = TF*IDF

[0102] That is, TF-IDF equals the TF value multiplied by the IDF value. The TF-IDF value calculates the local and global evaluation weights of a command in the current session and across all commands. It performs well in practical application fitting, thus yielding a weighted matrix-like result.

[0103] In this embodiment, the N-gram model uses a sliding window operation to maintain the execution order and dependencies, where n represents the size of the sliding window. For example, in the 2-gram model, two words are treated as a group, then shifted one word length to the right to form another group of words. These are used to generate a dictionary, where the dictionary value is the word following each pair of slid words. In the aforementioned step S1, each command is formatted into three parts. This reflects the continuity of the commands and the contextual dependencies between different commands. For example, for two consecutive commands: `cat / etc / sudoers` and `sudo -i account`, the 2-gram representation would be `[[cat, ...`. <1> ],[ <1> sudo],[sudo-i]] and [[sudo],[-i],[ <1> Of course, these commands are represented after numerical encoding. The above methods can be used to obtain the relevant feature matrices;

[0104] S5, HMM model specification estimation, output model parameters;

[0105] The HMM model used in this embodiment is a double stochastic process, including: the state transitions of the model and the stochasticity of observable events under a specific state;

[0106] like Figure 5 As shown, in this embodiment, X represents the hidden state sequence, which in this embodiment represents two types: [this user, intruding user], and S represents the observation state, which in this embodiment represents the command stream data after feature processing; the value of the state sequence at time t depends only on the value of the state sequence at time t-1, and the value of the observation sequence at time t depends only on the value of the observation sequence at time t. Based on known parameters, a Hidden Markov Model is generally denoted as:

[0107] λ = [π, A, B]

[0108] Wherein, the initial state probability π: the probability of each state occurring in the model at the initial time, that is, the probability of normal user account operation and the probability of intrusion user account operation.

[0109] State transition probability A: The probability of the model transitioning between various states, that is, the probability of switching from normal account use to command operation of an intrusive account. It should be a 2x2 probability matrix.

[0110] Output observation probability B: The probability matrix of each observation obtained by the model based on the current state, that is, the feature representation of each command stream;

[0111] In this embodiment, an Hidden Markov Model (HMM) is used for evaluation, prediction, and learning operations. In the evaluation operation, given the model parameters λ = [π, A, B] and the observation sequence Q, the probability of the observed sequence occurring is calculated. After training a model or obtaining model parameters from existing data, the probability of a given observation sequence occurring can be calculated. This problem is generally solved using a forward-backward algorithm.

[0112] In the prediction operation, i.e., the decoding operation, of this embodiment, given the model parameters λ = [π, A, B] and the observation sequence Q, the most likely state sequence corresponding to the observation sequence is calculated. After training a model or obtaining model parameters from existing data, given an observation sequence, we are more concerned with the hidden state sequence—whether it represents normal user operation or an operation by a disguised intruder. A commonly used method to solve this prediction problem is the Viterbi algorithm, a dynamic programming algorithm, which can also be understood as a search space pruning method that guarantees finding the globally optimal path.

[0113] In the learning operation of this embodiment, the parameter estimation problem involves unknown model parameters λ = [π, A, B]. Given an observation sequence, the model parameters are inferred. When there is no readily available label data, a convergent model needs to be trained and learned from a large stream of execution commands. This model can be the Hidden Markov Model (HMM) required for evaluation and prediction problems. The Expectation-Maximization (EM) algorithm is generally used to solve this problem.

[0114] In this embodiment, the Baum-Welch algorithm is used for parameter estimation to construct a contour Hidden Markov Model (HMM), which incorporates a position score matrix into the structure. In this embodiment, based on the data preprocessing and feature representation steps above, the required observation matrix is ​​constructed, input into the constructed model, and training and validation begin until the set convergence threshold is reached, at which point the model is output.

[0115] like Figure 6 As shown, in this embodiment, the process of parameter estimation using the Baum-Welch algorithm further includes the following specific steps:

[0116] S51. Initialization;

[0117] In this embodiment, initial values ​​are assigned through initialization, and model parameters are randomly selected:

[0118] λ = [π, A, B], when n = 0.

[0119] S52. Iterate through the parameter updates, updating the parameters iteratively for n = 1, 2, 3...

[0120] S53. Stop iterating until the model parameters converge;

[0121] In this embodiment, parameter estimation can be performed directly using, for example, the hmmlearn library. In this embodiment, it is known that the evaluated parameters have a very low rate of spoofing intrusion. Further model testing was conducted using log data with historical intrusion traces, and it was found that the HMM model with added location parameters has better performance.

[0122] In summary, this invention utilizes Profile Hidden Markov Models (PHMM) for host intrusion detection. It directly uses operational commands, such as Linux commands, for further analysis. It considers the location information of command execution to perform host spoofing intrusion detection, improving the accuracy and precision of host intrusion detection. Furthermore, this invention differs in its feature extraction methods. This application presents methods borrowing from natural language processing, such as word embedding, word sliding windows, word frequency, and inverse text frequency statistics, further considering command execution order dependencies and word frequency issues.

[0123] This invention adds an index identifier to the optimal score matrix to generate the required contour characterization matrix, which is then added as a parameter factor in model training to achieve better detection results.

[0124] This invention utilizes a magnetic tape model to identify frequently used command words or operations by an account, assigning different levels of attention to different commands to further improve the model's practical scalability. Simultaneously, it employs an N-gram model to execute sliding window operations, maintaining the execution order and dependencies.

[0125] The Hidden Markov Model used in this invention incorporates a contour matrix, which can be understood as a multi-sequence alignment score matrix. This is equivalent to adding a weighted term during feature processing, taking into account different levels of host command execution and execution order.

[0126] The contour hidden map model used in this invention considers the dependency of command position and context execution order. The model trained with a large amount of data has higher accuracy and faster running speed, and better model extensibility. It can detect unknown intrusion behavior in an unsupervised manner.

[0127] This invention solves the technical problems of existing technologies, such as the single detection method, lack of flexibility, and low robustness, scalability, and host intrusion detection accuracy due to reliance on a large amount of data annotation.

[0128] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A camouflage intrusion detection method based on a profile hidden Markov model, characterized by, The method includes: S1. During the data loading and extraction process, the server's built-in audit logs are obtained to acquire log data. The Linux operation commands in the log data are then formatted in a standard way to complete the data loading and extraction operation and obtain standardized command data. S2. Use the encoder's self-encoding method to perform numerical transcoding mapping on the standardized command data to generate the execution command stream block; S2 includes: S21. Input the standardized command data into the autoencoder to generate the corresponding ID; S22. Encode and map the values ​​in the standardized command according to a preset threshold. S23. Use the value less than or equal to the preset threshold as the beginning and end of each standardized command and the start and end flag of the preset session; S24. Sort the pre-encoded sequence according to the execution of the preset session, and generate the execution command stream block corresponding to each account in blocks; S3. By comparing the global, semi-global and local contents of the precoded sequence through multiple sequence alignment, a comparison score and a similarity score are generated. A pre-set test segment in the standardized command data is taken and aligned with the user signature to construct the user command execution profile parameter matrix and obtain intrusion indicator data within the pre-set test segment. S3 includes: S31. Prepare the historical operation command records of each account on the current host, and use them as a reference comparison sequence. Convert the historical operation command records into an encoded historical sequence. S32. Alignment operation is performed on the reference comparison sequence and the encoded history sequence. The reference comparison sequence and the encoded history sequence are compared in a dictionary manner to obtain a matching pattern score. The alignment operation includes padding and truncate. S33. Calculate the score matrix based on the matching pattern score to obtain the local anomaly intrusion degree data of the sequence string of the standardized command data; S34. Initialize the floating matrix to store the alignment operation scores, calculate the optimal score matrix based on it, add an index identifier to the optimal score matrix to generate the contour characterization matrix, which serves as the model fitting parameter factor for model training. S4. Using the feature engineering module, in the case of discrete data, input the command stream data of the encoded annotated commands to perform word embedding operations, and use pre-set labeling methods to identify the importance and weight of each input unit. The pre-set labeling methods include: Bag-of-words model, TF-IDF (Term Frequency-Inverse Document Frequency) model and N-gram model. S5. Use Hidden Markov Model (HMM) to perform model state transitions and observable event operations under specific states to obtain command flow feature data for setting estimation, training and setting the model parameters of the Hidden Markov Model, and using the Hidden Markov Model (HMM) to perform evaluation, prediction and learning operations to complete observable event operations under specific states, obtain the command flow feature performance probability matrix, and obtain the spoofing intrusion detection results. S5 includes: S51. Based on the model parameters, use the forward-backward algorithm to perform an evaluation operation to obtain the probability of occurrence of a given observation sequence; S52. Based on the model parameters, use the Viterbi algorithm to perform prediction operations, determine the hidden state column in the given observation sequence, and process the data of the disguised intruder operation. S53. Solve using the EM expectation-maximization algorithm, train and learn to obtain a suitable Hidden Markov Model, wherein the Baum-Welch algorithm is used for parameter estimation, and a position score matrix is ​​added to the pre-set network structure to construct a contour Hidden Markov Model, train and validate the contour Hidden Markov Model, and obtain a suitable Hidden Markov Model.

2. The camouflage intrusion detection method based on profile hidden Markov model according to claim 1, characterized in that, Step S33 further includes: S331. Use the Smith-Waterman local sequence alignment algorithm to find the optimal local alignment modification parameters relative to the preset scoring system; S332. Calculate the semi-global alignment data of the sequence of the standardized command data to obtain the score matrix, wherein the preset scoring system includes: a replacement matrix and a gap scoring scheme.

3. The camouflage intrusion detection method based on profile hidden Markov model according to claim 1, characterized in that, Step S34 further includes: S341. Perform diagonal processing on the floating matrix to obtain the instruction alignment penalty and reward of the historical operation command set his_data and the input dataset val_data; S342. Process the floating matrix from top to bottom to introduce gap penalty in the historical operation command set his_data; S343. Perform left-right processing on the floating matrix to introduce a penalty for gaps in the input dataset val_data.

4. The camouflage intrusion detection method based on profile hidden Markov model according to claim 1, characterized in that, Step S4 includes: S41. Using the bag-of-words model, perform word frequency descending statistics on all unit commands in the command stream data, and vectorize each segment of the command stream data to obtain the vector value of the number of occurrences of specific elements, so as to obtain the attention of differential commands. S42. Use the term frequency-inverse document frequency model TF-IDF to perform weighting operations, thereby determining and obtaining intrusion features, and process to obtain the local assessment weight value and global assessment weight value of the standardized command corresponding to the command stream data, so as to obtain a similar weighting matrix. S43. Perform a sliding window operation using the N-gram model to obtain the relevant feature matrix.

5. The camouflage intrusion detection method based on profile hidden Markov model according to claim 1, characterized in that, In step S5, the Hidden Markov Model (HMM) is expressed using the following logic: wherein is the initial state probability, is the state transition probability, is the output observation probability.

6. A camouflage intrusion detection system based on profile hidden Markov model for performing the camouflage intrusion detection method based on profile hidden Markov model according to any one of the preceding claims 1 to 5, characterized in that, The system includes: The data loading and extraction module is used to obtain the server's built-in audit logs during the data loading and extraction process, thereby obtaining log data. The Linux operation commands in the log data are then formatted in a standard way to complete the data loading and extraction operation and obtain standardized command data. The transcoding and mapping module is used to perform numerical transcoding and mapping of the standardized command data using the encoder's self-encoding method to generate execution command flow blocks. The transcoding and mapping module is connected to the data loading and extraction module. The alignment module is used to compare the global, semi-global, and local contents of the precoded sequence through multi-sequence alignment to generate a comparison score and a similarity score. It takes a preset test segment from the standardized command data, aligns it with the user signature, constructs a user command execution profile parameter matrix based on it, and obtains intrusion indicator data within the preset test segment. The alignment module is connected to the data loading and extraction module. The feature engineering module is used to input the encoded annotated command stream data in the case of discrete data to perform the word embedding operation, and to identify the importance and weight of each input unit using a preset labeling method, wherein the preset labeling method includes: Bag-of-words model, TF-IDF, and N-gram model. The feature engineering module is connected to the transcoding and mapping module and the data loading and extraction module. The HMM model intrusion detection module is used to perform model state transitions and observable event operations under specific states using a Hidden Markov Model (HMM) to obtain command flow feature data for setting estimation, training and setting the model parameters of the HMM, and using the HMM to perform evaluation, prediction and learning operations to complete the observable event operations under the specific states, thereby obtaining a command flow feature performance probability matrix to obtain the disguised intrusion detection result. The HMM model intrusion detection module is connected to the feature engineering module.

Images