Hybrid internal threat detection method based on DACGAN-Transform

Abstract

The invention relates to the technical field of data processing, provides a hybrid internal threat detection method based on DACGAN-Transform, and solves the problems of data imbalance and lack of fine-grained analysis of a detection method in internal threat detection. According to the method, samples which are similar to normal data in distribution but have abnormal characteristics are generated through a generative adversarial network (GAN), a data set is enhanced, abnormity is preliminarily judged, hierarchical characteristic extraction is conducted on log data through a Transform model, abnormity detection comprises single log abnormity detection and context abnormity detection, and the fine granularity and accuracy of detection are improved. And finally, in combination with the overall abnormal score of the GAN and the layered abnormal score of the Transform, comprehensive evaluation is performed through a multi-layer perceptron, and whether the log entry is abnormal is judged, so that the internal threat detection precision and the system security are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a hybrid internal threat detection method based on DACGAN-Transformer. Background Art

[0002] Companies or enterprises are always at high risk of various cyber attacks. Cyber ​​attacks from within a company or enterprise's trusted entities are called insider threats, which usually include intentional fraud, theft of valuable information, or destruction of computer systems. In recent years, insider attacks have been on the rise. Although insider threat detection has been studied for many years, due to the lack of real data and data imbalance, it is difficult to find an effective solution for insider threat detection and it cannot achieve the desired effect. There is still a lot of room for development in this field.

[0003] The main difference between internal and external threats is the access and familiarity of the attacker. External threats usually come from external individuals or organizations that do not have access to the organization. Attackers may invade the system through phishing attacks, malware, etc. Internal threats, on the other hand, come from people or systems within the organization who already have access and may use these permissions to commit malicious or negligent acts. Detecting and defending against internal threats is relatively complex because they often occur within the legitimate users and systems of the organization.

[0004] Current internal threat detection methods can be mainly divided into two categories: detection methods based on abnormal behavior analysis and detection methods based on formal modeling. Detection methods based on abnormal behavior analysis are mainly divided into two detection methods based on audit logs and based on user commands. Audit log-based detection methods mainly analyze user behavior patterns (such as the frequency of accessing files, login time, email records, etc.) through logs left by user operations to identify abnormal behaviors, including those that have never occurred before, because they believe that any unusual situation is a potential attack. Most current methods do not consider the structural relevance of logs, but directly extract features from log data. They lack fine-grained analysis and cannot conduct in-depth analysis of different levels of information in log data. The user command anomaly detection method takes the user's command sequence as the analysis object, calculates the probability of adjacent command patterns, and determines whether it is abnormal by the degree of matching between new commands and historical commands. The internal threat discovery detection method based on formal modeling refers to detecting abnormal behaviors that deviate from the model by establishing a normal behavior model of the user and comparing the user's existing behavior. Another important work of this method is the internal threat detection method based on graph methods, which identifies potential internal threats by modeling and analyzing graph structures related to users, systems, resources and their interactions, such as attack graphs, attack trees, Petri nets, etc.

[0005] In addition, with the continuous development of machine learning and deep learning technologies, the trend of using machine learning and deep learning for internal threat detection is rising. However, traditional machine learning methods rely heavily on feature engineering and it is difficult to accurately capture the difference in behavior patterns between normal users and internal attackers. Advanced deep learning technology can extract high-dimensional features from complex data, but it still relies on data enhancement to alleviate data shortages.

[0006] In order to alleviate the problem of data imbalance and improve the generalization ability of the model, data augmentation technology is widely used. Data augmentation expands the data set by generating new samples, thereby enhancing the training effect and robustness of the model. Common data augmentation strategies include transforming the data, adding noise, or randomly selecting samples from the transformation space. However, even if a certain enhancement strategy can effectively improve the performance of the data set in a specific application, it does not mean that it is also applicable to other data sets and applications. In addition, due to the limitation of dimensionality, this sampling method is limited in its expressiveness and cannot be extended to complex transformations that rely on many parameters.

[0007] Generative adversarial network (GAN) is a generative deep learning model that generates new data from a given original data set through the mutual game learning between the generator and the discriminator, where the generated data points simulate the real data distribution. As a strategy for data enhancement, GAN is different from other synthetic data generation methods in that it can create realistic data samples, thereby effectively reducing the problem of data imbalance and the final risk of overfitting. GAN was originally widely used in the fields of image and video processing, but has now been extended to process structured and unstructured data. Based on traditional GAN, conditional GAN ​​(CGAN) guides data generation by introducing conditional variables (such as category labels or other auxiliary information), so that the generated samples are more in line with the requirements of specific conditions. Auxiliary classifier GAN (ACGAN) further develops this idea. In addition to using the discriminator to determine whether the data is real or generated, an auxiliary classifier is added at the end of the discriminator to output the category label of the data, thereby further enhancing the ability of the generator. However, the auxiliary classifier shares model parameters with the discriminator, and the classification effect is limited.

[0008] The existing technologies have the following main shortcomings: the existing data sets are small in number, and the scenarios provided are not rich and real enough, and are not representative enough to effectively reflect the real internal threat situation. The existing data sets are unbalanced, and the number of negative samples is far less than the positive samples, which affects the training effect of the deep learning model. Considering that the log formats of different systems are also different, direct text processing is complex and has a lot of data redundancy. Through template parsing, the fixed part of the log is extracted as a template, and the changing part is stored as a parameter. The log structure is clearer, which reduces the processing difficulty and reduces the data dimension and redundancy. The existing detection methods lack fine-grained analysis, and traditional detection methods cannot conduct in-depth analysis of different levels of information in log data. Summary of the invention

[0009] In view of the shortcomings of the existing technology, the present invention utilizes the ability of generative adversarial networks (GANs) to generate data that conforms to real scenarios and learn features at the same time, and the advantage of Transformer in processing long sequence data, and proposes a hybrid deep learning model of generative adversarial networks (DACGAN) for data enhancement and classification and Transformer hierarchical anomaly detection to generate high-quality data and parse log data, extract features from the template and parameter parts of the log respectively, and simultaneously detect and analyze single log anomalies and context anomalies, thereby improving the performance of internal threat detection.

[0010] In order to solve the above technical problems, the specific technical solutions of the present invention are as follows: The present invention provides a hybrid internal threat detection method based on DACGAN-Transformer, comprising: Step S101, obtaining log data, the log data including server logs, application logs and network device logs, preprocessing the log data to obtain preprocessed log data; Step S102, substituting the preprocessed log data into the generative adversarial network, the preprocessed log data is trained by the generator, discriminator, and independent classifier in the generative adversarial network, the generator generates samples with similar distribution to normal data but with abnormal characteristics, and the classifier gives an overall abnormality score; Step S103, using a preset algorithm or other log parsing method to analyze the log data, extracting the fixed structure part of each log entry, generating a log template, and extracting the dynamic information in each log entry as a parameter part; Step S104, mapping a single log entry template into a vector, extracting global and local features, outputting a fixed-length template representation vector, mapping log parameters into a vector, extracting global and local features, outputting a fixed-length parameter representation vector, encoding the template and parameter feature vectors of the log sequence respectively, and representing the template context information and parameter context information of the log sequence; Step S105: linearly transform and normalize the template representation and parameter representation of a single log entry, output the anomaly score of a single log, linearly transform and fuse the template sequence representation and parameter sequence representation of the log sequence, learn the context dependency through the Transformer Block preset tool model, output the context anomaly score, and integrate the single log anomaly score and the context anomaly score through linear weighting to obtain the Transformer hierarchical anomaly score. Step S106, input the GAN overall anomaly score and the Transformer layered anomaly score into a multi-layer perceptron, output the final anomaly probability through a preset activation function, and determine whether the log entry is abnormal based on a preset threshold.

[0011] Furthermore, in the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S101 includes: Get server logs, application logs, and network device logs from the logging system; Remove noise data from server logs, application logs, and network device logs. Noise data includes redundant log records, abnormal characters, and missing values. Filter low-frequency server logs, application logs, and network device logs by setting thresholds, filter useless data in server logs, application logs, and network device logs, use character filters to remove abnormal characters in server logs, application logs, and network device logs, and use mean interpolation methods to complete log entries for missing values ​​in server logs, application logs, and network device logs. Obtain data conversion rules, map timestamps, IP addresses, user IDs, and operation types in server logs, application logs, and network device logs to standard fields, use the time parsing library to convert timestamps in different formats into a unified time format, use a sorting algorithm to sort log records by timestamp, and output a chronologically ordered log list; The server logs, application logs, and network device log data are output after cleaning, formatting, timestamp alignment, and quality checking as input for subsequent generative adversarial network processing and log parsing.

[0012] Furthermore, in the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S102 includes: Input the preprocessed log data into the generative adversarial network and initialize the network parameters of the generator, discriminator, and independent classifier; The generator receives random noise and label inputs and generates samples that are similar to normal data distribution but have abnormal characteristics; The discriminator receives samples generated by the generator that are similar to normal data distribution but have abnormal characteristics and real samples, judges their authenticity, and feeds the results back to the generator. After receiving the feedback results, the generator dynamically updates the weights and optimizes the network through continuous iterative weights. The independent classifier classifies the samples generated by the generator, determines whether the samples corresponding to the samples generated by the generator are normal or abnormal, and obtains the output result of the independent classifier; According to the output results of the independent classifier, the loss functions of the generator, discriminator and classifier are calculated, and the network parameters are optimized through back propagation. During the training process, the classification results of the independent classifier for each generated sample, that is, the overall anomaly score, are recorded.

[0013] Furthermore, in the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S103 includes: The preprocessed log data is used as input, and the preprocessed log data is input into the log parsing module, and each log entry is selected one by one from the input log data for parsing, and the log parsing algorithm is used. The log parsing algorithm adopts the Drain algorithm to extract the fixed structure part of each log entry and generate a log template; Extract dynamic information from each log entry, the dynamic information including file path, visitor ID and timestamp, the file path, visitor ID and timestamp as parameter parts; The extracted templates are paired with corresponding parameters to form template-parameter correspondence information, and the template-parameter correspondence information is integrated to obtain log entry representation.

[0014] Furthermore, in the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S104 includes: Map the single log entry template into a vector form and input it into the single log template encoder for feature extraction; The global and local features of the log template are extracted through the preset tool model in the single log template encoder, and the preset tool model adopts Transformer Block; The extracted features are processed through an average pooling layer to generate a template representation vector of fixed length; A single log entry parameter is mapped into a vector form and input into a single log parameter encoder for feature extraction. The global and local features of the log parameter are extracted through the preset tool model in the single log parameter encoder. The preset tool model uses Transformer Block to process the extracted features through an average pooling layer to generate a fixed-length parameter representation vector.

[0015] Furthermore, in the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S105 includes: Perform linear transformation and normalization on the template representation and parameter representation of a single log entry, and output the anomaly score of the single log; Perform linear transformation and fusion on the template sequence representation and parameter sequence representation of the log sequence, learn the context dependency through the Transformer Block preset tool model, and then output the context anomaly score through linear transformation and normalization; The obtained single log anomaly score and context anomaly score are fused in a linear weighted manner to obtain a hierarchical anomaly score.

[0016] Furthermore, in the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S106 includes: Transformer hierarchical anomaly scores include single log anomaly scores and contextual anomaly scores, integrating the local and global anomaly levels at the fine-grained level of log entries; Take the overall anomaly score of GAN and the layered anomaly score of Transformer architecture as input data; The multi-layer perceptron processes the input scores and extracts and integrates features through a multi-layer neural network; The output of the multilayer perceptron is processed by a preset activation function, and the output is limited to the range of [0,1] as the final abnormal probability; The final abnormal probability is determined according to the preset threshold. If the abnormal probability exceeds the threshold, the log entry is determined to be abnormal. Otherwise, it is determined to be normal and the abnormal determination result of the log entry is output.

[0017] Beneficial effects of the present invention: This invention proposes innovative solutions to key issues in internal threat detection by introducing GAN data enhancement and Transformer hierarchical detection technology, especially solving the challenges of positive and negative sample imbalance, multimodal log data processing, fine-grained feature extraction and multi-layer anomaly detection. First, the abnormal samples generated by GAN significantly alleviate the problem of scarce abnormal samples, allowing the model to learn richer feature representations during training, enhancing the model's defense capabilities against unknown attacks, thereby improving the accuracy and detection effect of internal threat detection. In addition, the independent classifier design not only enhances the classification performance of GAN, but also optimizes the quality of the generator's abnormal data, reducing false positives and false negatives.

[0018] The data preprocessing module cleans, formats, and timestamp-aligns the raw log data, ensuring data availability and consistency, and providing a high-quality data foundation for subsequent processing. The application of log parsing technologies such as the Drain algorithm makes the parsing of log data more efficient and accurate, improving the efficiency of the entire detection process. The overall architecture fully utilizes the advantages of multimodal data processing, and can automatically and comprehensively analyze different types of log data, reducing the need for manual intervention and improving the practicality and scalability of the internal threat detection system.

[0019] On the other hand, the feature extraction and anomaly detection module based on the Transformer hierarchical structure can effectively extract templates and numerical features from log data and conduct detailed abnormal behavior analysis. By simultaneously detecting anomalies in single logs and contextual log sequences, the detection accuracy of complex threat patterns is improved. The weights of the two are adjusted according to actual needs, so as to respond to different types of internal threats more flexibly, significantly improving the generalization ability and robustness of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, for ordinary technicians in this field, other drawings can be obtained based on the drawings without paying any creative labor.

[0021] Figure 1 A flow chart of a hybrid internal threat detection method based on DACGAN-Transformer provided in an embodiment of the present invention.

[0022] Figure 2 A schematic diagram of a model provided for an embodiment of the present invention.

[0023] Figure 3 A schematic diagram of a log parsing example provided by an embodiment of the present invention.

[0024] Figure 4 A schematic diagram of the structure of a single log template encoder provided by an embodiment of the present invention.

[0025] Figure 5 A schematic diagram of the structure of a single log parameter encoder provided in an embodiment of the present invention.

[0026] Figure 6 A schematic diagram of the structure of a template sequence encoder provided in an embodiment of the present invention.

[0027] Figure 7 A schematic diagram of single log anomaly detection provided by an embodiment of the present invention.

[0028] Figure 8A schematic diagram of contextual anomaly detection provided by an embodiment of the present invention.

[0029] Fig. 9 A schematic diagram of a comprehensive evaluation process provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described in conjunction with the specific embodiments of the present invention and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention. The technical solutions provided by the embodiments of the present invention are described in detail below in conjunction with the drawings.

[0031] In order to better understand the purpose of the present invention, the present invention is further described in detail below. Figures 1 to 9 .

[0032] The present invention provides a hybrid internal threat detection method based on DACGAN-Transformer, comprising: Step S101, obtaining log data, the log data including server logs, application logs and network device logs, preprocessing the log data to obtain preprocessed log data; Step S102, substituting the preprocessed log data into the generative adversarial network, the preprocessed log data is trained by the generator, discriminator, and independent classifier in the generative adversarial network, the generator generates samples with similar distribution to normal data but with abnormal characteristics, and the classifier gives an overall abnormality score; Step S103, using a preset algorithm or other log parsing method to analyze the log data, extracting the fixed structure part of each log entry, generating a log template, and extracting the dynamic information in each log entry as a parameter part; Step S104, mapping a single log entry template into a vector, extracting global and local features, outputting a fixed-length template representation vector, mapping log parameters into a vector, extracting global and local features, outputting a fixed-length parameter representation vector, encoding the template and parameter feature vectors of the log sequence respectively, and representing the template context information and parameter context information of the log sequence; Step S105: linearly transform and normalize the template representation and parameter representation of a single log entry, output the anomaly score of a single log, linearly transform and fuse the template sequence representation and parameter sequence representation of the log sequence, learn the context dependency through the Transformer Block preset tool model, output the context anomaly score, and integrate the single log anomaly score and the context anomaly score through linear weighting to obtain the Transformer hierarchical anomaly score. Step S106, input the GAN overall anomaly score and the Transformer layered anomaly score into a multi-layer perceptron, output the final anomaly probability through a preset activation function, and determine whether the log entry is abnormal based on a preset threshold.

[0033] Specifically, the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S101, includes: Get server logs, application logs, and network device logs from the logging system; Remove noise data from server logs, application logs, and network device logs. Noise data includes redundant log records, abnormal characters, and missing values. Filter low-frequency server logs, application logs, and network device logs by setting thresholds, filter useless data in server logs, application logs, and network device logs, use character filters to remove abnormal characters in server logs, application logs, and network device logs, and use mean interpolation methods to complete log entries for missing values ​​in server logs, application logs, and network device logs. Obtain data conversion rules, map timestamps, IP addresses, user IDs, and operation types in server logs, application logs, and network device logs to standard fields, use the time parsing library to convert timestamps in different formats into a unified time format, use a sorting algorithm to sort log records by timestamp, and output a chronologically ordered log list; The server logs, application logs, and network device log data are output after cleaning, formatting, timestamp alignment, and quality checking as input for subsequent generative adversarial network processing and log parsing.

[0034] Specifically, the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S102 includes: Input the preprocessed log data into the generative adversarial network and initialize the network parameters of the generator, discriminator, and independent classifier; The generator receives random noise and label inputs and generates samples that are similar to normal data distribution but have abnormal characteristics; The discriminator receives samples generated by the generator that are similar to normal data distribution but have abnormal characteristics and real samples, judges their authenticity, and feeds the results back to the generator. After receiving the feedback results, the generator dynamically updates the weights and optimizes the network through continuous iterative weights. The independent classifier classifies the samples generated by the generator, determines whether the samples corresponding to the samples generated by the generator are normal or abnormal, and obtains the output result of the independent classifier; According to the output results of the independent classifier, the loss functions of the generator, discriminator and classifier are calculated, and the network parameters are optimized through back propagation. During the training process, the classification results of the independent classifier for each generated sample, that is, the overall anomaly score, are recorded.

[0035] Specifically, the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S103 includes: The preprocessed log data is used as input, and the preprocessed log data is input into the log parsing module, and each log entry is selected one by one from the input log data for parsing, and the log parsing algorithm is used. The log parsing algorithm adopts the Drain algorithm to extract the fixed structure part of each log entry and generate a log template; Extract dynamic information from each log entry, the dynamic information including file path, visitor ID and timestamp, the file path, visitor ID and timestamp as parameter parts; The extracted templates are paired with corresponding parameters to form template-parameter correspondence information, and the template-parameter correspondence information is integrated to obtain log entry representation.

[0036] Specifically, the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S104, includes: Map the single log entry template into a vector form and input it into the single log template encoder for feature extraction; The global and local features of the log template are extracted through the preset tool model in the single log template encoder, and the preset tool model adopts Transformer Block; The extracted features are processed through an average pooling layer to generate a template representation vector of fixed length; The log parameters are mapped into vector form and input into the single log parameter encoder for feature extraction. The global and local features of the log parameters are extracted through the preset tool model in the single log parameter encoder. The preset tool model uses Transformer Block to process the extracted features through the average pooling layer to generate a fixed-length parameter representation vector.

[0037] Specifically, the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S105, includes: Perform linear transformation and normalization on the template representation and parameter representation of a single log entry, and output the anomaly score of the single log; Perform linear transformation and fusion on the template sequence representation and parameter sequence representation of the log sequence, learn the context dependency through the Transformer Block preset tool model, and then output the context anomaly score through linear transformation and normalization; The obtained single log anomaly score and context anomaly score are fused in a linear weighted manner to obtain a hierarchical anomaly score.

[0038] Specifically, the hybrid internal threat detection method based on DACGAN-Transformer described in the present invention, step S106, includes: Transformer hierarchical anomaly scores include single log anomaly scores and contextual anomaly scores, integrating the local and global anomaly levels at the fine-grained level of log entries; Take the overall anomaly score of GAN and the layered anomaly score of Transformer architecture as input data; The multi-layer perceptron processes the input scores and extracts and integrates features through a multi-layer neural network; The output of the multilayer perceptron is processed by a preset activation function, and the output is limited to the range of [0,1] as the final abnormal probability; The final abnormal probability is determined according to the preset threshold. If the abnormal probability exceeds the threshold, the log entry is determined to be abnormal. Otherwise, it is determined to be normal and the abnormal determination result of the log entry is output.

[0039] The present invention provides a hybrid internal threat detection method based on DACGAN-Transformer, which is applied to a hybrid internal threat detection system based on DACGAN-Transformer, such as Figure 2 As shown, a hybrid internal threat detection system based on DACGAN-Transformer includes a data preprocessing module, a GAN data enhancement module, a comprehensive evaluation module, a hierarchical feature extraction module and a hierarchical anomaly detection module.

[0040] The data preprocessing module is responsible for obtaining raw log data from the log system and cleaning, formatting and timestamp aligning it to ensure data availability and consistency.

[0041] Log acquisition: Acquire raw log data including server logs, application logs, network device logs, etc., and ensure that the log data contains information that may be abnormal.

[0042] Log cleaning: Clean the raw data, including methods for dealing with useless or redundant log records, abnormal characters, and missing values. Identify and filter useless logs by setting thresholds to filter low-frequency log records. Use regular expressions or character filters to remove abnormal characters. Use mean interpolation to complete log entries for missing values.

[0043] Formatting: Convert logs in different formats into standard JSON format through data conversion tools or custom scripts. Map key information fields in logs, such as timestamp, IP address, user ID, operation type, etc., to standard fields by writing data conversion rules.

[0044] JSON log format: { "timestamp":"2024-09-18T14:33:12Z", "level":"ERROR", "source":"server-1", "message":"Failedtoconnecttodatabase,errorcode:1049", "user_id":"12345", "ip_address":"192.0.2.x" } Original timestamp: Sep1814:33:12 Processed timestamp: 2024-09-18T14:33:12Z The format after extraction is as follows. The log information generated by subsequent GAN satisfies this format and can be parsed by the Drain algorithm.

[0045] 2024-09-18T14:33:12ZERRORserver-1:Failedtoconnecttodatabase,errorcode:1049,UserID:12345,IP:192.0.2.x Timestamp alignment: Use the time parsing library to convert timestamps of different formats into a unified time format that complies with the ISO8601 standard. Then use the sorting algorithm to sort the log records by timestamp, and the final output is a log list in chronological order.

[0046] The GAN data enhancement module generates more samples with similar distribution to normal data but with abnormal characteristics through generative adversarial networks (GAN) to solve the data imbalance problem in internal threat detection and improve the generalization ability of the detection model; at the same time, an independent classifier is added to generate abnormal samples and perform overall abnormality judgment.

[0047] The optimized adversarial loss of the traditional GAN ​​generator and discriminator is:

[0048] in, represents the probability that the discriminator believes that x is a real sample, and is the probability that the discriminator considers the synthetic sample to be fake. When training GAN, the discriminator wants to maximize the objective function, while the generator wants to minimize it. During the training process, one party is fixed while the other party updates the network weights, and the network is optimized alternately and iteratively until a dynamic balance is reached.

[0049] Generator: Uses standard generator architecture, such as DCGAN, WGAN, and the network can contain multiple layers of convolutional networks. Receives random noise and label inputs, and the output layer uses Sigmoid or Tanh activation function to map the generated samples to a range similar to the normal data distribution.

[0050] In the present invention, due to the addition of a separate classifier, the generator loss function includes adversarial loss and classification loss. The adversarial loss means that the generator hopes to maximize the probability that the discriminator determines that the generated sample is true. Therefore, the loss function output by the discriminator is used as the adversarial loss of the generator:

[0051] Where G(z) is the sample generated by the generator, and D(G(z)) is the output probability of the discriminator for the generated sample.

[0052] The generator also needs to generate samples with abnormal characteristics, so the classification loss is considered when generating data. The classification loss measures whether the generator generates abnormal samples by the classifier's prediction of the generated samples. The cross entropy loss can be used:

[0053] Among them, C(G(z)) is the predicted probability of the classifier for the generated sample, is the target label (normal or abnormal) of the generated sample.

[0054] The total loss of the generator combines the adversarial loss and the classification loss:

[0055] Discriminator: The structure is similar to that of the generator, and can use convolutional layers or fully connected layers. It is used to determine whether the input data is real data, and feed the results back to the generator for optimization to ensure that the generated data is close to the real distribution.

[0056] The discriminator loss function only has adversarial loss, whose goal is to distinguish between real samples and generated samples and update the weights of the generator through back propagation. The traditional discriminator loss uses binary cross entropy loss to distinguish between real data and generated data:

[0057] Among them, D(x) is the output of the discriminator for real samples, and D(G(z)) is the output of the discriminator for generated samples.

[0058] The independence of the classifier allows it to optimize the classification task of normal and abnormal samples without being affected by the discriminator.

[0059] Classifier: It is designed independently from the discriminator and is responsible for classifying the generated data. It can use traditional classification models, such as support vector machines (SVMs) and decision trees, or deep learning models, such as building a fully connected neural network classifier to classify the generated samples into two categories, i.e. normal or abnormal samples. The independence of the classifier allows for more flexible adjustment of weights and improves classification performance. Its output can be used as part of the model decision to enhance the accuracy of detection.

[0060] The classifier loss can be used to calculate the classification error of normal and abnormal samples through cross entropy loss.

[0061]

[0062] Among them, C(x) is the output of the classifier for the real sample, is the label of the real sample, 0 represents normal and 1 represents abnormal.

[0063] Finally, the losses of the generator, discriminator, and classifier are combined to form the loss function of the entire GAN model:

[0064] Log parsing module; Task: As Figure 3 As shown, the original log data is parsed into a template part and a parameter part, which facilitates subsequent modules to perform feature extraction and detection at different levels.

[0065] This module uses the Drain algorithm, which is a hierarchical tree structure mining method for log parsing. It can automatically convert structured and semi-structured logs into templates and parameter pairs. The original log is a structured JSON format after standard processing. Its "message" field can be extracted for Drain parsing, while retaining other structured fields, including timestamp, log level, log source, and log message.

[0066] Template extraction: Extract the fixed structure part of each log entry to generate a log template that represents the operation behavior or event type. For example, extract and standardize a template from the log, such as "[timestamp][log level][log source][log message]".

[0067] Parameter extraction: Extract dynamic information from each log entry, such as specific file path, visitor ID, timestamp, etc. Parameter information reflects the detailed content of the log, including more personalized and dynamic behavior patterns, such as visitor ID 67890, IP address 192.0.2.x, etc.

[0068] Transformer hierarchical feature extraction module; Task: This module performs multi-level feature extraction on log data through a single log encoder and a sequence encoder. The single log encoder is used to process the template and parameter parts of a single log, while the sequence encoder is used to process the template and parameters in a log sequence to capture the fine-grained information, time dependency, and context dependency in the log.

[0069] Single log template encoder: used to encode the template part in a single log, such as Figure 4 As shown, first map each word in the log template to a vector x, , where d is the dimension of the word vector. Assuming the log template length is n, we can get the vector list .

[0070] The preprocessed template vector is input into two Transformer Blocks. Each Transformer Block includes a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and residual connections to capture global and local features in the log template. The output of each Transformer Block is layer normalized and weighted updated through residual connections. After passing through two Transformer Blocks, the output Finally, the average pooling layer is used to reduce the dimension and generate a fixed-length template representation vector , .

[0071] Single log parameter encoder: Figure 5As shown, the structure is similar to the single log template encoder structure, which is used to process the parameter part in a single log. First, each feature in the log parameter is mapped to a vector y, y∈R1×d, where d is the dimension of the word vector. Assuming that the log parameter length is m, the vector list can be obtained. After two Transformer Block blocks extract the details and dependencies in the parameters, the output of each Block is optimized through layer normalization and residual connection, and the output Finally, the parameter feature vector is reduced to a parameter representation of the same length as the template representation using the same pooling strategy as the single log template encoder. , .

[0072] Template sequence encoder: Receives the template feature vector sequence of all logs, performs further encoding, and obtains the template sequence representation with context information. Similarly, the parameter sequence encoder further encodes the parameter sequence to obtain the parameter sequence representation. Its structure is consistent with the template sequence encoder, and is composed of 4 Transformer Blocks. Each block also has a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and residual connection. Finally, a fixed-length vector is output through the pooling layer. To avoid repetition, only the structure and process of the template sequence encoder are shown, as shown in the figure. Figure 6 shown.

[0073] Assuming that a log sequence contains N logs, the template features are extracted from these N logs through a single log encoder to obtain ,in Then, the global context features are extracted through Transformer to capture deeper dependencies and obtain the template sequence representation. The parameter sequence encoding is similar, and the final output is expressed as .

[0074] The Transformer hierarchical anomaly detection module is used to perform anomaly detection on single log entries and log sequences. The module first performs simple anomaly detection on single log entries to identify anomalies such as log format and field values; then, it performs contextual anomaly detection on log sequences containing multiple logs to capture more complex attack patterns and behaviors. Through multi-level detection methods, the accuracy of anomaly detection can be effectively improved, especially for potential complex threats.

[0075] Single log anomaly detection: Figure 7 As shown, the input is the template representation of a single log entry obtained from the Transformer hierarchical feature extraction module And parameter representation These representation vectors are feature extracted and contain global and local information in the log entries. The template representation and parameter representation are passed through a linear transformation layer respectively, which projects the high-dimensional vectors into a lower-dimensional anomaly detection space so that it is easier to identify abnormal relationships between features.

[0076]

[0077] The template and parameter features after linear transformation are mapped to the [0,1] interval through sigmoid to represent the anomaly score.

[0078] Contextual anomaly detection: e.g. Figure 8 As shown, the input is a set of log template sequence representations and parameter sequence representations, which are obtained from the Transformer hierarchical feature extraction module and the global context dependency information extracted by the encoder. and parameter sequence representation Linear transformation is performed to fuse log sequence features, compressing high-dimensional context features into a lower-dimensional space that is easier to detect. Through this processing, potential abnormal patterns in the sequence can be identified.

[0079]

[0080] Then, six Transformer Blocks are used to learn the correlation between events in the log sequence. Finally, the features are linearly transformed and fused, and the contextual anomaly score of each log is output through softmax. , It represents the anomaly score of the i-th log under the log context condition. There are N logs in the log sequence.

[0081] After obtaining the single log anomaly score and the contextual anomaly score, the results of the two levels are integrated in a linear weighted manner to obtain the hierarchical anomaly score.

[0082]

[0083] in represents the stratified anomaly score, represents the anomaly score of a single log, Represents the anomaly score of the context (sequence). represents the weight of the single log score, Represents the weight of the context score.

[0084] The weights can be set based on the business needs of the system, the threat model is relatively stable, or the experience of domain experts can be used to set the weights. If there are more abnormal single logs, you can set =0.3, otherwise increase .

[0085] In actual scenarios, the technical solution of the present invention is implemented. For example, in a company, an internal attacker may not directly expose anomalies, but steal information in a more complex way. For example, if a user copies a file, the single anomaly score is 0.4, but the file is copied on a holiday. The anomaly score combined with the context is 0.9, and the weight is set to , ,but .

[0086] Comprehensive assessment module: Fig. 9 As shown in the figure, the overall anomaly score of GAN and the hierarchical anomaly score of Transformer are combined, and the MLP performs a comprehensive anomaly score evaluation to finally determine whether the log entry is abnormal.

[0087] The input consists of two parts: GAN classifier overall anomaly score : Output from the GAN module, used to characterize the abnormality of the log entry relative to the GAN generation model and perform preliminary detection on the log.

[0088] Transformer Layered Anomaly Scores : Combined with single log anomaly score and contextual anomaly score The weighted result represents the degree of abnormality of the log entry from both local and global perspectives.

[0089] The output layer uses the Sigmoid activation function to limit the output to the range of [0,1] as the final abnormal probability. The abnormality is determined based on the set threshold.

[0090] The technical solution of the present invention solves the shortcomings of the prior art in the following ways: The present invention uses a generative adversarial network (GAN) to generate samples with abnormal characteristics that are similar to normal data distribution, thereby enhancing the data set and effectively alleviating the problem of scarcity of abnormal samples, thereby improving the training effect of the model. The Transformer model is used to perform hierarchical feature extraction on log data, which includes single log anomaly detection and contextual anomaly detection, significantly improving the granularity and accuracy of detection. The fixed structure part (i.e., template) and dynamic information (such as parameters) of the log entry are extracted through a log parsing algorithm (such as the Drain algorithm), which reduces the processing difficulty and reduces the data dimension and redundancy.

[0091] The present invention designs an independent classifier in GAN, which not only enhances the classification performance of GAN, but also optimizes the quality of abnormal data of the generator, thereby reducing false positives and false negatives. Transformer Block is used to extract global and local features of log templates and parameters, further improving the ability of feature representation. At the same time, anomaly detection is performed on single log entries and log sequences, and the single log anomaly score is integrated with the context anomaly score, which significantly improves the detection accuracy. The overall anomaly score of GAN and the hierarchical anomaly score of Transformer are input into a multi-layer perceptron (MLP), and the final anomaly probability is output through a preset activation function, and whether the log entry is abnormal is determined based on a preset threshold.

[0092] The present invention comprehensively evaluates the outputs of GAN and Transformer through a multi-layer perceptron, which significantly improves the accuracy of internal threat detection and enhances the security of the system.

[0093] In summary, the problems faced by existing internal threat detection technologies, such as data imbalance, insufficient fine-grained analysis, and weak model generalization ability, are effectively solved, greatly improving the accuracy of internal threat detection and the security of the entire system.

Claims

1. A hybrid internal threat detection method based on DACGAN-Transformer, characterized in that: include: Step S101, obtaining log data, the log data including server logs, application logs and network device logs, preprocessing the log data to obtain preprocessed log data; Step S102, substituting the preprocessed log data into the generative adversarial network, the preprocessed log data is trained by the generator, discriminator, and independent classifier in the generative adversarial network, the generator generates samples with similar distribution to normal data but with abnormal characteristics, and the classifier gives an overall abnormality score; Step S103, using a preset algorithm or other log parsing method to analyze the log data, extracting the fixed structure part of each log entry, generating a log template, and extracting the dynamic information in each log entry as a parameter part; Step S104, mapping a single log entry template into a vector, extracting global and local features, outputting a fixed-length template representation vector, mapping log parameters into a vector, extracting global and local features, outputting a fixed-length parameter representation vector, encoding the template and parameter feature vectors of the log sequence respectively, and representing the template context information and parameter context information of the log sequence; Step S105, linearly transform and normalize the template representation and parameter representation of a single log entry, output the anomaly score of a single log, linearly transform and fuse the template sequence representation and parameter sequence representation of the log sequence, learn the context dependency through the Transformer Block preset tool model, output the context anomaly score, and integrate the single log anomaly score and the context anomaly score through linear weighting to obtain the hierarchical anomaly score of the Transformer; Step S106, input the GAN overall anomaly score and the Transformer layered anomaly score into a multi-layer perceptron, output the final anomaly probability through a preset activation function, and determine whether the log entry is abnormal based on a preset threshold.

2. The hybrid internal threat detection method based on DACGAN-Transformer as claimed in claim 1, characterized in that: The step S101 includes: Get server logs, application logs, and network device logs from the logging system; Remove noise data from server logs, application logs, and network device logs. Noise data includes redundant log records, abnormal characters, and missing values. Filter low-frequency server logs, application logs, and network device logs by setting thresholds, filter useless data in server logs, application logs, and network device logs, use character filters to remove abnormal characters in server logs, application logs, and network device logs, and use mean interpolation methods to complete log entries for missing values ​​in server logs, application logs, and network device logs. Obtain data conversion rules, map timestamps, IP addresses, user IDs, and operation types in server logs, application logs, and network device logs to standard fields, use the time parsing library to convert timestamps in different formats into a unified time format, use a sorting algorithm to sort log records by timestamp, and output a chronologically ordered log list; The server logs, application logs, and network device log data are output after cleaning, formatting, timestamp alignment, and quality checking as input for subsequent generative adversarial network processing and log parsing.

3. The hybrid internal threat detection method based on DACGAN-Transformer as claimed in claim 1, characterized in that: The step S102 includes: Input the preprocessed log data into the generative adversarial network and initialize the network parameters of the generator, discriminator, and independent classifier; The generator receives random noise and label inputs and generates samples that are similar to normal data distribution but have abnormal characteristics; The discriminator receives samples generated by the generator that are similar to normal data distribution but have abnormal characteristics and real samples, judges their authenticity, and feeds the results back to the generator. After receiving the feedback results, the generator dynamically updates the weights and optimizes the network through continuous iterative weights. The independent classifier classifies the samples generated by the generator, determines whether the samples corresponding to the samples generated by the generator are normal or abnormal, and obtains the output result of the independent classifier; According to the output results of the independent classifier, the loss functions of the generator, discriminator and classifier are calculated, and the network parameters are optimized through back propagation. During the training process, the classification results of the independent classifier for each generated sample, that is, the overall anomaly score, are recorded.

4. The hybrid internal threat detection method based on DACGAN-Transformer as claimed in claim 1, characterized in that: The step S103 includes: The preprocessed log data is used as input, and the preprocessed log data is input into the log parsing module, and each log entry is selected one by one from the input log data for parsing, and the log parsing algorithm is used. The log parsing algorithm adopts the Drain algorithm to extract the fixed structure part of each log entry and generate a log template; Extract dynamic information from each log entry, the dynamic information including file path, visitor ID and timestamp, the file path, visitor ID and timestamp as parameter parts; The extracted templates are paired with corresponding parameters to form template-parameter correspondence information, and the template-parameter correspondence information is integrated to obtain log entry representation.

5. The hybrid internal threat detection method based on DACGAN-Transformer as claimed in claim 1, characterized in that: The step S104 includes: Map the single log entry template into a vector form and input it into the single log template encoder for feature extraction; The global and local features of the log template are extracted through the preset tool model in the single log template encoder, and the preset tool model adopts Transformer Block; The extracted features are processed through an average pooling layer to generate a template representation vector of fixed length; A single log entry parameter is mapped into a vector form and input into a single log parameter encoder for feature extraction. The global and local features of the log parameter are extracted through the preset tool model in the single log parameter encoder. The preset tool model uses Transformer Block to process the extracted features through an average pooling layer to generate a fixed-length parameter representation vector.

6. The hybrid internal threat detection method based on DACGAN-Transformer as claimed in claim 1, characterized in that: The step S105 includes: Perform linear transformation and normalization on the template representation and parameter representation of a single log entry, and output the anomaly score of the single log; Perform linear transformation and fusion on the template sequence representation and parameter sequence representation of the log sequence, learn the context dependency through the TransformerBlock preset tool model, and then output the context anomaly score through linear transformation and normalization; The obtained single log anomaly score and context anomaly score are fused in a linear weighted manner to obtain a hierarchical anomaly score.

7. The hybrid internal threat detection method based on DACGAN-Transformer as claimed in claim 1, characterized in that: The step S106 includes: Transformer hierarchical anomaly scores include single log anomaly scores and contextual anomaly scores, integrating the local and global anomaly levels at the fine-grained level of log entries; Take the overall anomaly score of GAN and the layered anomaly score of Transformer architecture as input data; The multi-layer perceptron processes the input scores and extracts and integrates features through a multi-layer neural network; The output of the multilayer perceptron is processed by a preset activation function, and the output is limited to the range of [0,1] as the final abnormal probability; The final abnormal probability is determined according to the preset threshold. If the abnormal probability exceeds the threshold, the log entry is determined to be abnormal. Otherwise, it is determined to be normal and the abnormal determination result of the log entry is output.

Images