A hybrid internal threat detection method based on DACGAN-Transformer

Abstract

The application relates to the technical field of data processing, and provides a hybrid internal threat detection method based on a DACGAN-Transformer, which solves the problems of data imbalance and lack of fine-grained analysis in internal threat detection. A generative adversarial network (GAN) is used to generate samples similar to normal data distribution but having abnormal characteristics, data set is enhanced, and abnormalities are preliminarily judged. A Transformer model is used for hierarchical feature extraction of log data. Abnormality detection includes single-log abnormality detection and context abnormality detection, and the fine granularity and accuracy of detection are improved. Finally, the overall abnormality score of the GAN and the hierarchical abnormality score of the Transformer are combined, a multilayer perceptron is used for comprehensive evaluation, and whether a log entry is abnormal is determined. The application effectively improves the precision of internal threat detection and the security of a system.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a hybrid internal threat detection method based on DACGAN-Transformer. BACKGROUND

[0002] Companies or enterprises are always at high risk of various network attacks. Network attacks from trusted entities within a company or enterprise are called internal threats, which usually include intentional fraud, theft of valuable information, or destruction of computer systems. In recent years, internal attacks have been on the rise. Although internal threat detection has been studied for many years, due to the lack of real data and data imbalance, internal threat detection has difficulty finding effective solutions and cannot achieve the desired effect, and there is still a lot of room for development in this field.

[0003] The main difference between internal threats and external threats lies in the access rights and familiarity of the attacker. External threats usually come from external individuals or organizations without access to the organization's internal access, and attackers may break into the system through phishing attacks, malware, etc. Internal threats are from personnel or systems within the organization, who already have access rights and may use these rights for malicious or negligent behavior. The detection and defense of internal threats are relatively complex because they often occur within the organization's legitimate users and systems.

[0004] Current internal threat detection methods can be mainly divided into two categories: detection methods based on abnormal behavior analysis and detection methods based on formal modeling. Detection methods based on abnormal behavior analysis mainly include two detection means based on audit logs and user commands. The detection method based on audit logs mainly analyzes the user's behavior pattern (such as the frequency of accessing files, login time, email records, etc.) by analyzing the logs left by user operations to identify abnormal behavior, including those activities that have never occurred before, as they believe that any unusual situation is a potential attack. Most current methods do not consider the structural relevance of logs, but directly extract features from log data, lack fine-grained analysis, and cannot perform in-depth analysis of different levels of information in log data. The abnormal detection method based on user commands takes the command sequence of the user as the analysis object, respectively calculates the probability of occurrence of adjacent command patterns, and the matching degree of new commands and historical commands to determine whether it is abnormal. The internal threat detection method based on formal modeling refers to establishing a normal behavior model of the user, and detecting abnormal behavior that deviates from the model by comparing the user's existing behavior. Another important work of this method is the internal threat detection method based on graph method, which models and analyzes the graph structure related to users, systems, resources, and their interactions, such as attack graphs, attack trees, Petri nets, etc. to identify potential internal threats.

[0005] In addition, with the continuous development of machine learning and deep learning technologies, the trend of using machine learning and deep learning for internal threat detection is rising. However, traditional machine learning methods rely heavily on feature engineering and are difficult to accurately capture the differences in behavior patterns between normal users and internal attackers. Advanced deep learning techniques can extract high-dimensional features from complex data, but still rely on data augmentation to alleviate data shortages.

[0006] To alleviate the problem of data imbalance and improve the generalization ability of the model, data augmentation techniques are widely used. Data augmentation expands the dataset by generating new samples, thereby enhancing the training effect and robustness of the model. Common data augmentation strategies include transforming data, adding noise, or randomly selecting samples from the transformed space. However, even if a certain augmentation strategy can effectively improve the performance of a dataset in a specific application, it does not mean that it is equally applicable to other datasets and applications. In addition, due to the limitation of dimensions, this sampling method is constrained in expression ability and cannot be extended to complex transformations that rely on numerous parameters.

[0007] Generative Adversarial Networks (GAN) is a generative deep learning model that generates new data from a given original dataset through the mutual game between the generator and the discriminator, where the generated data points simulate the true data distribution. As a strategy for data augmentation, GAN, unlike other synthetic data generation methods, can create realistic data samples, thereby effectively reducing the problem of data imbalance and reducing the risk of overfitting in the end. GAN was initially widely used in image and video processing fields, but has now been extended to processing structured and unstructured data. Based on traditional GAN, Conditional GAN (CGAN) introduces a conditional variable (such as a class label or other auxiliary information) to guide data generation, making the generated samples more consistent with the requirements of specific conditions. Auxiliary Classifier GAN (ACGAN) further develops this idea, in addition to determining whether the data is real or generated by the discriminator, it also adds an auxiliary classifier at the end of the discriminator to output the class label of the data, thereby further enhancing the ability of the generator, but the auxiliary classifier and the discriminator share model parameters, and the classification effect is limited.

[0008] The prior art mainly has the following disadvantages: the number of existing data sets is small, the provided scenarios are not rich and real enough, and the representativeness is not strong, so that the real internal threat situation cannot be effectively reflected. The existing data set is unbalanced, and the number of negative samples is much less than that of positive samples, thereby affecting the training effect of the deep learning model. Considering that the log formats of different systems are also different, directly processing the text is complex and has a large amount of data redundancy. Through template analysis, the fixed part of the log is extracted as a template, and the variable part is stored as a parameter, so that the log structure is more clear, the processing difficulty is reduced, and the data dimension and redundancy are reduced. The existing detection method lacks fine-grained analysis, and the traditional detection method cannot deeply analyze different levels of information in the log data. SUMMARY

[0009] In view of the deficiencies of the prior art, the present application utilizes the ability of the generative adversarial network (GANs) to generate data conforming to the real scene while learning features and the advantage of the Transformer in processing long sequence data, and proposes a hybrid deep learning model of a generative adversarial network (DACGAN) for data enhancement and classification and a Transformer hierarchical anomaly detection, to generate high-quality data and analyze the log data, extract features from the template and parameter parts of the log respectively, and simultaneously detect and analyze single-log anomalies and context anomalies, thereby improving the performance of internal threat detection.

[0010] To solve the above technical problems, the specific technical solutions of the present application are as follows:

[0011] The DACGAN-Transformer-based hybrid internal threat detection method of the present application comprises:

[0012] Step S101, log data is obtained, the log data comprising server logs, application program logs and network device logs, and the log data is preprocessed to obtain preprocessed log data;

[0013] Step S102, the preprocessed log data is substituted into the generative adversarial network, and the preprocessed log data is trained by the generator, the discriminator and the independent classifier in the generative adversarial network, the generator generates samples similar to the normal data distribution but with abnormal features, and the classifier gives an overall abnormal score;

[0014] Step S103, a preset algorithm or other log analysis method is used on the log data to extract the fixed structure part of each log entry, generate a log template, and extract dynamic information in each log entry as a parameter part;

[0015] Step S104, mapping the single log entry template into a vector, extracting global and local features, outputting a fixed-length template representation vector, mapping the log parameters into a vector, extracting global and local features, outputting a fixed-length parameter representation vector, respectively encoding the template and parameter feature vectors of the log sequence, representing the template context information and parameter context information of the log sequence;

[0016] Step S105, performing linear transformation and normalization on the template representation and parameter representation of the single log entry, outputting the anomaly score of the single log, performing linear transformation and fusion on the template sequence representation and parameter sequence representation of the log sequence, learning the context dependency relationship through the Transformer Block preset tool model, outputting the context anomaly score, integrating the single log anomaly score and the context anomaly score through linear weighting, obtaining the hierarchical anomaly score of the Transformer

[0017] Step S106, inputting the GAN overall anomaly score and the hierarchical anomaly score of the Transformer into a multilayer perceptron, outputting the final anomaly probability through a preset activation function, and determining whether the log entry is abnormal according to a preset threshold.

[0018] Further, the DACGAN-Transformer-based hybrid internal threat detection method provided by the present application, the step S101 comprises:

[0019] Obtaining server logs, application logs and network device logs from a log system;

[0020] Removing noise data of the server logs, the application logs and the network device logs, the noise data including redundant log records, abnormal characters and missing values, filtering low-frequency server logs, application logs and network device logs by setting a threshold, filtering useless data in the server logs, the application logs and the network device logs, removing abnormal characters in the server logs, the application logs and the network device logs using a character filter, and using a mean interpolation method to complete the log entries in the server logs, the application logs and the network device logs with missing values;

[0021] Obtaining data conversion rules, mapping timestamps, IP addresses, user IDs and operation types in the server logs, the application logs and the network device logs to standard fields, converting different formats of timestamps into a unified time format using a time parsing library, sorting the log records according to the timestamps using a sorting algorithm, and outputting a log list arranged in chronological order;

[0022] The server log, application log and network device log data after cleaning, formatting, timestamp alignment and quality check are output as input for subsequent generation of the adversarial network processing and log analysis.

[0023] Further, the DACGAN-Transformer-based hybrid internal threat detection method comprises the following steps:

[0024] The preprocessed log data is input into the generative adversarial network, and the network parameters of the generator, discriminator and independent classifier are initialized.

[0025] The generator receives random noise and label input, and generates samples similar to normal data distribution but with abnormal characteristics.

[0026] The discriminator receives the samples generated by the generator and the real samples, judges the authenticity of the samples, and feeds back the results to the generator.

[0027] The independent classifier classifies the samples generated by the generator, judges whether the samples generated by the generator are normal or abnormal, and obtains the output result of the independent classifier.

[0028] According to the output result of the independent classifier, the loss function of the generator, discriminator and classifier is calculated, and the network parameters are optimized through back propagation.

[0029] Further, the DACGAN-Transformer-based hybrid internal threat detection method comprises the following steps:

[0030] The preprocessed log data is input into the log analysis module, and each log entry is selected from the input log data for analysis.

[0031] The dynamic information is extracted from each log entry, and the dynamic information includes file path, visitor ID and timestamp.

[0032] The extracted template and corresponding parameters are paired to form template and parameter corresponding information, and the template and parameter corresponding information are integrated to obtain log entry representation.

[0033] Further, the DACGAN-Transformer-based hybrid internal threat detection method provided in the present application, the step S104 comprises:

[0034] The single log entry template is mapped into a vector form and input into a single log template encoder for feature extraction;

[0035] Global and local features of the log template are extracted by a preset tool model in the single log template encoder, and the preset tool model adopts a Transformer Block;

[0036] The extracted features are processed by an average pooling layer to generate a template representation vector with a fixed length;

[0037] The single log entry parameters are mapped into a vector form and input into a single log parameter encoder for feature extraction, global and local features of the log parameters are extracted by a preset tool model in the single log parameter encoder, the preset tool model adopts a Transformer Block, and the extracted features are processed by an average pooling layer to generate a parameter representation vector with a fixed length.

[0038] Further, the DACGAN-Transformer-based hybrid internal threat detection method provided in the present application, the step S105 comprises:

[0039] The template representation and the parameter representation of the single log entry are linearly transformed and normalized to output an abnormal score of the single log;

[0040] The template sequence representation and the parameter sequence representation of the log sequence are linearly transformed and fused, a context dependency relationship is learned by a Transformer Block preset tool model, and then a context abnormal score is output by linear transformation and normalization;

[0041] The obtained single log abnormal score and context abnormal score are fused by linear weighting to obtain a layered abnormal score.

[0042] Further, the DACGAN-Transformer-based hybrid internal threat detection method provided in the present application, the step S106 comprises:

[0043] The Transformer layered abnormal score comprises a single log abnormal score and a context abnormal score, and the abnormal degree of local and global under the log entry fine granularity is fused;

[0044] The overall abnormal score of the GAN and the layered abnormal score of the Transformer architecture are taken as input data;

[0045] The multilayer perceptron processes the score of the input, and features are extracted and integrated through a multilayer neural network;

[0046] The output of the multilayer perceptron is processed through a preset activation function, and the output is limited in the range of [0, 1] as the final anomaly probability;

[0047] The final anomaly probability is determined according to a preset threshold, if the anomaly probability exceeds the threshold, the log entry is determined as abnormal, otherwise, it is determined as normal, and the abnormal determination result of the log entry is output.

[0048] The beneficial effects of the present application are:

[0049] The present application introduces GAN data enhancement and Transformer hierarchical detection technology, and proposes an innovative solution for key problems in internal threat detection, especially solving the problems of positive and negative sample imbalance, multi-modal log data processing, fine-grained feature extraction and multi-layer anomaly detection challenges. First, the abnormal samples generated by GAN significantly alleviate the problem of scarcity of abnormal samples, so that the model can learn more rich feature representation in the training process, enhance the defense ability of unknown attacks, and thus improve the accuracy and detection effect of internal threat detection. In addition, the independent classifier design not only enhances the classification performance of GAN, but also optimizes the quality of abnormal data of the generator, reduces false positives and false negatives.

[0050] The original log data is cleaned, formatted and timestamp aligned through the data preprocessing module, ensuring the availability and consistency of the data, and providing a high-quality data basis for subsequent processing. The application of log parsing technology such as Drain algorithm makes the log data parsing more efficient and accurate, and improves the efficiency of the whole detection process. The overall architecture fully utilizes the advantages of multi-modal data processing, can automatically and comprehensively analyze different types of log data, reduces the need for manual intervention, and improves the practicality and expandability of the internal threat detection system.

[0051] On the other hand, the feature extraction and anomaly detection module based on the Transformer hierarchical structure can effectively extract the template and numerical features in the log data, and conduct detailed abnormal behavior analysis. By detecting anomalies in single logs and context log sequences at the same time, the detection accuracy of complex threat patterns is improved. And according to the actual demand, the weight of the two is adjusted, so as to more flexibly cope with different types of internal threats, and significantly improve the generalization ability and robustness of the model. BRIEF DESCRIPTION OF DRAWINGS

[0052] In order to more clearly illustrate the technical solutions of the present application, the drawings required to be used in the embodiments will be briefly introduced as follows. Obviously, for those skilled in the art, other drawings can also be obtained from the drawings without any creative labor.

[0053] Figure 1 A flowchart of a hybrid internal threat detection method based on DACGAN-Transformer is provided for the embodiments of the present application.

[0054] Figure 2 A model schematic diagram is provided for the embodiments of the present application.

[0055] Figure 3 A log analysis example schematic diagram is provided for the embodiments of the present application.

[0056] Figure 4 A single log template encoder structure schematic diagram is provided for the embodiments of the present application.

[0057] Figure 5 A single log parameter encoder structure schematic diagram is provided for the embodiments of the present application.

[0058] Figure 6 A template sequence encoder structure schematic diagram is provided for the embodiments of the present application.

[0059] Figure 7 A single log anomaly detection schematic diagram is provided for the embodiments of the present application.

[0060] Figure 8 A context anomaly detection schematic diagram is provided for the embodiments of the present application.

[0061] Figure 9 A comprehensive evaluation process schematic diagram is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application will be described clearly and completely in combination with specific embodiments of the present application and corresponding drawings. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative labor are within the scope of protection of the present application. The technical solutions provided by the embodiments of the present application will be described in detail below in combination with the drawings.

[0063] In order to better understand the purpose of the present application, the present application will be described in further detail below, please refer to Figures 1 to 9 .

[0064] The application discloses a hybrid internal threat detection method based on a DACGAN-Transformer.

[0065] In step S101, log data is acquired, the log data including server logs, application program logs and network device logs, and the log data is preprocessed to obtain preprocessed log data.

[0066] In step S102, the preprocessed log data is substituted into a generative adversarial network, the preprocessed log data being trained by a generator, a discriminator and an independent classifier in the generative adversarial network, the generator generating samples similar to normal data distribution but having abnormal characteristics, and the classifier giving an overall abnormal score.

[0067] In step S103, a preset algorithm or other log analysis method is used on the log data to extract a fixed structure part of each log entry, generate a log template, and extract dynamic information in each log entry as a parameter part.

[0068] In step S104, a single log entry template is mapped into a vector, global and local features are extracted, and a fixed-length template representation vector is outputted, log parameters are mapped into a vector, global and local features are extracted, and a fixed-length parameter representation vector is outputted, and the template and parameter feature vectors of the log sequence are encoded respectively to represent template context information and parameter context information of the log sequence.

[0069] In step S105, linear transformation and normalization are performed on the template representation and the parameter representation of the single log entry, an abnormal score of the single log is outputted, linear transformation and fusion are performed on the template sequence representation and the parameter sequence representation of the log sequence, context dependency is learned by a Transformer Block preset tool model, a context abnormal score is outputted, and the single log abnormal score and the context abnormal score are linearly weighted and integrated to obtain a hierarchical abnormal score of the Transformer.

[0070] In step S106, the GAN overall abnormal score and the hierarchical abnormal score of the Transformer are inputted into a multilayer perceptron, an ultimate abnormal probability is outputted by a preset activation function, and whether a log entry is abnormal is determined according to a preset threshold.

[0071] Specifically, the DACGAN-Transformer-based hybrid internal threat detection method comprises the following steps.

[0072] Server logs, application program logs and network device logs are acquired from a log system.

[0073] The server log, application log and network device log noise data including redundant log records, abnormal characters and missing values are removed, the low-frequency server log, application log and network device log are filtered by setting a threshold, the useless data in the server log, application log and network device log are filtered, the abnormal characters in the server log, application log and network device log are removed by using a character filter, and the missing values in the server log, application log and network device log are completed by using a mean interpolation method.

[0074] The data conversion rules are obtained, the timestamps, IP addresses, user IDs and operation types in the server log, application log and network device log are mapped to standard fields, different formats of timestamps are converted into a unified time format by using a time resolution library, the log records are sorted according to timestamps by using a sorting algorithm, and a log list arranged in chronological order is output.

[0075] The server log, application log and network device log data cleaned, formatted, timestamp-aligned and quality-checked are output as inputs for subsequent generation of a generative adversarial network and log analysis.

[0076] Specifically, the DACGAN-Transformer-based hybrid internal threat detection method comprises the following steps:

[0077] The preprocessed log data is input into the generative adversarial network, and the network parameters of the generator, discriminator and independent classifier are initialized.

[0078] The generator receives random noise and label inputs, generates samples similar to normal data distribution but with abnormal characteristics.

[0079] The discriminator receives the samples generated by the generator and the real samples, judges the authenticity of the samples, and feeds back the results to the generator, and the generator dynamically updates the weights after receiving the feedback results, and optimizes the network by continuously iterating the weights.

[0080] The independent classifier classifies the samples generated by the generator, judges whether the samples generated by the generator correspond to normal or abnormal samples, and obtains the independent classifier output results.

[0081] According to the independent classifier output results, the loss functions of the generator, discriminator and classifier are calculated, and the network parameters are optimized by back propagation, and in the training process, the classification results of the independent classifier for each generated sample, i.e., the overall abnormal score, are recorded.

[0082] Specifically, the DACGAN-Transformer-based hybrid internal threat detection method provided in the present application comprises the following steps:

[0083] The preprocessed log data is input into a log parsing module, each log entry is selected from the input log data one by one for parsing, a log parsing algorithm is used, the log parsing algorithm adopts a Drain algorithm, a fixed structure part of each log entry is extracted, and a log template is generated;

[0084] Dynamic information is extracted from each log entry, the dynamic information includes a file path, a visitor ID and a timestamp, and the file path, the visitor ID and the timestamp serve as parameter parts;

[0085] The extracted template is paired with the corresponding parameters to form template-parameter corresponding information, and the template-parameter corresponding information is integrated to obtain log entry representation.

[0086] Specifically, the DACGAN-Transformer-based hybrid internal threat detection method provided in the present application comprises the following steps:

[0087] The single log entry template is mapped into a vector form and input into a single log template encoder for feature extraction;

[0088] Global and local features of the log template are extracted through a preset tool model in the single log template encoder, the preset tool model adopts a Transformer Block;

[0089] The extracted features are processed through an average pooling layer to generate a fixed-length template representation vector;

[0090] The log parameters are mapped into a vector form and input into a single log parameter encoder for feature extraction, global and local features of the log parameters are extracted through a preset tool model in the single log parameter encoder, the preset tool model adopts a Transformer Block, and the extracted features are processed through an average pooling layer to generate a fixed-length parameter representation vector.

[0091] Specifically, the DACGAN-Transformer-based hybrid internal threat detection method provided in the present application comprises the following steps:

[0092] The template representation and the parameter representation of the single log entry are linearly transformed and normalized to output an abnormal score of the single log;

[0093] The template sequence representation and parameter sequence representation of the log sequence are linearly transformed and fused, context dependency is learned through a preset tool model of a Transformer Block, and then a linear transformation and normalization are performed to output a context anomaly score;

[0094] The single-log anomaly score and the context anomaly score are fused by linear weighting to obtain a hierarchical anomaly score.

[0095] Specifically, the DACGAN-Transformer-based hybrid internal threat detection method comprises the following steps:

[0096] The Transformer hierarchical anomaly score includes a single-log anomaly score and a context anomaly score, and fuses the local and global anomaly degrees at the log entry granularity;

[0097] The overall anomaly score of the GAN and the hierarchical anomaly score of the Transformer architecture are taken as input data;

[0098] The multi-layer perceptron processes the input score, and performs feature extraction and integration through a multi-layer neural network;

[0099] The output of the multi-layer perceptron is processed by a preset activation function, and the output is limited to the range of [0, 1] as the final anomaly probability;

[0100] According to a preset threshold, the final anomaly probability is determined, if the anomaly probability exceeds the threshold, the log entry is determined to be abnormal, otherwise, it is determined to be normal, and the anomaly determination result of the log entry is output.

[0101] The DACGAN-Transformer-based hybrid internal threat detection method can be applied to a DACGAN-Transformer-based hybrid internal threat detection system. Figure 2 As shown in the figure, the DACGAN-Transformer-based hybrid internal threat detection system comprises a data preprocessing module, a GAN data enhancement module, a comprehensive evaluation module, a hierarchical feature extraction module and a hierarchical anomaly detection module.

[0102] The data preprocessing module is responsible for obtaining raw log data from the log system, and cleaning, formatting and timestamp alignment to ensure data availability and consistency.

[0103] Log acquisition: acquire raw log data including server logs, application logs, network device logs, etc., and ensure that the log data contains possible abnormal information.

[0104] Log Cleaning: Cleaning the raw data, including methods for handling useless or redundant log records, abnormal characters, and missing values. Filter low-frequency log records by setting thresholds, identify and filter useless logs. Remove abnormal characters using regular expressions or character filters. Use mean interpolation to complete log entries for missing values.

[0105] Formatting: Convert logs of different formats to standard JSON format through data conversion tools or custom scripts. Map key information fields in logs such as timestamps, IP addresses, user IDs, operation types, etc. to standard fields by writing data conversion rules.

[0106] JSON log format:

[0107] {

[0108] "timestamp":"2024-09-18T14:33:12Z",

[0109] "level":"ERROR",

[0110] "source":"server-1",

[0111] "message":"Failedtoconnecttodatabase,errorcode:1049",

[0112] "user_id":"12345",

[0113] "ip_address":"192.0.2.x"

[0114] }

[0115] Original timestamp: Sep 18 14:33:12

[0116] Processed timestamp: 2024-09-18T14:33:12Z

[0117] The extracted format is as follows, and the log information generated by the subsequent GAN meets this format, which can be parsed by the Drain algorithm.

[0118] 2024-09-18T14:33:12Z ERROR server-1: Failed to connect to database, error code: 1049, UserID: 12345, IP: 192.0.2.x

[0119] Timestamp alignment: Use the time resolution library to convert timestamps of different formats into a unified time format, which requires compliance with the ISO8601 standard. Then use the sorting algorithm to sort the log records by timestamp, and the final output is a list of logs in chronological order.

[0120] The GAN data enhancement module generates more samples similar to the normal data distribution but with abnormal features through a generative adversarial network (GAN) to solve the data imbalance problem in internal threat detection and improve the generalization ability of the detection model; meanwhile, an independent classifier is added to generate abnormal samples and make overall abnormality judgment.

[0121] The optimization of the traditional GAN generator and discriminator is the adversarial loss:

[0122]

[0123] Among them, represents the probability that the discriminator considers x to be a real sample, and is the probability that the discriminator considers the synthetic sample to be false. When training the GAN, the discriminator wants to maximize the objective function, on the contrary, the generator wants to minimize the objective function. During the training process, one party is fixed, and the other party updates the network weight, and the optimization of its own network is alternately iterated until a dynamic balance is reached.

[0124] Generator: adopt a standard generator architecture such as DCGAN, WGAN, the network can contain multiple layers of convolutional networks. Receive random noise and label input, the output layer uses Sigmoid or Tanh activation function, and the generated sample is mapped to a range similar to the normal data distribution.

[0125] In the present application, because a separate classifier is added, the generator loss function includes an adversarial loss and a classification loss, the adversarial loss indicates that the generator wants to maximize the probability that the discriminator judges the generated sample to be real. Therefore, the loss function output by the discriminator is used as the adversarial loss of the generator:

[0126]

[0127] Where G(z) is the sample generated by the generator, and D(G(z)) is the output probability of the discriminator for the generated sample.

[0128] The generator also generates samples with abnormal features, so the classification loss is considered when generating data. The classification loss measures whether the generator generates abnormal samples through the classifier's prediction of the generated samples. Cross-entropy loss can be used:

[0129]

[0130] where C(G(z)) is the predicted probability of the generated sample by the classifier, is the target label of the generated sample (normal or abnormal).

[0131] The generator total loss combines the adversarial loss and the classification loss:

[0132]

[0133] Discriminator: Similar structure to the generator, can use convolutional layers or fully connected layers. Used to judge whether the input data is real data, and feedback the result to the generator for optimization, to ensure that the generated data approaches the real distribution.

[0134] The discriminator loss function only has an adversarial loss, its goal is to distinguish between real samples and generated samples, and update the weights of the generator through backpropagation. The traditional discriminator loss uses binary cross-entropy loss to distinguish between real data and generated data:

[0135]

[0136] where D(x) is the output of the discriminator on real samples, and D(G(z)) is the output of the discriminator on generated samples.

[0137] The independence of the classifier allows it to optimize the classification task of normal and abnormal samples without being affected by the discriminator.

[0138] Classifier: Designed independently of the discriminator, responsible for classifying generated data, can use traditional classification models such as support vector machines SVM, decision trees, or deep learning models such as building a fully connected neural network classifier to perform binary classification on generated samples, i.e. normal or abnormal samples. The independence of this classifier allows more flexible adjustment of weights to improve classification performance. Its output can be used as part of the model decision-making process to enhance the accuracy of detection.

[0139] The classifier loss can be calculated by cross-entropy loss to calculate the classification error of normal and abnormal samples.

[0140]

[0141] where C(x) is the output of the classifier on real samples, is the label of the real sample, 0 represents normal, and 1 represents abnormal.

[0142] Finally, the losses of the generator, discriminator and classifier are combined to form the loss function of the entire GAN model:

[0143]

[0144] Log parsing module;

[0145] Task: As shown in Figure 3 , the original log data is parsed into template parts and parameter parts, facilitating subsequent modules for different levels of feature extraction and detection.

[0146] This module uses the Drain algorithm, which is a hierarchical tree structure mining method for log parsing, capable of automatically converting structured and semi-structured logs into template and parameter pairs. The original log is a standard structured JSON format after processing, and its "message" field can be extracted for Drain parsing, while other structured fields, including timestamps, log levels, log sources, and log messages, are retained.

[0147] Template extraction: Extract the fixed structure part of each log entry to generate a log template representing the operation behavior or event type. For example, extract and standardize the template from the log, such as "[timestamp][log level][log source][log message]".

[0148] Parameter extraction: Extract dynamic information in each log entry, such as specific file paths, visitor IDs, timestamps, etc. Parameter information reflects the detailed content of the log, containing more personalized and dynamic behavior patterns. For example, visitor ID 67890, IP address 192.0.2.x, etc.

[0149] Transformer hierarchical feature extraction module;

[0150] Task: This module performs multi-level feature extraction on log data through single log encoders and sequence encoders. Single log encoders are used to process the template and parameter parts of individual logs, while sequence encoders are used to process templates and parameters in log sequences to capture fine-grained information, time dependencies, and context dependencies in logs.

[0151] Single log template encoder: used to encode the template part in a single log, such as Figure 4 As shown in the figure, first map each word in the log template to a vector x, where d is the dimension of the word vector, and assuming the length of the log template is n, then the vector list can be obtained.

[0152] The preprocessed template vector is input into two Transformer Blocks, each of which includes a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and a residual connection, to capture global and local features in the log template. The output of each Transformer Block is processed by layer normalization and updated by a weighted residual connection. After passing through two Transformer Blocks, the output is dimensionally reduced by an average pooling layer to generate a fixed-length template representation vector . .

[0153] Single-log parameter encoder: As shown in Figure 5 , the structure is similar to that of the single-log template encoder and is used to process the parameter part of a single log. Each feature in the log parameter is first mapped to a vector y, where y ∈ R1×d, d is the dimension of the word vector, and assuming the length of the log parameter is m, a list of vectors can be obtained. The details and dependencies in the parameters are extracted by two Transformer Block blocks, and the output of each Block is optimized by layer normalization and a residual connection, outputting . Finally, the parameter feature vector is reduced to a parameter representation with the same length as the template representation using the same pooling strategy as the single-log template encoder , .

[0154] Template sequence encoder: The template feature vector sequence of all logs is received for further encoding to obtain a template sequence representation with context information. The parameter sequence encoder is the same, and the parameter sequence is further encoded to obtain a parameter sequence representation. Both the template sequence encoder and the parameter sequence encoder are composed of four Transformer Blocks, each of which has a multi-head self-attention mechanism, a feedforward neural network, layer normalization, and a residual connection. Finally, a fixed-length vector is output through a pooling layer. To avoid repetition, only the structure and process of the template sequence encoder are shown, as shown in Figure 6 .

[0155] Assuming that a log sequence contains N logs, the template features of the N logs are extracted by the single-log encoder to obtain , where . Then, the global context features are extracted by a Transformer to capture deeper dependencies, and the template sequence representation is obtained. The parameter sequence encoding is the same, and the final output is represented as .

[0156] The Transformer hierarchical anomaly detection module is used for anomaly detection on single log entries and log sequences. The module first performs simple anomaly detection on single log entries, identifying log format, field values, and other anomalies. Subsequently, it performs contextual anomaly detection on log sequences containing multiple logs to capture more complex attack patterns and behaviors. Through multi-level detection methods, the accuracy of anomaly detection can be effectively improved, especially for potential complex threats.

[0157] Single log anomaly detection: As shown in Figure 7 , the input is the template representation and parameter representation of a single log entry obtained from the Transformer hierarchical feature extraction module. These representation vectors, after feature extraction, contain global and local information in the log entry. The template representation and parameter representation are respectively passed through a linear transformation layer, which projects high-dimensional vectors into a lower-dimensional anomaly detection space to more easily identify abnormal relationships between features.

[0158]

[0159] The template and parameter features after linear transformation are mapped to the [0, 1] interval by sigmoid, representing the anomaly score.

[0160] Contextual anomaly detection: As shown in Figure 8 , the input is the template sequence representation and parameter sequence representation of a group of logs, which are obtained from the Transformer hierarchical feature extraction module and contain global context-dependent information extracted by the encoder. Linear transformation is performed on the template sequence representation and parameter sequence representation to fuse log sequence features, compressing high-dimensional context features into a lower-dimensional space that is easier to detect. Through this processing, potential abnormal patterns in the sequence can be identified.

[0161]

[0162] The relevance of events in the log sequence is then learned through 6 Transformer Blocks, and finally the context anomaly score of each log is output after linear transformation and feature fusion through softmax. , represents the anomaly score of the i-th log under the log context condition, and the log sequence contains N logs.

[0163] After obtaining the single log anomaly score and the context anomaly score, the hierarchical anomaly score is obtained by linearly weighting the results of the two levels.

[0164]

[0165] wherein represents the abnormal score of the hierarchical anomaly, represents the abnormal score of the single log, represents the abnormal score of the context (sequence). represents the weight of the single log score, represents the weight of the context score.

[0166] The weight can be set according to the business needs of the system, the relatively stable threat mode, or the experience of a domain expert. If more attention is paid to single log anomalies, can be set = 0.3, and vice versa .

[0167] In actual scenarios, for example, in a company, an internal attacker may not directly expose anomalies, but may steal information in a more complex way. For example, the single abnormal score of a user copying a file is 0.4, but the file is copied on a certain rest day, and the abnormal score of the context is 0.9, and the weight is set to , , then .

[0168] The comprehensive evaluation module: as shown in Figure 9 , the overall abnormal score of the GAN and the hierarchical abnormal score of the Transformer are evaluated by the MLP, and the final judgment of whether the log entry is abnormal is given.

[0169] The input includes two parts:

[0170] GAN classifier overall abnormal score : the output from the GAN module, used to represent the abnormality of the log entry relative to the GAN generation model, and to preliminarily detect the log.

[0171] Transformer hierarchical abnormal score : the weighted result of the single log abnormal score and the context abnormal score , representing the abnormality of the log entry from the local and global aspects.

[0172] The output layer uses a Sigmoid activation function to limit the output to the range [0, 1] as the final abnormal probability. According to the set threshold, it is determined whether it is abnormal.

[0173] The technical scheme of the present application solves the defects of the prior art in the following ways:

[0174] The application adopts a generative adversarial network (GAN) to generate samples similar to normal data distribution but with abnormal characteristics, thereby enhancing the data set, effectively alleviating the problem of scarcity of abnormal samples, and further improving the training effect of the model. The Transformer model is used for hierarchical feature extraction of log data, which includes single log anomaly detection and context anomaly detection, significantly improving the granularity and accuracy of detection. The log parsing algorithm (such as Drain algorithm) is used to extract the fixed structure part (i.e. template) and dynamic information (such as parameters) of the log entry, reducing the processing difficulty and reducing the data dimension and redundancy.

[0175] The application designs an independent classifier in the GAN, which not only enhances the classification performance of the GAN, but also optimizes the quality of the abnormal data of the generator, thereby reducing the false positives and false negatives. The Transformer Block is used to extract the global and local features of the log template and parameters, further improving the feature representation capability. At the same time, the single log entry and the log sequence are detected for abnormality, the single log anomaly score and the context anomaly score are fused, and the detection accuracy is significantly improved. The overall anomaly score of the GAN and the hierarchical anomaly score of the Transformer are input into the multi-layer perceptron (MLP), and the final anomaly probability is output through the preset activation function, and the log entry is determined to be abnormal according to the preset threshold.

[0176] The application comprehensively evaluates the output of the GAN and the Transformer through the multi-layer perceptron, significantly improves the accuracy of the internal threat detection, and enhances the security of the system.

[0177] In summary, the problems of data imbalance, insufficient fine-grained analysis, weak model generalization ability and other problems in the existing internal threat detection technology are effectively solved, and the accuracy of the internal threat detection and the security of the entire system are greatly improved.

Claims

1. A hybrid internal threat detection method based on DACGAN-Transformer, characterized in that, include: Acquire server logs, application logs, and network device logs, and perform noise reduction, formatting, and timestamp alignment and sorting. The processed logs are input into a generative adversarial network to generate an enhanced set of anomaly samples and output an overall anomaly score. The Drain algorithm is used to parse logs and separate log templates and parameters. The template representation vector and parameter representation vector of a single log are extracted by a single log template encoder and a single log parameter encoder, respectively. The template context representation and parameter context representation of the log sequence are extracted by a template sequence encoder and a parameter sequence encoder, respectively. All encoders adopt the Transformer Block structure and output fixed-length vectors through an average pooling layer. Linear transformation and normalization are performed on the template and parameter representations of a single log to output a single log anomaly score; linear transformation and fusion are performed on the template and parameter sequence representations of the log sequence, and context dependencies are learned through Transformer Block to output a context anomaly score; the single log anomaly score and the context anomaly score are fused by linear weighting to obtain a hierarchical anomaly score; The overall anomaly score and hierarchical anomaly score of the generative adversarial network are input into the multilayer perceptron for comprehensive evaluation. The overall anomaly score corresponds to the unparsed raw log granularity, while the hierarchical anomaly score corresponds to the parsed single log granularity and context log granularity. The anomaly probability is output through the MLP, and each log is determined to be abnormal based on the threshold.

2. The hybrid internal threat detection method based on DACGAN-Transformer as described in claim 1, characterized in that, Acquire server logs, application logs, and network device logs, and perform noise reduction, formatting, and timestamp alignment and sorting. The processed logs are input into a generative adversarial network (GAN) to generate an enhanced set of anomalous samples and output an overall anomaly score, including: Retrieve server logs, application logs, and network device logs from the logging system; Remove noisy log data, including redundant log records, abnormal characters, and missing values. Filter low-frequency useless logs by setting a threshold, remove abnormal characters using a character filter, and use mean imputation to complete log entries for missing values. Obtain data transformation rules, map timestamps, IP addresses, user IDs, and operation types in logs to standard fields, use a time parsing library to convert timestamps of different formats into a unified time format, and use a sorting algorithm to sort log records by timestamp, outputting a log list arranged in chronological order; The cleaned, formatted, timestamped, and sorted log data is fed into a Generative Adversarial Network (GAN). A GAN consists of a generator, a discriminator, and a classifier that is independent of the discriminator. The generator generates samples that are similar to the normal log distribution but have anomalous features. The discriminator distinguishes between real samples and generated samples. The classifier classifies the generated samples as normal or anomalous, thereby guiding the generator to generate anomalous samples and outputting an overall anomalous score.

Images