An end-flow fusion intrusion detection method and system and storage medium

Abstract

The application discloses an end flow fusion intrusion detection method and system and a storage medium. The method comprises collecting system logs and traffic data in a terminal, and unifying data formats and lengths. An intrusion detection model based on a Transformer architecture is designed, and the model is trained based on prior learning. The trained intrusion detection model is deployed to an actual environment for intrusion detection. A small amount of data is sampled and labeled to form a training data set and a test sample, which is input into the trained model. During the detection process, features are aggregated once every period of time to generate a multi-dimensional numerical feature table. The model directly outputs the prediction probability distribution of the test sample through single forward propagation. The classification result is obtained according to the prediction probability distribution, and whether it is an attack is judged in combination with a threshold value. The application only needs to train one large model at a time, does not need to train different models for different data sets, avoids a lot of repeated and redundant work, and has good migration.

Description

Technical Field

[0001] This invention relates to the field of network security technology, and specifically to an intrusion detection method, system, and storage medium that integrates endpoint and stream processing. Background Technology

[0002] Traditional intrusion detection systems (IDS) have long relied on two main paradigms: rule-based detection and anomaly-based detection, both of which have significant limitations. Rule-based IDS systems depend on a predefined attack signature database for pattern matching. The biggest weakness of this approach is its inability to identify zero-day attacks or variants of known attacks, meaning defenses always lag behind threats. Security analysts must continuously analyze new threats and manually update the rule database, a process that is not only time-consuming and labor-intensive but also prone to errors. Furthermore, the large rule database incurs high performance overhead and management complexity, severely limiting the system's scalability.

[0003] To overcome the limitations of rule-based detection, anomaly-based detection methods attempt to establish a baseline of normal behavior using machine learning models to identify deviations. However, this approach introduces new challenges. Supervised learning models require large amounts of high-quality, labeled attack and normal sample data for training, but obtaining such security datasets is extremely difficult and costly in reality. Model performance is highly sensitive to hyperparameter selection, requiring significant expert knowledge and computational resources for tuning, making deployment prohibitively expensive. More problematic is the high false positive rate these models often suffer from, as the boundary distinguishing new legitimate behavior from malicious activity is inherently blurred. A large number of false alarms can lead to "alarm fatigue," causing genuine threats to be overwhelmed and difficult to detect. Finally, the computational overhead of many complex models makes them ill-equipped to handle high-speed network traffic, hindering real-time detection and response, thus missing the crucial window for attack interception.

[0004] Berkeley Packet Filter (BPF) was originally designed for the efficient capture and filtering of network packets. Its extension, eBPF (extended Berkeley Packet Filter), brought a revolutionary breakthrough to data acquisition. It allows users to safely and efficiently execute user-defined code in kernel mode without modifying kernel source code or loading kernel modules. This technology fundamentally changes the way system observability is implemented, enabling the capture of fine-grained thread-level data with extremely low overhead, including information on all system calls, process execution flow, and kernel function calls. eBPF effectively solves the problem of "how to acquire data," shifting the bottleneck of monitoring from data acquisition to data analysis.

[0005] However, while eBPF delivers massive, high-dimensional, dynamic, real-time data streams, it also presents a new core challenge: how to quickly and accurately analyze and classify this data. The data streams collected by eBPF are highly real-time and exhibit significant temporal characteristics, making it difficult for traditional simple rule engines to handle the complex relationships within them. While traditional machine learning models possess stronger expressive power, they still rely on tedious and specialized manual feature engineering, and the training, tuning, and update cycles of these models cannot keep pace with the rapid evolution of attack methods. Therefore, although eBPF provides the ability to understand fine-grained system behavior, it still lacks a "brain" capable of comprehending this behavior and making intelligent judgments in real time. The current technological landscape thus presents a clear gap: on one hand, the powerful data acquisition capabilities provided by eBPF and detection devices; on the other hand, the lag and inadequacy of analytical methods. How to bridge this gap and build an intelligent analysis layer capable of understanding eBPF data streams in real time becomes crucial to the success of the next generation of intrusion detection systems.

[0006] Besides system behavior information, cyberspace also contains massive amounts of network traffic data. Due to the significant differences in format and the different meanings of information contained in network traffic and system logs, combining network traffic and system logs for intrusion detection presents certain challenges. The traditional machine learning approach of "training a new model for each new dataset" requires training different models separately for different types and formats of data, resulting in a large amount of repetitive and redundant work, generating unnecessary computational and human resource costs. Summary of the Invention

[0007] To address the aforementioned problems, this invention provides an intrusion detection method, system, and storage medium that integrates terminal and traffic data for real-time intrusion detection. This transforms the extensive work on terminals and traffic into a single pre-training of a large model, enabling it to learn and adapt to different tasks.

[0008] According to a first aspect of the present disclosure, an intrusion detection method with end-to-end fusion is provided, the method comprising the following steps:

[0009] Collect system logs and traffic data on the terminal, and standardize the data format and length;

[0010] Design an intrusion detection model based on the Transformer architecture and train the model based on prior learning;

[0011] The trained intrusion detection model is deployed to a real environment for intrusion detection. A small amount of data is sampled and labeled to form a training dataset and test samples, which are then input into the trained model. During the detection process, features are aggregated at regular intervals to generate a multi-dimensional numerical feature table. The model directly outputs the predicted probability distribution of the test samples through a single forward propagation.

[0012] The classification results are obtained based on the predicted probability distribution, and a threshold is used to determine whether it is an attack.

[0013] In some embodiments, eBPF is used to collect system logs and traffic data at the endpoint.

[0014] In some embodiments, unifying the data format and length specifically includes:

[0015] Based on the intrusion detection model, a maximum number of features is preset. For the number of features The data is then populated and scaled.

[0016] Zero fill: for those with Data with features, create a data set with a length of . The zero vector, then the original The eigenvalues ​​are placed into the zero vector before One position, the rest Each position remains 0;

[0017] Scaling: Introducing a scaling factor All shortcomings Dimensional data is obtained by multiplying each feature by a scaling factor. This amplifies the eigenvalues.

[0018] In some embodiments, training the intrusion detection model includes the following:

[0019] Training a model , making the model It is possible to estimate the true distribution of the data from the sampled data: ,in For training data, For test data features, For test data Corresponding tags;

[0020] Based on the prior learning method, the collected dataset is separated according to different labels, that is, different attack types are regarded as the prior of the attack data distribution. In each training step, a dataset is sampled from the prior. , Includes training set and test set ,Will and Features As input to the model, the model's goal is to predict... The corresponding real tags .

[0021] In some embodiments, the intrusion detection model loss function employs the cross-entropy loss function.

[0022] In some embodiments, attention masks are used during training so that training samples can pay attention to each other, while test samples can only pay attention to training samples and cannot pay attention to other test samples.

[0023] In some embodiments, the classification result is obtained by using the argmax function based on the predicted probability distribution.

[0024] According to a second aspect of the present disclosure, an intrusion detection system with end-to-end fusion is provided, the system comprising:

[0025] The terminal and traffic data collection and fusion module is used to collect system logs and traffic data on the terminal and to unify the data format and length.

[0026] The model design and training module is used to design an intrusion detection model based on the Transformer architecture and train the model based on prior learning.

[0027] The intrusion detection module is used to deploy the trained intrusion detection model to the actual environment for intrusion detection. It samples a small amount of data for annotation to form a training dataset and test samples, which are then input into the trained model. During the detection process, features are aggregated at regular intervals to generate a multi-dimensional numerical feature table. The model directly outputs the predicted probability distribution of the test samples through a single forward propagation. The classification result is obtained based on the predicted probability distribution, and a threshold is used to determine whether it is an attack.

[0028] According to a third aspect of the present disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described intrusion detection method steps of end-to-end fusion.

[0029] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium is provided, the storage medium storing computer instructions that, when executed by a processor, implement the above-described intrusion detection method steps of end-to-end fusion.

[0030] This disclosure provides an end-to-end flow fusion intrusion detection method, system, and storage medium, addressing the problem of existing technologies: the failure to simultaneously utilize terminal and traffic data. To improve the accuracy and precision of intrusion detection, existing machine learning or deep learning methods require training different detection models for terminal logs and network traffic separately. These methods often require large amounts of labeled data, resulting in significant duplication and redundancy. Similar to rule-based intrusion detection, these supervised learning methods rely on prior knowledge of the data, i.e., known attack characteristics or expert knowledge, to label the data, failing to detect unknown (unlabeled) or novel attack methods. This is because the model only remembers the data characteristics and lacks the ability to learn, leading to its inability to adapt well to the diverse user behaviors and constantly evolving intrusion methods in the real-world network environment. eBPF can simultaneously collect network traffic and terminal log data. Utilizing existing public datasets and simulated attack tools to generate corresponding attack and normal behavior data in a controlled environment can further expand the data volume. Therefore, this invention, based on eBPF's ability to simultaneously collect terminal and traffic data and combining it with other data sources to expand the data volume, designs an end-to-end flow fusion intrusion detection method. Specifically, compared with existing technologies, it has the following beneficial effects:

[0031] 1) This invention utilizes network traffic and terminal log data simultaneously. Compared with existing intrusion detection systems, this invention unifies the traffic and log formats, enabling comprehensive intrusion detection from both network and terminal levels.

[0032] 2) This invention only requires training one large model at a time, eliminating the need to train different models for different datasets, thus avoiding a lot of repetitive and redundant work;

[0033] 3) This invention designs a scaling factor to scale data with insufficient dimensions, avoiding the systematic bias in the weighted sum (or average value) of effective features in existing intrusion detection methods caused by the different number of data features.

[0034] 4) This invention uses neural networks to approximate Bayesian inference, which can learn to learn. In actual intrusion detection environments, only a small number of new labeled data samples are needed to adapt to new attack types. Compared with existing intrusion detection systems, this invention has good transferability.

[0035] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0036] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0037] Figure 1 This is a schematic diagram of the intrusion detection method using end-to-end flow fusion in an embodiment of the present invention;

[0038] Figure 2 This is a schematic diagram of the intrusion detection method of end-to-end flow fusion in an embodiment of the present invention;

[0039] Figure 3 This is a schematic diagram of the attention mask in the method of this embodiment of the invention;

[0040] Figure 4 This is a structural diagram of the intrusion detection system with end-to-end fusion in the method of this embodiment of the invention;

[0041] Figure 5 This is a schematic diagram of an electronic device according to an embodiment of the present invention. Detailed Implementation

[0042] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the drawings, not the entire structure.

[0043] Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the steps as sequential processes, many of these steps can be performed in parallel, concurrently, or simultaneously. Furthermore, the order of the steps can be rearranged. The process can be terminated when its operation is complete, but may also have additional steps not included in the figures. The process can correspond to a method, function, procedure, subroutine, subroutine, etc.

[0044] This invention proposes an endpoint-to-flow fusion intrusion detection method, transforming the traditional machine learning approach of "training a model for each dataset" into a method of "pre-training a large model all at once, enabling it to learn how to learn and detect." This combines endpoint and traffic data for attack detection, reducing the significant amount of repetitive and redundant work involved in data labeling and model training during intrusion detection. This invention leverages eBPF's capabilities for capturing traffic data and observing endpoint behavior, utilizing both endpoint and traffic data simultaneously. First, it collects existing publicly available intrusion detection datasets, including endpoint and traffic data. Simultaneously, it generates a large amount of normal and intrusion behavior data in a secure and controlled environment using tools such as CyberFlood and Burp Suite. The data is transformed into fixed-length numerical data by extracting numerical features, statistical features, and hash mapping, and then a large model is pre-trained using this data. After training, when encountering a new dataset or scenario, if the data is unlabeled, only a small amount of data needs to be labeled, constructed into an input sequence as during pre-training, and fed into the model. The model directly outputs the probability distribution for the new data through a single forward propagation, without the need for parameter retuning or retraining.

[0045] This invention proposes an end-to-end fusion intrusion detection method that combines terminal and traffic data for simultaneous attack detection, avoiding the need to train different models for different types of data. When collecting data, it is necessary to address how to fuse terminal and traffic data; after collecting a large amount of data, it is necessary to consider how to train a large, "learning" model that ensures both performance and good transferability; in the actual intrusion detection process, it should be able to understand the current task based on a small amount of labeled data and infer the probability distribution of the current task data from this limited data.

[0046] The following embodiments are provided for an intrusion detection method, system, and storage medium that integrates end-to-end and flow fusion:

[0047] like Figure 1 As shown, an intrusion detection method using end-to-end fusion includes the following steps:

[0048] S1. Collect system logs and traffic data on the terminal and standardize the data format and length; specifically, use eBPF to collect system logs and traffic data on the terminal.

[0049] S2. Design an intrusion detection model based on the Transformer architecture and train the model based on prior learning;

[0050] S3. Deploy the trained intrusion detection model to a real-world environment for intrusion detection. Sample a small amount of data for annotation to form a training dataset and test samples. Input these samples into the trained model. During the detection process, aggregate features periodically to generate a multi-dimensional numerical feature table. The model directly outputs the predicted probability distribution of the test samples through a single forward propagation. The classification result is obtained based on the predicted probability distribution, and a threshold is used to determine whether it constitutes an attack. Specifically, the classification result is obtained using the argmax function based on the predicted probability distribution.

[0051] Specifically, such as Figure 2 As shown, in the specific implementation of S1, existing public datasets such as CICIDS2017, UGR-16, NSL-KDD, and CREMEv2 Datasets are collected. Simultaneously, tools such as CyberFlood and Burp Suite are used to generate a large amount of normal and attack traffic in a secure and controllable environment (such as a network range). eBPF is then used to collect system logs and traffic data passing through the network interface card at the terminal. The collected terminal and traffic data need to be converted into fixed-length (e.g., 100) integers. For network traffic data, record information such as the destination port of the network flow, total number of packets, maximum packet size, minimum packet size, maximum time interval between two packets, average packet size, packet size variance, and packet header size. For terminal log data, record system behavior information such as timestamp, user ID, thread ID, process ID, return value, command parameters, and system call functions, as well as system audit information such as MEM (percentage of memory used by the process in the past 10 seconds), RSIZE (size of memory used by the process in the past 10 seconds), VSIZE (size of virtual space used by the process in the past 10 seconds), VGROW (size of virtual space increased by the process in the past 10 seconds), RGROW (size of memory increased by the process in the past 10 seconds), DSK (percentage of disk space used by the process in the past 10 seconds), RDDSK (amount of data read from disk by the process in the past 10 seconds), and WRDSK (amount of data written to disk by the process in the past 10 seconds).

[0052] Considering the need for a unified format and fixed length for subsequent model inputs, further fusion of the collected terminal and traffic data is required. Network traffic data and system audit data are already numerical data and require no further processing. For text data in system logs, such as system call functions, since the system call functions are fixed, a dictionary can be maintained to record system calls, using function indices instead of function names. For irregular text data such as command parameters, hash functions like FNV1_a can be used to map them to a fixed length (e.g., an 8-digit integer). The model presets a maximum number of features. For the characteristic number, it is 100. The data is further populated and scaled to... Dimensions, specifically including:

[0053] Based on the intrusion detection model, a maximum number of features is preset. For the number of features The data is then populated and scaled.

[0054] Zero fill: for those with Data with features, create a data set with a length of . The zero vector, then the original The eigenvalues ​​are placed into the zero vector before One position, the rest Each position remains 0; Scaling: Introduces a scaling factor. All shortcomings Dimensional data is obtained by multiplying each feature by a scaling factor. This amplifies the feature values ​​to avoid systematic bias in the weighted sum (or average) of effective features due to differences in the number of data features.

[0055] like Figure 2 As shown, the training of the intrusion detection model in S2 includes the following:

[0056] Training a model , making the model It is possible to estimate the true distribution of the data from the sampled data: ,in For training data, For test data, For test data The corresponding tags, i.e., real data tags, This represents the true probability distribution of the data.

[0057] Based on the prior learning method, the collected dataset is separated according to different labels, that is, different attack types are regarded as the prior of the attack data distribution. In each training step, a dataset is sampled from the prior. , Includes training set and test set ,Will and Features As input to the model, the model's goal is to predict... The corresponding real tags .

[0058] Specifically, the model employs a Transformer architecture, utilizing neural networks to approximate Bayesian inference in the intrusion detection process. Traditional Bayesian inference aims to compute the posterior predictive distribution (PPD). For a new test sample... Given training data Under these conditions, its label distribution is given by the following formula: (1)

[0059] in It represents the hypothesis space, that is, all possible scenarios; in the field of intrusion detection, it represents all possible intrusion methods and all possible logs, traffic, system auditing and other information.

[0060] Due to the different attack methods and attacking organizations / individuals involved in actual intrusion detection, the hypothetical space... Complex and vast, formula (1) assumes all possible assumptions Solving this problem is virtually impossible. Therefore, this invention aims to directly train a single model to learn the mapping of this integral, i.e., to train a single model. This makes the model It is possible to estimate the true distribution of the data from the sampled data: (2)

[0061] The model training process employs a prior-based learning approach, separating the collected dataset according to different attack types (labels), treating different attack types as the "prior" of the attack data distribution. In each training step, a dataset is sampled from the prior. , Includes training and test sets:

[0062] (3)

[0063] in , This represents the probability of data D appearing in the hypothesis space, that is, the probability of the attack method reflected by data D appearing in the real environment.

[0064] Will and Features Together, they serve as input to the model, and the model's goal is to predict... The corresponding real tags The model loss function uses the cross-entropy loss function:

[0065] (4)

[0066] The entire model is trained offline in one go and then used for subsequent intrusion detection. For example... Figure 3 As shown, the training process uses an attention mask, which allows training samples to directly attend to each other (bidirectional attention), while test samples can only attend to training samples and cannot attend to other test samples.

[0067] When deploying S3 in a real-world intrusion detection environment, it is necessary to sample a small amount of data for annotation to form a small training dataset. and test samples Similar to pre-training, the input sequence is fed into the trained model. In practice, the detection process aggregates features every 10 seconds, generating a 100-dimensional numerical feature table. The model outputs the result directly through a single forward propagation. The predicted probability distribution is used. Based on the predicted probability distribution, the classification result is obtained through the argmax function, and combined with the threshold, it is determined whether it is a certain type of attack and whether an alarm should be triggered.

[0068] The above embodiments address the issue of fragmented traffic and terminal logs in existing intrusion detection systems by designing a method that integrates terminal logs and network traffic data, utilizing both for intrusion detection. To address the need for different models to be designed for different data in existing intrusion detection methods, a large model requiring only one pre-training is designed, transforming the traditional machine learning approach of "training a new model for each new dataset" into "pre-training a large model all at once, allowing it to learn how to learn." To address the potential for systematic bias in existing intrusion detection systems due to varying numbers of data features, a scaling factor is designed to scale data with insufficient dimensionality. To rigorously simulate the mathematical principles of Bayesian inference and prevent information leakage, ensuring the model's generalization ability, test samples are restricted to focusing only on training samples and not on other test samples. Specifically, traffic and system log data are combined for unified representation, including but not limited to conversion to numerical data and hash mapping; neural networks are used to approximate the Bayesian inference process, and the probability distribution of new data / new scenarios is predicted by learning the prior probability distribution of different attack types, without the need to train different models for different datasets; scaling factors are used to scale data of different dimensions to improve model stability; attention masks are used for restriction to prevent information leakage and ensure the generalization ability of the model.

[0069] Another embodiment illustrates an intrusion detection system with end-to-end flow fusion, such as Figure 4 As shown, the system 400 includes:

[0070] Terminal and traffic data collection and fusion module 410 is used to collect system logs and traffic data on the terminal and unify the data format and length;

[0071] The model design and training module 420 is used to design an intrusion detection model based on the Transformer architecture and train the model based on prior learning.

[0072] The intrusion detection module 430 is used to deploy the trained intrusion detection model to the actual environment for intrusion detection. It samples a small amount of data for annotation to form a training dataset and test samples, which are then input into the trained model. During the detection process, features are aggregated at regular intervals to generate a multi-dimensional numerical feature table. The model directly outputs the predicted probability distribution of the test samples through a single forward propagation. The classification result is obtained based on the predicted probability distribution, and a threshold is used to determine whether it is an attack.

[0073] In addition to the modules described above, the system 400 may also include other components; however, since these components are not relevant to the embodiments of this disclosure, their illustrations and descriptions are omitted here.

[0074] Other specific working processes of the end-to-end fusion intrusion detection system 400 are described in the above embodiment of the end-to-end fusion intrusion detection method, and will not be repeated here.

[0075] Another embodiment illustrating that the system of the present invention can also be achieved by means of... Figure 5 The architecture of the computing device shown is used to implement this. Figure 5 The architecture of the computing device is shown. For example... Figure 5 As shown, the computer system 510 includes a system bus 530, one or more CPUs 540, input / output 520, and memory 550. Memory 550 can store various data or files used for computer processing and / or communication, as well as program instructions executed by the CPU, including the intrusion detection method of the embodiment's end-to-end fusion. Figure 5 The architecture shown is merely exemplary and should be adjusted according to actual needs when implementing different devices. Figure 5 One or more components in the system. The memory 550, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the endpoint-stream fusion intrusion detection method in this embodiment of the invention (e.g., the terminal and traffic data collection and fusion module 410, model design and training module 420, and intrusion detection module 430 in the endpoint-stream fusion intrusion detection system 400). One or more CPUs 540 execute various functional applications and data processing of the system of the present invention by running the software programs, instructions, and modules stored in the memory 550, thereby implementing the above-described endpoint-stream fusion intrusion detection method, which includes the following steps:

[0076] Collect system logs and traffic data on the terminal, and standardize the data format and length;

[0077] Design an intrusion detection model based on the Transformer architecture and train the model based on prior learning;

[0078] The trained intrusion detection model is deployed to a real environment for intrusion detection. A small amount of data is sampled and labeled to form a training dataset and test samples, which are then input into the trained model. During the detection process, features are aggregated at regular intervals to generate a multi-dimensional numerical feature table. The model directly outputs the predicted probability distribution of the test samples through a single forward propagation.

[0079] The classification results are obtained based on the predicted probability distribution, and a threshold is used to determine whether it is an attack.

[0080] Of course, the processor of the server provided in the embodiments of the present invention is not limited to performing the method operations described above, but can also perform related operations in the intrusion detection method of end-to-end fusion provided in any embodiment of the present invention.

[0081] The memory 550 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on terminal usage. Furthermore, the memory 550 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory, or other non-volatile solid-state storage device. In some instances, the memory 550 may further include memory remotely configured relative to one or more CPUs 540, which can be connected to the device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0082] Input / output 520 can be used to receive input digital or character information, and to generate key signal inputs related to user settings and function control of the device. Input / output 520 may also include a display device such as a display screen.

[0083] This invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, this computer program implements the intrusion detection method of end-to-end fusion described in the above embodiments. The computer-readable storage medium of this invention can be any combination of one or more computer-readable media. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0084] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in connection with an instruction execution system, apparatus, or device.

[0085] The program code contained on the storage medium can be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0086] Furthermore, other specific operational processes of a non-transitory computer-readable storage medium are described in the above-described embodiment of the intrusion detection method with end-to-end fusion, and will not be repeated here.

[0087] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a step or method that comprises a list of elements includes not only those elements but also other elements not expressly listed or inherent to such a step or method.

[0088] The above description, in conjunction with specific preferred embodiments, provides a further detailed explanation of the present invention. It should not be construed that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art, various simple deductions or substitutions can be made without departing from the concept of the present invention, and all such modifications and substitutions should be considered within the scope of protection of the present invention.

Claims

1. An intrusion detection method using end-to-end flow fusion, characterized in that, The method includes the following steps: Collect system logs and traffic data on the terminal, and standardize the data format and length; Design an intrusion detection model based on the Transformer architecture and train the model based on prior learning; The trained intrusion detection model is deployed to a real environment for intrusion detection. A small amount of data is sampled and labeled to form a training dataset and test samples, which are then input into the trained model. During the detection process, features are aggregated at regular intervals to generate a multi-dimensional numerical feature table. The model directly outputs the predicted probability distribution of the test samples through a single forward propagation. The classification results are obtained based on the predicted probability distribution, and a threshold is used to determine whether it is an attack. Training the intrusion detection model includes the following: Training a model , making the model It is possible to estimate the true distribution of the data from the sampled data: ,in For the training dataset, For test data features, For test data features The corresponding tags, i.e., real data tags, This represents the true probability distribution of the data; Based on the prior learning method, the collected dataset is separated according to different labels, that is, different attack types are regarded as the prior of the attack data distribution. In each training step, a dataset is sampled from the prior. , Includes training dataset and test samples ,Will and Test data characteristics As input to the model, the model's goal is to predict... The corresponding real data labels ; The intrusion detection model uses the cross-entropy loss function as its loss function. During training, attention masks are used to allow the training datasets to pay attention to each other, while the test samples can only pay attention to the training dataset and cannot pay attention to other test samples.

2. The intrusion detection method based on end-to-end flow fusion according to claim 1, characterized in that, Use eBPF to collect system logs and traffic data on the terminal.

3. The intrusion detection method based on end-to-end flow fusion according to claim 1, characterized in that, Standardize data format and length, specifically including: Based on the intrusion detection model, a maximum number of features is preset. For the number of features The data is then populated and scaled. Zero fill: for those with Data with features, create a data set with a length of . The zero vector, then the original The eigenvalues ​​are placed into the zero vector before One position, the rest Each position remains 0; Scaling: Introducing a scaling factor All shortcomings 3D data is obtained by multiplying each feature by a scaling factor. This amplifies the eigenvalues.

4. The intrusion detection method based on end-to-end flow fusion according to claim 1, characterized in that, The classification result is obtained by using the argmax function based on the predicted probability distribution.

5. An intrusion detection system with end-to-end fusion, characterized in that, The system includes: The terminal and traffic data collection and fusion module is used to collect system logs and traffic data on the terminal and to unify the data format and length. The model design and training module is used to design an intrusion detection model based on the Transformer architecture and train the model based on prior learning. The intrusion detection module is used to deploy the trained intrusion detection model to the actual environment for intrusion detection. It samples a small amount of data for annotation to form a training dataset and test samples, which are then input into the trained model. During the detection process, features are aggregated at regular intervals to generate a multi-dimensional numerical feature table. The model directly outputs the predicted probability distribution of the test samples through a single forward propagation. The classification result is obtained based on the predicted probability distribution, and a threshold is used to determine whether it is an attack. Training the intrusion detection model includes the following: Training a model , making the model It is possible to estimate the true distribution of the data from the sampled data: ,in For the training dataset, For test data features, For test data features The corresponding tags, i.e., real data tags, This represents the true probability distribution of the data; Based on the prior learning method, the collected dataset is separated according to different labels, that is, different attack types are regarded as the prior of the attack data distribution. In each training step, a dataset is sampled from the prior. , Includes training dataset and test samples ,Will and Test data characteristics As input to the model, the model's goal is to predict... The corresponding real data labels ; The intrusion detection model uses the cross-entropy loss function as its loss function. During training, attention masks are used to allow the training datasets to pay attention to each other, while the test samples can only pay attention to the training dataset and cannot pay attention to other test samples.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the intrusion detection method steps of the end-to-end fusion method as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium, wherein computer instructions are stored on the storage medium, characterized in that, When the instruction is executed by the processor, it implements the steps of the intrusion detection method with end-to-end fusion as described in any one of claims 1 to 4.

Images