Network attack detection model training method and device, computer equipment, medium and product

Abstract

The invention relates to a training method and device of a network attack detection model, computer equipment, a medium and a product. The method comprises the following steps: constructing an original traffic feature according to current network data, constructing a detection model, generating an action feature vector according to the original traffic feature through a reinforcement learning agent, screening from the current network data through the action feature vector to obtain first training data, on the basis of the first training data and second training data output by the generator, performing multi-round alternate training on the detection model; wherein each round of alternate training comprises training the discriminator based on the first training data and the second training data and training the generator based on a judgment result of the trained discriminator on the second training data. By adopting the method, the detection accuracy can be improved.

Description

Technical Field

[0001] This application relates to the field of cybersecurity technology, and in particular to a training method, apparatus, computer equipment, medium, and product for a network attack detection model. Background Technology

[0002] With the continuous growth in the scale and complexity of cyberspace, cyberattack methods have shown significant characteristics of concealment, intelligence, and multi-stage approaches. Traditional intrusion detection systems (IDS) based on signature or rule matching rely on known attack signature databases and are therefore unable to effectively identify unknown attacks and dynamically evolving attack patterns.

[0003] Currently, machine learning-based network attack detection schemes are commonly used to improve the ability to capture dynamic attack behaviors through temporal feature extraction, thereby optimizing the performance of detection models in complex network environments. However, machine learning-based network attack detection schemes lack the ability to effectively model the global temporal evolution characteristics of attack behaviors. They typically rely on static feature selection or feature extraction methods with fixed time windows, making it difficult to capture the correlation and dynamic evolution patterns of each link in a multi-stage attack chain. As a result, when facing latent, multi-stage attacks such as APT penetration, they are prone to missed detections due to incomplete feature capture. Summary of the Invention

[0004] Therefore, it is necessary to provide a training method, apparatus, computer equipment, medium, and product for a network attack detection model that can improve detection accuracy in response to the above-mentioned technical problems.

[0005] Firstly, this application provides a method for training a network attack detection model, including:

[0006] Based on current network data, construct raw traffic characteristics and build a detection model; the detection model includes a generator and a discriminator;

[0007] The reinforcement learning agent generates action feature vectors based on the original traffic features; the action feature vectors are used to represent the combination of target features selected from the original traffic features for the current training iteration.

[0008] The first training data is obtained by filtering the current network data through action feature vectors. Based on the first training data and the second training data output by the generator, the detection model is trained in multiple rounds of alternating training. Each round of alternating training includes training the discriminator based on the first and second training data and training the generator based on the judgment result of the discriminator on the second training data after training.

[0009] In one embodiment, the current network data includes first network data collected by an edge device and second network data collected by a transmission device; the step of constructing original traffic characteristics based on the current network data includes:

[0010] Obtain time-series sample sequences based on the first network data and the second network data;

[0011] Based on the preset feature dimensions, feature extraction is performed on the time-series sample sequence to obtain the original traffic features.

[0012] In one embodiment, the step of generating action feature vectors by a reinforcement learning agent based on the original traffic features includes:

[0013] The initial action set of the reinforcement learning agent is obtained based on the original traffic features; each element in the initial action set corresponds to the selection state of an original traffic feature.

[0014] The initial action probability is updated based on the current number of training iterations to obtain the current action probability;

[0015] The initial set of actions is filtered based on the current action probability to obtain action feature vectors.

[0016] In one embodiment, the discriminator includes a first discriminator and a second discriminator; the process of training the discriminator based on the first training data and the second training data includes:

[0017] The first training data and the second training data are used together as input samples. The first discriminator judges the attack behavior of the input samples to obtain the first prediction result, and the second discriminator judges the authenticity of the data of the input samples to obtain the second prediction result.

[0018] Based on the difference between the first prediction result and the data labels of the first training data, a first loss function is obtained, and based on the difference between the second prediction result and the data labels of the second training data, a second loss function is obtained.

[0019] The network parameters of the first discriminator and the second discriminator are updated respectively, and the corresponding prediction results are obtained again until the corresponding loss function converges.

[0020] In one embodiment, the process of training the generator based on the judgment result of the trained discriminator on the second training data includes:

[0021] The second training data is input into the trained discriminator to obtain the true or false judgment result output by the discriminator;

[0022] The third loss function is obtained based on the true / false determination result; the third loss function is used to characterize the probability that the discriminator will determine the second training data as real data.

[0023] The network parameters of the generator are updated using gradient descent, and a second training data is reacquired; the second training data is generated by the generator based on random noise.

[0024] Return to the step of inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator, and continue to execute until the three loss functions reach convergence and stop.

[0025] In one embodiment, the method further includes:

[0026] Obtain the action experience tuple of the reinforcement learning agent, and randomly sample the action experience tuple to obtain the sample experience tuple;

[0027] Based on the sample empirical tuples, the temporal difference error corresponding to the original traffic characteristics is obtained;

[0028] Update the action value function value corresponding to the original flow characteristics based on the time series difference error;

[0029] The feature selection action corresponding to the original traffic feature with the maximum action value function value within a preset training period is obtained to obtain a subset of target features.

[0030] The probability of the current action of the reinforcement learning agent is updated based on a subset of target features.

[0031] Secondly, this application also provides a training device for a network attack detection model, comprising:

[0032] The feature construction module is used to construct raw traffic features based on current network data and build a detection model; the detection model includes a generator and a discriminator.

[0033] The vector acquisition module is used to generate action feature vectors based on the original traffic features by the reinforcement learning agent; the action feature vectors are used to represent the combination of target features selected from the original traffic features for the current training.

[0034] The model training module is used to select first training data from the current network data through action feature vectors, and to perform multiple rounds of alternating training on the detection model based on the first training data and the second training data output by the generator. Each round of alternating training includes training the discriminator based on the first training data and the second training data, and training the generator based on the judgment result of the discriminator on the second training data after training.

[0035] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method steps of any one of the first aspects.

[0036] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the method steps of any one of the first aspects.

[0037] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the method steps of any one of the first aspects.

[0038] The training method, apparatus, computer equipment, medium, and product of the aforementioned network attack detection model constructs original traffic features based on current network data and builds a detection model. A reinforcement learning agent generates action feature vectors based on the original traffic features. The action feature vectors are used to filter the first training data from the current network data. Based on the first training data and the second training data output by the generator, the detection model is trained in multiple rounds of alternating training. This allows the model to adapt to different network environments and attack types, accurately capture temporal patterns in network traffic, and improve the accuracy of network risk detection. Attached Figure Description

[0039] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0040] Figure 1 This is a diagram illustrating the application environment of a training method for a network attack detection model in one embodiment.

[0041] Figure 2 This is a flowchart illustrating the training method of a network attack detection model in one embodiment;

[0042] Figure 3 This is a flowchart illustrating the training method of the network attack detection model in another embodiment;

[0043] Figure 4 This is a structural block diagram of a training device for a network attack detection model in one embodiment;

[0044] Figure 5 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0046] The training method for the network attack detection model provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. Terminal 102 is used to construct original traffic features based on current network data and build a detection model. Through a reinforcement learning agent, it generates action feature vectors based on the original traffic features. First training data is obtained by filtering from the current network data using these action feature vectors. Based on the first training data and second training data output by the generator, the detection model undergoes multiple rounds of alternating training. Each round of alternating training includes training the discriminator based on the first and second training data, and training the generator based on the discriminator's judgment result on the second training data. Terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, drones, low-altitude aircraft, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, projection devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Headset devices can be virtual reality (VR) devices, augmented reality (AR) devices, smart glasses, etc. Server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0047] In one exemplary embodiment, such as Figure 2 As shown, a training method for a network attack detection model is provided, which is then applied to... Figure 1 Taking terminal 102 as an example, the explanation includes the following steps 202 to 206. Wherein:

[0048] S202: Construct raw traffic characteristics based on current network data and build a detection model; the detection model includes a generator and a discriminator.

[0049] Optionally, the current network data includes data collected from multiple sources. Taking a digital power grid as an example, the power grid is collected in layers. The current network data is mainly divided into two categories: edge layer data and transmission layer data. Edge layer data refers to the collection of communication traffic, control commands, and telemetry messages (such as smart terminal remote signaling data and maintenance logs) of field equipment through lightweight detection agents at locations such as substations and distribution rooms. Transmission layer data refers to the collection of cross-domain traffic data through detection agents at switch aggregation nodes (avoiding the one-sidedness of single-domain data). The two types of heterogeneous data are converted into raw traffic characteristics to ensure that the characteristics cover both the static attributes and dynamic temporal sequence of network behavior.

[0050] Optionally, the detection model employs a combination of a generator and a discriminator. The generator uses a Temporal Convolutional Network (TCN) architecture, which captures the long-term temporal dependencies of network traffic through dilated convolutions and residual connections. Dilated convolutions expand the receptive field (covering traffic changes across multiple time windows) by inserting zero elements, while residual connections prevent gradient vanishing in deep networks, ensuring that the generator can output simulated samples consistent with the temporal characteristics of real attack traffic, thus addressing the problem of scarce attack samples. The discriminator includes an attack detection discriminator and a data authenticity discriminator. The attack detection discriminator determines whether a sample is attack traffic, while the data authenticity discriminator determines whether the sample is real or simulated data.

[0051] S204: The reinforcement learning agent generates an action feature vector based on the original traffic features; the action feature vector is used to represent the combination of target features selected from the original traffic features for the current training.

[0052] Optionally, the reinforcement learning agent uses a deep Q-network (DQN) as the agent, where its state S(p) is a 182-dimensional binary vector representing the currently selected feature subset (1 indicates selection, 0 indicates non-selection). The action A(p) is also a 182-dimensional vector, with each dimension representing the selection or deselection of the corresponding feature. The agent selects action A(p) based on the current state S(p) and the ε-greedy policy. The ε-greedy policy initially selects actions randomly with high probability (exploration), gradually increasing the probability of selecting the currently estimated optimal action (exploitation) as training progresses. By continuously updating the Q-value function, the agent learns which feature combinations can improve the discriminator's detection accuracy. The target feature combination is the execution result of the action feature vector (i.e., action A(p)), representing the next state S(p+1), i.e., the feature subset selected in the current round. This feature subset is used to filter some features from the original traffic features as features for training in the current round.

[0053] S206: First training data is obtained by filtering the current network data through action feature vectors. Based on the first training data and the second training data output by the generator, the detection model is trained in multiple rounds of alternating training. Each round of alternating training includes training the discriminator based on the first training data and the second training data, and training the generator based on the judgment result of the discriminator on the second training data after training.

[0054] Optionally, based on the action feature vector (i.e., the current feature subset), corresponding feature columns are selected from the current network data to obtain the first training data, which serves as the subset of real data. During alternating training, the discriminator is first trained using the first training data (real data) and the second training data (simulated data) generated by the generator. For the generator, training is performed based on the discriminator's discrimination results on the generated data. This alternating training process is repeated until the model converges or reaches the preset number of training iterations.

[0055] In the training method of the above network attack detection model, the original traffic features are constructed based on the current network data, and the detection model is constructed. The reinforcement learning agent generates action feature vectors based on the original traffic features. The action feature vectors are used to filter the first training data from the current network data. Based on the first training data and the second training data output by the generator, the detection model is trained in multiple rounds of alternating training. This method can adapt to different network environments and attack types, accurately capture the temporal patterns in network traffic, and improve the accuracy of network risk detection.

[0056] In an exemplary embodiment, the current network data includes first network data collected by an edge device and second network data collected by a transmission device; the step of constructing original traffic features based on the current network data includes: obtaining a time-series sample sequence based on the first network data and the second network data; and extracting features from the time-series sample sequence according to a preset feature dimension to obtain the original traffic features.

[0057] Optionally, in complex network environments such as digital power grids, data acquisition is divided into an edge layer and a transmission layer. The edge layer collects traffic data at field equipment such as substations and distribution rooms, while the transmission layer collects cross-domain traffic data at aggregation nodes. The data collected by the edge and transmission layers (such as communication traffic, control commands, and telemetry messages) are organized into a time-series sample sequence, where each sample represents network behavior within a time window. Features are extracted from each time-series sample according to 182 preset feature dimensions, including communication rate, command type, data field change rate, session duration, and source / destination node identifiers. When abnormal traffic surges, control command deviations, or abnormal port access frequencies occur, the edge layer can trigger rapid data transmission.

[0058] In this embodiment, by obtaining time-series sample sequences based on first network data and second network data, and extracting features from the time-series sample sequences according to preset feature dimensions, the original traffic features are obtained. This can improve the comprehensiveness of feature data, accurately identify the time-series features of network features, and thus improve the accuracy of network attack detection.

[0059] In an exemplary embodiment, the step of generating an action feature vector by a reinforcement learning agent based on the original traffic features includes: obtaining an initial action set of the reinforcement learning agent based on the original traffic features; each element in the initial action set corresponds to a selection state of an original traffic feature; updating the initial action probability based on the current training iterations to obtain the current action probability; and filtering the initial action set based on the current action probability to obtain the action feature vector.

[0060] Optionally, the original traffic features are 182-dimensional. The core task of the reinforcement learning agent is to select the optimal subset from these 182 features. Therefore, the definition of the initial action set must correspond one-to-one with the feature dimensions to ensure that each feature has a clear selection operation. The initial action set is a set of 182-dimensional binary vectors. Each element (action) corresponds to only one selection state of the original traffic feature, and each action has only two values: when the action value is 1, it indicates that the original traffic feature is selected (i.e., the feature is included in the target feature combination for subsequent model training); when the action value is 0, it indicates that the original traffic feature is not selected (i.e., the feature is removed to avoid redundant noise interference). The initial action set provides the agent with clear operational boundaries, avoiding problems such as the agent having no actions to execute or actions not matching features.

[0061] Furthermore, in the initial state, the agent has no historical decision-making experience. Therefore, the initial action probabilities aim for high exploratory potential. For example, the probability of randomly selecting an action (exploration) is set to be as high as 99.9%, while the probability of selecting the optimal action based on experience (exploitation) is only 0.1%, avoiding getting trapped in local optima due to insufficient experience in the early stages. As the training rounds iterate, the action probabilities gradually approach the target value; that is, random exploration gradually decreases, while the probability of experience utilization gradually increases. Among these, the action probability of experience utilization depends on the action value function. In the updated current action probabilities, the probability of selecting the action that maximizes the action value function increases with the number of training rounds, ensuring that later decisions rely more on proven effective experience.

[0062] Optionally, the current action probability defines the execution probability of each action (selecting / not selecting a certain feature). The filtering process is essentially selecting the optimal action according to probability, combining them into a vector representing the target feature combination. For the 182 actions in the initial action set, decisions are made one by one according to the current action probability. If the random exploration probability is triggered, then 1 or 0 is randomly selected as the action for that feature; if the experience utilization probability is triggered, then the action that maximizes the action value function is selected. The filtered actions of the 182 features are combined into a 182-dimensional binary vector, which is the action feature vector. The features corresponding to 1 in the vector constitute the target feature combination for the current training, and the features corresponding to 0 are discarded.

[0063] In this embodiment, the initial action set of the reinforcement learning agent is obtained based on the original traffic characteristics. The initial action probability is updated according to the current training number to obtain the current action probability. The initial action set is then filtered based on the current action probability to obtain the action feature vector. This enables dynamic feature optimization, improves feature selection accuracy, and thus enhances model training efficiency and detection performance.

[0064] In an exemplary embodiment, the discriminator includes a first discriminator and a second discriminator. The process of training the discriminator based on the first training data and the second training data includes: using the first training data and the second training data together as input samples, using the first discriminator to judge the attack behavior of the input samples to obtain a first prediction result, and using the second discriminator to judge the authenticity of the input samples to obtain a second prediction result; obtaining a first loss function based on the difference between the first prediction result and the data labels of the first training data, and obtaining a second loss function based on the difference between the second prediction result and the data labels of the second training data; updating the network parameters of the first discriminator and the network parameters of the second discriminator respectively, and re-obtaining the corresponding prediction results, until the corresponding loss function reaches convergence.

[0065] Optionally, the first and second discriminators have different task objectives and are adapted to their respective tasks by differentiating themselves through the labels of the input samples. The input samples are a mixed dataset of first and second training data. The first training data consists of real network data filtered through action feature vectors, including normal traffic samples and real attack traffic samples, and includes two types of labeling. The second training data consists of simulated attack samples output by the generator (mimicking the timing and characteristics of real attacks). The first and second training data are mixed in a preset ratio to ensure that the input samples cover both real / simulated data and normal / attack data, preventing the first discriminator from only learning real attacks and the second discriminator from only recognizing a single data type, thus providing comprehensive training samples for both discriminators.

[0066] Optionally, after receiving mixed input samples, the first discriminator extracts temporal attack features (such as abnormal command sequences and packet length mutations within multiple time windows) from the samples using a TCN, and outputs a first prediction result indicating whether the sample is attack traffic. The core of this approach is learning the behavioral differences between normal and attack traffic. The second discriminator extracts real / simulated features (such as random noise distribution in real data and pattern bias in simulated data) from the samples using a TCN, and outputs a second prediction result indicating whether the sample is real data. The core of this approach is learning the distribution differences between real collected data and generator-simulated data. Both discriminators use a binary cross-entropy loss function. Since both discrimination tasks are binary classification problems, the discriminator parameters are updated using stochastic gradient descent, and the generator parameters are fixed during training (to avoid generator fluctuations affecting discriminator training). In multiple consecutive training rounds, when the decrease in the loss value is less than a preset threshold, it indicates that the discriminator has sufficiently learned attack and real / false features, and its performance has stabilized. Continuing training will not significantly improve accuracy, at which point discriminator training is stopped, and the subsequent generator training phase begins.

[0067] In this embodiment, by training the discriminator based on the first training data and the second training data, the discrimination accuracy of the discriminator can be improved, providing reliable data feedback for subsequent generator training, thereby improving the reliability of generator training.

[0068] In an exemplary embodiment, the process of training the generator based on the judgment result of the discriminator on the second training data includes: inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator; obtaining a third loss function based on the true / false judgment result; the third loss function is used to characterize the probability that the discriminator judges the second training data as real data; updating the network parameters of the generator by gradient descent and re-acquiring the second training data; the second training data is generated by the generator based on random noise; returning to the step of inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator, and continuing to execute until the three loss functions converge.

[0069] Optionally, the core task of the generator is to generate simulated samples (second training data) that are consistent with the temporal sequence and characteristics of real attack traffic. The trained discriminator, especially the second discriminator, can evaluate the realism of the simulated samples. The generator receives random noise and target feature vectors filtered by reinforcement learning as input. The random noise is used to introduce diversity, and the target feature vectors are used to constrain the feature range of the simulated samples. The output is a temporally sequenced simulated attack sample, i.e., the second training data. The newly generated second training data is input into the trained and converged second discriminator. The second discriminator extracts the true and false features of the second training data through TCN and outputs the probability that the sample is real data as the true or false judgment result.

[0070] Furthermore, if the fidelity is low, the generator needs to adjust its parameters to make the subsequently generated samples more closely resemble real data; if the fidelity is high, the current parameter trend needs to be maintained to avoid over-optimization leading to pattern collapse, providing a direct basis for subsequent loss calculation. A third loss function is calculated based on the fidelity judgment results to quantify the fidelity deviation of the simulated samples. The generator parameters are updated using gradient descent, and the second discriminator parameters are fixed during training (to avoid fluctuations in the second discriminator affecting the generator's optimization direction). The third loss function is gradually reduced through successive adjustments until convergence is achieved, i.e., the decrease in the third loss function is less than a preset threshold, indicating that the generator can no longer further reduce the deviation, and the fidelity of the generated simulated samples is stable. At this point, generator training is stopped, completing one round of discriminator-generator alternating training.

[0071] In this embodiment, by training the generator based on the judgment results of the second training data by the trained discriminator, the generalization ability of the model can be improved, the model can be accurately adapted to various attack data, and the stability and reliability of the prediction model can be improved.

[0072] In an exemplary embodiment, the method further includes: obtaining an action experience set of the reinforcement learning agent, and randomly sampling the action experience set to obtain a sample experience set; obtaining the temporal difference error corresponding to the original traffic feature based on the sample experience set; updating the action value function value corresponding to the original traffic feature according to the temporal difference error; obtaining the feature selection action of the original traffic feature corresponding to the action value function value with the maximum value within a preset training period to obtain a target feature subset; and updating the current action probability of the reinforcement learning agent based on the target feature subset.

[0073] Optionally, the action experience tuple includes the current feature selection state, the optimal action in the current round, the reward for the current action, and the updated state for the next round after executing the optimal action in the current round. After each round of feature selection and model training, one experience tuple is generated and stored in a buffer. By randomly sampling the experience tuples in the buffer, sample experience tuples are obtained, ensuring that the experience for each Q-network update comes from different rounds and states, thus ensuring more stable parameter updates and avoiding local optima. The Q-network update employs temporal difference learning. The core of this process is to calculate the target action value function value using the immediate reward of the current experience and the estimated reward of the next state, and then compare this error with the current action value function value, using this error as the basis for updates. If the temporal difference error is positive, it indicates that the current action value function value underestimates the true benefit of executing the optimal action in the current round; if it is negative, it overestimates the benefit. This error directly quantifies the degree of bias in the Q-network's prediction, providing a correction direction for subsequent action value function value updates. During the update, the gradient of the loss target action value function with respect to the Q network parameters is calculated, and the parameters are updated using the gradient descent method. After each update, the action value function value will move closer to the target value, enabling the agent to more accurately predict the reward of the optimal action in the current round corresponding to the selected feature combination, and making the decision more reliable.

[0074] Furthermore, the system selects actions that yield the greatest benefit from the updated target action values ​​and adjusts the action probabilities to make the agent more inclined to choose these high-yield actions in subsequent training. Specifically, the optimal action combinations of all features are combined to form a target feature subset. Based on this target feature subset, the agent's current action probabilities are adjusted: for features in the target feature subset, the probability of selecting an action is increased; for features outside the target feature subset, the probability of selecting an action is decreased. This ensures that in subsequent training, the agent is more likely to choose feature combinations with high action value function values ​​and high benefits, reducing ineffective exploration.

[0075] In this embodiment, by acquiring the action experience tuples of the reinforcement learning agent and randomly sampling the action experience tuples to obtain sample experience tuples, the temporal difference error corresponding to the original traffic features is obtained based on the sample experience tuples. According to the temporal difference error, the action value function value corresponding to the original traffic features is updated. The feature selection action corresponding to the original traffic feature with the maximum action value function value within a preset training period is obtained, and the target feature subset is obtained. The current action probability of the reinforcement learning agent is updated based on the target feature subset. This can improve the convergence speed and reliability of the reinforcement learning agent, accurately adapt to the dynamic network environment, improve feature selection efficiency, and thus improve the model detection accuracy.

[0076] In one exemplary embodiment, such as Figure 3 As shown, a training method for a network attack detection model is provided, which includes the following steps:

[0077] (1) Feature initialization: Obtain time-series sample sequences based on the first network data and the second network data; extract features from the time-series sample sequences according to the preset feature dimensions to obtain the original traffic features; construct a detection model; the detection model includes a generator and a discriminator.

[0078] (2) Feature selection: Obtain the initial action set of the reinforcement learning agent based on the original traffic features; each element in the initial action set corresponds to the selection state of an original traffic feature; update the initial action probability based on the current training iteration to obtain the current action probability; filter the initial action set based on the current action probability to obtain the action feature vector. The action feature vector is used to represent the target feature combination selected from the original traffic features for the current training iteration.

[0079] (3) Discriminator training: First training data is obtained by filtering the current network data through action feature vectors. The first training data and the second training data are used together as input samples. The first discriminator judges the attack behavior of the input samples to obtain the first prediction result. The second discriminator judges the authenticity of the input samples to obtain the second prediction result. Based on the difference between the data labels of the first prediction result and the first training data, the first loss function is obtained. Based on the difference between the data labels of the second prediction result and the second training data, the second loss function is obtained. The network parameters of the first discriminator and the second discriminator are updated respectively, and the corresponding prediction results are obtained again until the corresponding loss function reaches convergence.

[0080] (4) Generator training: Input the second training data into the trained discriminator and obtain the true / false judgment result output by the discriminator; obtain the third loss function based on the true / false judgment result; the third loss function is used to characterize the probability that the discriminator judges the second training data as real data; update the network parameters of the generator by gradient descent and re-obtain the second training data; the second training data is generated by the generator based on random noise; return to the step of inputting the second training data into the trained discriminator and obtaining the true / false judgment result output by the discriminator, and continue to execute until the three loss functions reach convergence.

[0081] (5) Action update: Obtain the action experience tuple of the reinforcement learning agent and randomly sample the action experience tuple to obtain the sample experience tuple; based on the sample experience tuple, obtain the temporal difference error corresponding to the original flow feature; update the action value function value corresponding to the original flow feature according to the temporal difference error; obtain the feature selection action of the original flow feature corresponding to the action value function value with the maximum value within the preset training period to obtain the target feature subset; update the current action probability of the reinforcement learning agent based on the target feature subset.

[0082] In this embodiment, original traffic features are constructed based on current network data, and a detection model is built. A reinforcement learning agent generates action feature vectors based on the original traffic features. First training data is obtained by filtering the current network data using the action feature vectors. Based on the first training data and the second training data output by the generator, the detection model is trained in multiple rounds of alternating training. This allows the model to adapt to different network environments and attack types, accurately capture temporal patterns in network traffic, and improve the accuracy of network risk detection.

[0083] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0084] Based on the same inventive concept, this application also provides a training apparatus for a network attack detection model to implement the training method for the network attack detection model described above. The solution provided by this apparatus is similar to the implementation described in the above method. Therefore, the specific limitations of one or more network attack detection model training apparatus embodiments provided below can be found in the limitations of the network attack detection model training method described above, and will not be repeated here.

[0085] In one exemplary embodiment, such as Figure 4 As shown, a training device for a network attack detection model is provided, comprising: a feature construction module 10, a vector acquisition module 20, and a model training module 30, wherein:

[0086] The feature construction module 10 is used to construct raw traffic features based on the current network data and to build a detection model; the detection model includes a generator and a discriminator.

[0087] The vector acquisition module 20 is used to generate action feature vectors based on the original traffic features by the reinforcement learning agent; the action feature vectors are used to represent the combination of target features selected from the original traffic features for the current training.

[0088] The model training module 30 is used to select first training data from the current network data through action feature vectors, and to perform multiple rounds of alternating training on the detection model based on the first training data and the second training data output by the generator; wherein, each round of alternating training includes training the discriminator based on the first training data and the second training data, and training the generator based on the judgment result of the trained discriminator on the second training data.

[0089] In an exemplary embodiment, the current network data includes first network data collected by an edge device and second network data collected by a transmission device; the feature construction module 10 is further configured to obtain a time-series sample sequence based on the first network data and the second network data; and to extract features from the time-series sample sequence according to a preset feature dimension to obtain the original traffic features.

[0090] In an exemplary embodiment, the vector acquisition module 20 is further configured to acquire an initial action set of the reinforcement learning agent based on the original traffic features; each element in the initial action set corresponds to a selection state of an original traffic feature; the initial action probability is updated based on the current number of training iterations to obtain the current action probability; and the initial action set is filtered based on the current action probability to obtain an action feature vector.

[0091] In an exemplary embodiment, the discriminator includes a first discriminator and a second discriminator; the model training module 30 is further configured to use the first training data and the second training data together as input samples, use the first discriminator to judge the attack behavior of the input samples to obtain a first prediction result, and use the second discriminator to judge the authenticity of the input samples to obtain a second prediction result; based on the difference between the data labels of the first prediction result and the first training data, obtain a first loss function, and based on the difference between the data labels of the second prediction result and the second training data, obtain a second loss function; update the network parameters of the first discriminator and the second discriminator respectively, and re-obtain the corresponding prediction results until the corresponding loss function reaches convergence.

[0092] In an exemplary embodiment, the model training module 30 is further configured to input the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator; obtain a third loss function based on the true / false judgment result; the third loss function is used to characterize the probability that the discriminator judges the second training data as real data; update the network parameters of the generator by gradient descent and re-acquire the second training data; the second training data is generated by the generator based on random noise; return to the step of inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator, and continue to execute until the three loss functions converge.

[0093] In an exemplary embodiment, the vector acquisition module 20 is further configured to acquire the action experience tuples of the reinforcement learning agent, and randomly sample the action experience tuples to obtain sample experience tuples; based on the sample experience tuples, acquire the temporal difference error corresponding to the original traffic features; update the action value function value corresponding to the original traffic features according to the temporal difference error; acquire the feature selection action of the original traffic features corresponding to the action value function value with the maximum value within a preset training period, and obtain a target feature subset; and update the current action probability of the reinforcement learning agent based on the target feature subset.

[0094] Each module in the training device of the aforementioned network attack detection model can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0095] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 5 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a training method for a network attack detection model. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0096] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0097] In an exemplary embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor executes the computer program to perform the following steps: constructing original traffic features based on current network data and constructing a detection model; the detection model includes a generator and a discriminator; generating action feature vectors based on the original traffic features using a reinforcement learning agent; the action feature vectors are used to characterize the target feature combination used in the current training, selected from the original traffic features; obtaining first training data from the current network data using the action feature vectors; and performing multiple rounds of alternating training on the detection model based on the first training data and second training data output by the generator; wherein each round of alternating training includes training the discriminator based on the first training data and the second training data, and training the generator based on the judgment result of the trained discriminator on the second training data.

[0098] In one embodiment, the current network data includes first network data collected by an edge device and second network data collected by a transmission device; the process of constructing original traffic features based on the current network data when the processor executes a computer program includes: obtaining a time-series sample sequence based on the first network data and the second network data; and extracting features from the time-series sample sequence according to a preset feature dimension to obtain the original traffic features.

[0099] In one embodiment, when the processor executes a computer program, the generation of action feature vectors by a reinforcement learning agent based on original traffic features includes: obtaining an initial action set of the reinforcement learning agent based on the original traffic features; each element in the initial action set corresponds to a selection state of an original traffic feature; updating the initial action probability based on the current number of training iterations to obtain the current action probability; and filtering the initial action set based on the current action probability to obtain the action feature vector.

[0100] In one embodiment, the discriminator includes a first discriminator and a second discriminator. The process of training the discriminator based on the first training data and the second training data when the processor executes the computer program includes: using the first training data and the second training data together as input samples, using the first discriminator to judge the attack behavior of the input samples to obtain a first prediction result, and using the second discriminator to judge the authenticity of the input samples to obtain a second prediction result; obtaining a first loss function based on the difference between the first prediction result and the data labels of the first training data, and obtaining a second loss function based on the difference between the second prediction result and the data labels of the second training data; updating the network parameters of the first discriminator and the network parameters of the second discriminator respectively, and re-obtaining the corresponding prediction results, until the corresponding loss function reaches convergence.

[0101] In one embodiment, the process of training a generator based on the judgment result of a trained discriminator on second training data when the processor executes a computer program includes: inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator; obtaining a third loss function based on the true / false judgment result; the third loss function is used to characterize the probability that the discriminator judges the second training data as real data; updating the network parameters of the generator by gradient descent and re-acquiring the second training data; the second training data is generated by the generator based on random noise; returning to the step of inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator, and continuing execution until the three loss functions converge.

[0102] In one embodiment, when the processor executes the computer program, it further implements the following steps: obtaining the action experience tuple of the reinforcement learning agent, and randomly sampling the action experience tuple to obtain the sample experience tuple; obtaining the temporal difference error corresponding to the original traffic feature based on the sample experience tuple; updating the action value function value corresponding to the original traffic feature according to the temporal difference error; obtaining the feature selection action of the original traffic feature corresponding to the action value function value with the maximum value within a preset training period, and obtaining the target feature subset; updating the current action probability of the reinforcement learning agent based on the target feature subset.

[0103] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program performs the following steps: constructing original traffic features based on current network data and constructing a detection model; the detection model includes a generator and a discriminator; generating action feature vectors based on the original traffic features using a reinforcement learning agent; the action feature vectors are used to characterize the target feature combination used in the current training, selected from the original traffic features; obtaining first training data from the current network data using the action feature vectors; and performing multiple rounds of alternating training on the detection model based on the first training data and second training data output by the generator; wherein each round of alternating training includes training the discriminator based on the first training data and the second training data, and training the generator based on the judgment result of the trained discriminator on the second training data.

[0104] In one embodiment, the current network data includes first network data collected by an edge device and second network data collected by a transmission device; the process of constructing original traffic features based on the current network data when the computer program is executed by the processor includes: obtaining a time-series sample sequence based on the first network data and the second network data; and extracting features from the time-series sample sequence according to a preset feature dimension to obtain the original traffic features.

[0105] In one embodiment, when a computer program is executed by a processor, it involves generating an action feature vector based on original traffic features by a reinforcement learning agent, including: obtaining an initial action set of the reinforcement learning agent based on the original traffic features; each element in the initial action set corresponds to a selection state of an original traffic feature; updating the initial action probability based on the current number of training iterations to obtain the current action probability; and filtering the initial action set based on the current action probability to obtain the action feature vector.

[0106] In one embodiment, the discriminator includes a first discriminator and a second discriminator; the process of training the discriminator based on the first training data and the second training data when the computer program is executed by the processor includes: using the first training data and the second training data together as input samples, using the first discriminator to judge the attack behavior of the input samples to obtain a first prediction result, and using the second discriminator to judge the authenticity of the input samples to obtain a second prediction result; obtaining a first loss function based on the difference between the first prediction result and the data labels of the first training data, and obtaining a second loss function based on the difference between the second prediction result and the data labels of the second training data; updating the network parameters of the first discriminator and the network parameters of the second discriminator respectively, and re-obtaining the corresponding prediction results, until the corresponding loss function reaches convergence and stops.

[0107] In one embodiment, the process of training a generator based on the judgment result of a trained discriminator on second training data when the computer program is executed by a processor includes: inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator; obtaining a third loss function based on the true / false judgment result; the third loss function is used to characterize the probability that the discriminator judges the second training data as real data; updating the network parameters of the generator by gradient descent and re-acquiring the second training data; the second training data is generated by the generator based on random noise; returning to the step of inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator, and continuing execution until the three loss functions converge.

[0108] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the action experience tuple of the reinforcement learning agent, and randomly sampling the action experience tuple to obtain the sample experience tuple; obtaining the temporal difference error corresponding to the original traffic feature based on the sample experience tuple; updating the action value function value corresponding to the original traffic feature according to the temporal difference error; obtaining the feature selection action of the original traffic feature corresponding to the action value function value with the maximum value within a preset training period, and obtaining the target feature subset; updating the current action probability of the reinforcement learning agent based on the target feature subset.

[0109] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, performs the following steps: constructing original traffic features based on current network data and constructing a detection model; the detection model includes a generator and a discriminator; generating action feature vectors based on the original traffic features using a reinforcement learning agent; the action feature vectors are used to characterize the target feature combination selected from the original traffic features for the current training iteration; obtaining first training data from the current network data using the action feature vectors; and performing multiple rounds of alternating training on the detection model based on the first training data and second training data output by the generator; wherein each round of alternating training includes training the discriminator based on the first training data and the second training data, and training the generator based on the judgment result of the trained discriminator on the second training data.

[0110] In one embodiment, the current network data includes first network data collected by an edge device and second network data collected by a transmission device; the process of constructing original traffic features based on the current network data when the computer program is executed by the processor includes: obtaining a time-series sample sequence based on the first network data and the second network data; and extracting features from the time-series sample sequence according to a preset feature dimension to obtain the original traffic features.

[0111] In one embodiment, when a computer program is executed by a processor, it involves generating an action feature vector based on original traffic features by a reinforcement learning agent, including: obtaining an initial action set of the reinforcement learning agent based on the original traffic features; each element in the initial action set corresponds to a selection state of an original traffic feature; updating the initial action probability based on the current number of training iterations to obtain the current action probability; and filtering the initial action set based on the current action probability to obtain the action feature vector.

[0112] In one embodiment, the discriminator includes a first discriminator and a second discriminator; the process of training the discriminator based on the first training data and the second training data when the computer program is executed by the processor includes: using the first training data and the second training data together as input samples, using the first discriminator to judge the attack behavior of the input samples to obtain a first prediction result, and using the second discriminator to judge the authenticity of the input samples to obtain a second prediction result; obtaining a first loss function based on the difference between the first prediction result and the data labels of the first training data, and obtaining a second loss function based on the difference between the second prediction result and the data labels of the second training data; updating the network parameters of the first discriminator and the network parameters of the second discriminator respectively, and re-obtaining the corresponding prediction results, until the corresponding loss function reaches convergence and stops.

[0113] In one embodiment, the process of training a generator based on the judgment result of a trained discriminator on second training data when the computer program is executed by a processor includes: inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator; obtaining a third loss function based on the true / false judgment result; the third loss function is used to characterize the probability that the discriminator judges the second training data as real data; updating the network parameters of the generator by gradient descent and re-acquiring the second training data; the second training data is generated by the generator based on random noise; returning to the step of inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator, and continuing execution until the three loss functions converge.

[0114] In one embodiment, when the computer program is executed by the processor, it further performs the following steps: obtaining the action experience tuple of the reinforcement learning agent, and randomly sampling the action experience tuple to obtain the sample experience tuple; obtaining the temporal difference error corresponding to the original traffic feature based on the sample experience tuple; updating the action value function value corresponding to the original traffic feature according to the temporal difference error; obtaining the feature selection action of the original traffic feature corresponding to the action value function value with the maximum value within a preset training period, and obtaining the target feature subset; updating the current action probability of the reinforcement learning agent based on the target feature subset.

[0115] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0116] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0117] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for a network attack detection model, characterized in that, The method includes: Based on current network data, original traffic characteristics are constructed, and a detection model is built; the detection model includes a generator and a discriminator; The reinforcement learning agent generates an action feature vector based on the original traffic features; the action feature vector is used to characterize the target feature combination selected from the original traffic features for the current training iteration. First training data is obtained by filtering the current network data using the action feature vector. Based on the first training data and the second training data output by the generator, the detection model is trained in multiple rounds of alternating training. Each round of alternating training includes training the discriminator based on the first training data and the second training data, and training the generator based on the judgment result of the trained discriminator on the second training data.

2. The method according to claim 1, characterized in that, The current network data includes first network data collected through edge devices and second network data collected through transmission devices; The process of constructing the original traffic characteristics based on current network data includes: Obtain a time-series sample sequence based on the first network data and the second network data; Based on preset feature dimensions, feature extraction is performed on the time-series sample sequence to obtain the original traffic features.

3. The method according to claim 1, characterized in that, The step of generating action feature vectors based on the original traffic features using a reinforcement learning agent includes: An initial action set for the reinforcement learning agent is obtained based on the original traffic features; each element in the initial action set corresponds to a selection state of an original traffic feature. The initial action probability is updated based on the current number of training iterations to obtain the current action probability; The initial set of actions is filtered based on the current action probability to obtain an action feature vector.

4. The method according to claim 1, characterized in that, The discriminator includes a first discriminator and a second discriminator; The process of training the discriminator based on the first training data and the second training data includes: The first training data and the second training data are used together as input samples. The first discriminator is used to identify attack behavior in the input samples to obtain a first prediction result. The second discriminator is used to identify the authenticity of the data in the input samples to obtain a second prediction result. Based on the difference between the first prediction result and the data labels of the first training data, a first loss function is obtained, and based on the difference between the second prediction result and the data labels of the second training data, a second loss function is obtained. The network parameters of the first discriminator and the second discriminator are updated respectively, and the corresponding prediction results are obtained again until the corresponding loss function converges.

5. The method according to claim 1, characterized in that, The process of training the generator based on the judgment result of the discriminator on the second training data after training includes: The second training data is input into the trained discriminator to obtain the true / false judgment result output by the discriminator; A third loss function is obtained based on the authenticity determination result; the third loss function is used to characterize the probability that the discriminator determines the second training data as real data; The network parameters of the generator are updated using gradient descent, and the second training data is reacquired; the second training data is generated by the generator based on random noise. Return to the step of inputting the second training data into the trained discriminator to obtain the true / false judgment result output by the discriminator, and continue to execute until the three loss functions reach convergence and stop.

6. The method according to claim 3, characterized in that, The method further includes: Obtain the action experience tuple of the reinforcement learning agent, and randomly sample the action experience tuple to obtain the sample experience tuple; Based on the sample empirical tuples, the temporal difference error corresponding to the original traffic characteristics is obtained; Update the action value function value corresponding to the original flow characteristics based on the time-series difference error; Obtain the feature selection action corresponding to the original traffic feature with the maximum value of the action value function within a preset training period, and obtain the target feature subset; The probability of the current action of the reinforcement learning agent is updated based on the target feature subset.

7. A training device for a network attack detection model, characterized in that, The device includes: The feature construction module is used to construct raw traffic features based on current network data and build a detection model; the detection model includes a generator and a discriminator. The vector acquisition module is used to generate action feature vectors based on the original traffic features by the reinforcement learning agent; the action feature vectors are used to characterize the target feature combination selected from the original traffic features for the current training. The model training module is used to filter first training data from the current network data using the action feature vector, and to perform multiple rounds of alternating training on the detection model based on the first training data and the second training data output by the generator; wherein each round of alternating training includes training the discriminator based on the first training data and the second training data, and training the generator based on the judgment result of the trained discriminator on the second training data.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images