Intrusion detection unbalanced dataset processing method based on CVAE-GAN

Abstract

The application discloses an intrusion detection unbalanced dataset processing method based on CVAE-GAN, and comprises the following steps: preprocessing an intrusion detection dataset; performing dimension reduction processing on the preprocessed dataset by using a PCA algorithm; removing outliers by using an OPTICS clustering algorithm on the dimension-reduced data; expanding minority attack classes in the dataset by applying a constructed CVAE-GAN model; mixing the expanded data with original data to obtain a relatively balanced intrusion detection dataset, training a classifier, and testing a classification result. The application fully alleviates the unbalance degree of the intrusion detection dataset, and improves the classification accuracy of each attack class in a multi-classification problem.

Description

Technical Field

[0001] This invention relates to the field of imbalanced dataset processing optimization, and specifically to an intrusion detection imbalanced dataset processing method based on CVAE-GAN. Background Technology

[0002] With the continuous advancement of the information age, the internet needs to transmit increasingly more data streams to meet our increasingly frequent online activities. At the same time, cyberattacks such as Trojan viruses, malware, and website hijacking are also on the rise, making cybersecurity a pressing issue that requires our attention and solutions. Therefore, intrusion detection algorithms and intrusion detection systems (IDS) are becoming increasingly important, and their development and research are constantly being explored.

[0003] In natural network traffic, normal data constitutes the vast majority, while attack data, though only a small portion, can cause significant harm. Therefore, timely and accurate detection of the few abnormal traffic spikes within massive datasets is crucial. To meet the needs of more efficient and accurate Intrusion Detection System (IDS) development, realistic and high-quality datasets are also extremely important. However, the class imbalance problem present in a large portion of datasets significantly limits the development and application of IDS.

[0004] An imbalanced dataset refers to a dataset used for classification problems where the number of samples in each class is extremely unevenly distributed. When training a classifier using such a dataset, the larger number of classes can learn more extensive and comprehensive features, thus usually leading to classification results that tend to favor the majority class.

[0005] For example, the classic intrusion detection dataset KDD CUP 99 has five data categories: Normal, Probe (Surveillance or Probe), DOS (Denial-of-service attack), U2R (Unauthorized access to local superuser privileges by a local unprivileged user), and R2L (Unauthorized access from a remote machine to a local machine). The ratio of the number of categories in the training set is Normal:Probe:Dos:U2R:R2L = 97278:4107:391458:52:1126 = 1870.73:78.98:7528.04:1:21.65, which is a highly imbalanced dataset. When performing classification experiments on this dataset without any processing, the detection rates for Normal, Probe, and DOS classes can reach 80%-98%, while the detection rates for U2R and R2L classes are almost 0.

[0006] Currently, solutions to the imbalanced dataset problem are basically approached from three aspects: 1) Data: focusing on the dataset, undersampling, oversampling, or data synthesis to achieve class sample balance; 2) Algorithm: adjusting the cost function in the classification algorithm, adding cost-sensitive factors, etc., to increase the loss weight of minority class samples being misclassified as majority class samples; 3) Classifier: using ensemble classifiers, etc.

[0007] Some existing solutions use the SMOTE algorithm and its derivatives to oversample the minority class to balance the dataset. However, for intrusion detection datasets, which are large in volume and complex in composition, simple data synthesis is insufficient for effective data augmentation. Existing research indicates that using deep neural network generative models to generate minority class samples can achieve better classification results. Summary of the Invention

[0008] Purpose of the invention: To address the imbalance problem in existing technologies, this invention provides a method for handling imbalanced datasets in intrusion detection based on CVAE-GAN. This solution addresses the problem by starting with the dataset itself, effectively mitigating the imbalance in intrusion detection datasets and improving the classification accuracy of various attack categories in multi-classification problems.

[0009] Technical Solution: To achieve the above objectives, this invention provides a method for processing imbalanced datasets for intrusion detection based on CVAE-GAN, comprising the following steps:

[0010] S1: Preprocess the intrusion detection dataset;

[0011] S2: Use the PCA algorithm to reduce the dimensionality of the preprocessed dataset;

[0012] S3: Use the OPTICS clustering algorithm on the dimensionality-reduced data to remove outliers;

[0013] S4: Apply the constructed CVAE-GAN model to expand the few attack classes in the dataset;

[0014] S5: Mix the expanded data with the original data to obtain a relatively balanced intrusion detection dataset, train the classifier, and obtain the classification results.

[0015] Regarding step S1: The samples in the original dataset may contain string-type feature values, which cannot be used for subsequent processing steps; the data volume of different dimensions of features in the samples is inconsistent, which may cause the dimensionality reduction algorithm to deviate from the optimal value; therefore, this invention converts the string-type features in the data samples into numerical features and applies Z-Score standardization to each feature dimension, making the features of different dimensions more comparable in numerical terms, accelerating the convergence speed, and improving the classification accuracy.

[0016] Regarding step S2: After preprocessing the dataset, its string representation features are expanded to form high-dimensional sparse features, which is not conducive to the training of some nonlinear models and will lead to extremely low training efficiency and easy overfitting. To this end, the present invention uses the PCA algorithm to reduce the dimensionality of the preprocessed dataset, extract the main feature components of the data, maximize the preservation of the intrinsic information of the data, and reduce the computational overhead.

[0017] Regarding step S3: Intrusion detection datasets typically have a large amount of data, and data noise is unavoidable. The presence of noise points has an adverse effect on subsequent data expansion, increases calculation errors, and also affects data convergence speed and model accuracy. Therefore, this invention considers using the OPTICS clustering algorithm to achieve the noise reduction function. This algorithm clusters data according to the density of each attack class, and based on the obtained decision graph, reasonably sets the neighborhood radius threshold to remove outliers in each category.

[0018] Regarding step S4: In existing algorithms for augmenting imbalanced datasets, traditional algorithms use the SMOTE algorithm and its derivatives to synthesize minority class samples through random linear interpolation. However, these algorithms have many limitations. For large, complex, mixed datasets, existing research has shown that using deep neural networks to generate samples can achieve better results. Therefore, this invention considers using CVAE-GAN to augment the minority attack class in the dataset. This algorithm combines the advantages of VAE and GAN, and can generate fine-grained simulated samples through attribute deformation. At the same time, it uses sample label information as a guide to achieve better data generation.

[0019] Furthermore, the specific operation process of preprocessing in step S1 is as follows:

[0020] A1: Convert the string-type feature values ​​in each sample of the dataset into numerical features using one-hot encoding;

[0021] A2: For the transformed sample features, the Z-Score standardization algorithm is used to transform each dimension of the sample features into a standard normal distribution, i.e., the mean is 0 and the standard deviation is 1. The Z-Score standardization calculation formula is as follows:

[0022]

[0023] Where x is the value of each feature in the original dataset, μ is the mean of each feature, and σ is the standard deviation of each feature.

[0024] Furthermore, the specific process of dimensionality reduction in step S2 is as follows:

[0025] B1: For the obtained standardized dataset features X = [x1, x2, x3, ..., x...] n ], where x i =[x 1i ,x 2i ,x 3i ,...,x mi ] T Where m is the number of samples and n is the feature dimension, the covariance matrix of matrix X is calculated using the following formula:

[0026] C = E(XX) T (2)

[0027] B2: Find the eigenvalues ​​and corresponding eigenvectors of the covariance matrix C;

[0028] B3: Arrange the eigenvectors into a matrix from top to bottom according to the size of their corresponding eigenvalues, and take the first k rows to form a vector matrix P (keeping the k largest eigenvectors);

[0029] B4: Obtain the feature matrix after dimensionality reduction to k dimensions using the formula Y = PX.

[0030] Furthermore, the specific process of step S3 is as follows:

[0031] C1: Define an ordered queue Q and a result queue R;

[0032] C2: If all the data in Y has been processed, proceed to step C7; otherwise, randomly select an unprocessed sample that is a core object from Y, put the sample into queue R, and put its directly reachable points into queue Q in ascending order of reachability distance.

[0033] C3: If queue Q is empty, go to step C2; otherwise, take one sample point from it in sequence.

[0034] C4: Determine whether the sample is a core object. If not, proceed to step C3; otherwise, proceed to step C5.

[0035] C5: Place the sample in queue R, find all its directly density-reachable samples and place them in queue Q. If the sample is already in an ordered queue and the new reachability distance is small, update the reachability distance of the sample and reorder queue Q.

[0036] C6: Proceed to step C3 until all data in Y has been processed, and obtain the sample order and reachability distance in queue R;

[0037] C7: Draw a decision graph according to the sample order and reachability distance in queue R, and set a threshold ∈ i ,∈ i In R, the core distance between samples in [0, ∈] is greater than ∈ i These are then considered noise samples;

[0038] C8: Filter to obtain each dataset after removing noise samples.

[0039] Furthermore, the relevant definitions recorded in steps C1 to C8 are as follows:

[0040] Network input parameters: ∈ — ∈-neighborhood radius, MinPts — minimum number of samples required for a given sample to become a core object in the ∈-neighborhood; the processing sample set is Y;

[0041] For a given sample p, q ∈ Y, and parameters ∈ MinPts, the relevant concepts of clustering algorithms are:

[0042] ∈-neighborhood: The ∈-neighborhood of sample p represents a subset of samples in the sample set Y whose distance from p is no greater than ∈. The number of samples contained in the subset is denoted as |N|. ∈ (p)|;

[0043] Core point: If the ∈-neighborhood of sample p contains at least MinPts sample points (including p), i.e., |N ∈ If (p)|≥MinPts, then sample p is called the core object;

[0044] Directly density-reachable: If samples p and q satisfy q∈|N ∈ (p)|and|N ∈ If (p)|≥MinPts, then sample q is said to be directly density-reachable from sample p.

[0045] Core distance: The smallest neighborhood radius that makes sample p a core object is called the core distance of p, i.e.:

[0046]

[0047] in, For set N ∈ (p) is the sample point that is the i-th nearest neighbor of sample p;

[0048] Reachability distance: For a core sample p, the reachability distance from sample q to p is the maximum of the core distance of sample p and the actual distance between p and q, that is:

[0049]

[0050] Furthermore, the CVAE-GAN model in step S4 includes the following four parts:

[0051] Encoder network E: maps sample x to latent vector z by learning distribution P(z|x,c);

[0052] The generator network G generates samples x' by sampling from the distribution P(z|x,c) and learns the real data distribution through gradients passed from other parts of the model.

[0053] Discriminator Network D: Learns to distinguish between "real" and "fake" samples;

[0054] Classifier network C: measures the posterior probability P(c|x).

[0055] Furthermore, the specific process of step S4 is as follows:

[0056] D1: The processed training set is redefined as X, and the one-dimensional sequence samples in it are converted into two-dimensional image structure samples x;

[0057] D2: Divide the training set samples into batches of appropriate size, and execute the following steps for each batch:

[0058] D2-1: Train the encoder network E of the CVAE-GAN model using sample x and its label c, and output the latent vector z;

[0059] D2-2: Input the label c and the latent vector z into the generator network G to generate samples, and output the generated sample x'.

[0060] D2-3: Input the original sample x and the generated sample x' into the discriminator network D, and output the discrimination result;

[0061] D2-4: Input the original sample x and the generated sample x' into the classifier network C, and output the classification result;

[0062] D3: Repeat step 4.2 to perform iterative training. Update the network parameters of each part of the CVAE-GAN model until the outputs of the discriminator network D and the classifier network C meet the set threshold, the training parameters of the generator G converge, and the training is completed.

[0063] D4: Use the generator network G to generate the required number of generated samples for the minority class in the dataset, and mix them with the original samples to obtain a more balanced dataset.

[0064] Furthermore, in step S5, a CNN network is used as the classifier. The structure of the CNN network is as follows: the first layer is a convolutional layer with 128 3×3 convolutional kernels; the second layer is a max pooling layer with a pooling window of 2×2; the third layer is a dropout layer with a parameter of 0.5; the fourth layer is a fully connected layer with 1024 neurons; and the softmax activation function is used to achieve classification.

[0065] This invention first preprocesses the intrusion detection dataset; then, it applies Principal Component Analysis (PCA) to reduce the dimensionality of the preprocessed dataset; next, it uses an Ordering Points to Identify the Clustering Structure (OPTICS) clustering algorithm to remove outliers (noise points) from the dimensionality-reduced data; finally, it uses a Conditional Variational Autoencoder-Generative Adversarial Nets (CVAE-GAN) to augment the minority attack classes in the dataset; and then mixes the augmented data with the original data to obtain a relatively balanced intrusion detection dataset, which is used to train a classifier.

[0066] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0067] 1. This invention uses the PCA algorithm for dimensionality reduction, which is easy to operate, effectively extracts the main feature components of the data, maximizes the preservation of the data's intrinsic information, and reduces computational overhead, making it easier for subsequent program processing.

[0068] 2. This invention first uses the OPTICS clustering algorithm to remove noise and ensure the accuracy of the dataset, and then performs data augmentation to make the processed dataset more similar to the true distribution.

[0069] 3. The CVAE-GAN model used in this invention can generate fine-grained images under the same category by utilizing its "attribute deformation" function, and it has good performance in data generation. Attached Figure Description

[0070] Figure 1 This is a flowchart of the method of the present invention;

[0071] Figure 2 This is a schematic diagram of the CVAE-GAN model.

[0072] Figure 3 This is a schematic diagram of the structure of a CNN classifier. Detailed Implementation

[0073] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that these embodiments are for illustrative purposes only and are not intended to limit the scope of the invention. After reading this invention, any modifications of the invention in various equivalent forms by those skilled in the art will fall within the scope defined by the appended claims.

[0074] This invention provides a method for processing imbalanced datasets for intrusion detection based on CVAE-GAN, such as... Figure 1 As shown, it includes the following steps:

[0075] Step 1: Preprocess the intrusion detection dataset:

[0076] The samples in the original dataset may contain string-type feature values, which cannot be used for subsequent processing steps; the data volume of different dimensions of features in the samples is inconsistent, which may cause the dimensionality reduction algorithm to deviate from the optimal value. To address this, this embodiment converts each string-type feature of the data samples into a numerical feature in One-Hot encoding form, and performs Z-Score standardization on each sample dimension, making the features of different dimensions more numerically comparable, accelerating the convergence speed, and improving the classification accuracy.

[0077] In this embodiment, the KDD CUP 99 dataset is preprocessed. The specific operation process is as follows:

[0078] Step 1.1: The dataset has a total of 41 features, including 3 string features: flag, service, and protocol_type. Each feature contains multiple subclasses. Each feature is one-hot encoded to obtain a 122-dimensional sparse encoded feature set.

[0079] Step 1.2: For the transformed features, use the Z-Score standardization algorithm to convert each feature dimension of the sample into a standard normal distribution, i.e., with a mean of 0 and a standard deviation of 1. The Z-Score standardization algorithm calculation formula is as follows:

[0080]

[0081] Where x is the value of each feature in the original dataset, μ is the mean of each feature, and σ is the standard deviation of each feature.

[0082] Step 2: Perform dimensionality reduction on the preprocessed dataset using the PCA algorithm:

[0083] After preprocessing the dataset, the string representations of its features are expanded, forming high-dimensional sparse features, which is detrimental to the training of certain nonlinear models, leading to extremely low training efficiency and a high risk of overfitting. Therefore, this embodiment uses the PCA algorithm to reduce the dimensionality of the preprocessed dataset, extracting the main feature components to maximize the preservation of the data's intrinsic information while reducing computational overhead.

[0084] In this embodiment, considering both computational overhead and classification accuracy, the 122-dimensional features are reduced to 49 dimensions. The specific operation process is as follows:

[0085] Step 2.1: Apply the features X = [x1, x2, x3, ..., x] of the standardized dataset obtained in Step 1. m ] T , where x i =[x i1 ,x i2 ,x i3 ,...,x in [m is the number of samples, n is the feature dimension, and here n is 122.] The covariance matrix of matrix X is calculated using the following formula:

[0086] C = E(XX) T (6)

[0087] Step 2.2: Find the eigenvalues ​​and corresponding eigenvectors of the covariance matrix C;

[0088] Step 2.3: Arrange the feature vectors into a matrix from top to bottom according to the size of their corresponding feature values, and take the first k rows to form a vector matrix P (retain the k largest feature vectors, k is 49 in this embodiment);

[0089] Step 2.4: Obtain the feature matrix after dimensionality reduction to 49 dimensions using the formula Y = PX.

[0090] Step 3: Apply the OPTICS clustering algorithm to the dimensionality-reduced data to remove outliers (noise points):

[0091] Intrusion detection datasets typically have large volumes of data, and data noise is unavoidable. The presence of noisy points negatively impacts subsequent data expansion, increases computational errors, and affects data convergence speed and model accuracy. Therefore, this embodiment considers using the OPTICS clustering algorithm to achieve noise reduction, minimizing interference from noisy data in the majority classes. This algorithm clusters data based on the density of each attack class, and then, based on the resulting decision graph, appropriately sets a neighborhood radius threshold to remove outliers from each class.

[0092] Specifically, the parameters required by the OPTICS algorithm are: ∈—the radius of the ∈-neighborhood, and MinPts—the minimum number of samples required for a given sample to become a core object in the ∈-neighborhood. The relevant parts of this algorithm are defined as follows:

[0093] ∈-neighborhood: The ∈-neighborhood of sample p represents a subset of samples in the sample set Y whose distance from p is no greater than ∈. The number of samples contained in the subset is denoted as |N|. ∈ (p)|;

[0094] Core point: If the ∈-neighborhood of sample p contains at least MinPts sample points (including p), i.e., |N ∈ If (p)|≥MinPts, then sample p is called the core object;

[0095] Directly density-reachable: If samples p and q satisfy q∈|n ∈ (p)|and|n ∈ If (p)|≥MinPts, then sample q is said to be directly density-reachable from sample p.

[0096] Core distance: The smallest neighborhood radius that makes sample p a core object is called the core distance of p, i.e.:

[0097]

[0098] in, For set N ∈ (p) is the sample point that is the i-th nearest neighbor of sample p;

[0099] Reachability distance: For a core sample p, the reachability distance from sample q to p is the maximum of the core distance of sample p and the actual distance between p and q, that is:

[0100]

[0101] In this embodiment, the majority classes Normal, Probe, and DOS in the preprocessed dataset are processed separately, removing approximately 5% of noise and samples with indistinguishable overlapping regions. The other two classes, due to insufficient data, retain all samples for augmentation. The specific operation process of this step is as follows:

[0102] Step 3.1: Define an ordered queue Q and a result queue R;

[0103] Step 3.2: If all data in Y has been processed, proceed to step 3.7; otherwise, randomly select unprocessed samples that are core objects from Y and put them into queue R. The directly reachable points are sorted in ascending order of reachability distance and put into queue Q.

[0104] Step 3.3: If queue Q is empty, go to step 3.2; otherwise, take one sample point from it in sequence.

[0105] Step 3.4: Determine whether the sample is a core object. If not, proceed to step 3.3; otherwise, proceed to step 3.5.

[0106] Step 3.5: Place the sample into queue R, find all its directly density-reachable samples and place them into queue Q. If the sample is already in an ordered queue and the new reachability distance is small, update the reachability distance of the sample and reorder queue Q.

[0107] Step 3.6: Proceed to Step 3.3 until all data in Y has been processed, and obtain the sample order and reachability distance in queue R;

[0108] Step 3.7: Draw a decision graph according to the sample order and reachability distance in queue R, and set a threshold ∈ i ,∈ i In R, the core distance between samples in [0, ∈] is greater than ∈ i These are then considered noise samples;

[0109] Step 3.8: Filter to obtain each dataset after removing noise samples.

[0110] OPTICS is a density-based clustering algorithm, an improved version of DBSCAN, which reduces its sensitivity to the parameter ∈. The core of the OPTICS algorithm is to obtain a result queue R containing the processing order, reachability distance, etc., as shown in Algorithm 1.

[0111]

[0112]

[0113] Step 4: Apply the constructed CVAE-GAN model to expand the dataset to include a few attack classes:

[0114] In existing algorithms for augmenting imbalanced datasets, traditional methods employ the SMOTE algorithm and its derivatives, synthesizing minority class samples through random linear interpolation. However, these algorithms have several limitations. For large, complex, mixed datasets, existing research has demonstrated that using deep neural networks to generate samples yields better results. Therefore, this embodiment considers using the CVAE-GAN model to augment the minority attack class in the dataset. This algorithm combines the advantages of VAE and GAN, generating fine-grained simulated samples through attribute deformation, while using sample label information as a guide to achieve better data generation.

[0115] For details, see attached. Figure 2 As shown, the CVAE-GAN model consists of four parts: an encoder network E, which maps samples x to latent vector z by learning the distribution P(z|x,c); a generator network G, which generates samples x' by sampling from the distribution P(z|x,c) and learns the real data distribution through gradients passed from other parts of the model; a discriminator network D, which learns to distinguish between "real" and "fake" samples; and a classifier network C, which measures the posterior probability P(c|x). Algorithm 2 illustrates the training process of CVAE-GAN.

[0116]

[0117]

[0118] In this embodiment, the hidden vector z is set to 20 dimensions. The network parameters for each part are set according to the original CVAE-GAN paper, and the remaining parameters are adaptively modified. The specific operation process of this step is as follows:

[0119] Step 4.1: Expand the U2R and R2L classes in the training set by converting their one-dimensional sequence samples into two-dimensional image structure samples x of size 7×7.

[0120] Step 4.2: Divide the training set samples into batches of appropriate size. Here, the batch size is set to 200. For each batch, perform the following:

[0121] Step 4.2.1: Train the encoder network E of the CVAE-GAN model using sample x and its label c, and output the latent vector z;

[0122] Step 4.2.2: Input the label c and the latent vector z into the generator network G to generate samples, and output the generated sample x';

[0123] Step 4.2.3: Input the original sample x and the generated sample x' into the discriminator network D, and output the discrimination result;

[0124] Step 4.2.4: Input the original sample x and the generated sample x' into the classifier network C, and output the classification result;

[0125] Step 4.3: Repeat step 4.2 for iterative training. Update the network parameters of each part of the CVAE-GAN model until the outputs of the discriminator network D and the classifier network C meet the set threshold, the training parameters of the generator G converge, and the training is complete.

[0126] Step 4.4: Use the generator network G to generate the required number of generated samples for the minority class in the dataset, and mix them with the original samples to obtain a more balanced dataset.

[0127] CVAE-GAN keeps the training objectives of network E, D, and C unchanged, as shown in formulas (9)-(11):

[0128]

[0129] in, Let μ be the loss function of network E, and μ and ∈ be the mean and covariance of the latent vectors output by the encoder network.

[0130]

[0131] in, Let P be the loss function of network D; x be the data sample; P be the loss function of network D. r Let P be the prior distribution of sample x; z Let z be the prior distribution of the latent variable z output by E;

[0132]

[0133] in, Let C be the loss function of network C;

[0134] The loss function of the generator network G is modified to measure whether the generated sample x' is similar to:

[0135] Original sample x:

[0136]

[0137] Other samples within the same category:

[0138]

[0139] And, real samples:

[0140]

[0141] The final loss function of CVAE-GAN is:

[0142]

[0143] Where λ i This is the weighting factor.

[0144] Step 5: Mix the expanded data with the original data to obtain a relatively balanced intrusion detection dataset, train the classifier, and test the classification results.

[0145] The focus of this invention is to propose a method for processing datasets; the classifier is only used to demonstrate the performance of this method. (See attached diagram.) Figure 3 As shown, this embodiment uses a single-layer convolutional CNN network as the classifier. Specifically, the classifier structure is as follows: the first layer is a convolutional layer with 128 3×3 convolutional kernels; the second layer is a max pooling layer with a pooling window of 2×2; the third layer is a dropout layer with a parameter of 0.5; the fourth layer is a fully connected layer with 1024 neurons; and the softmax activation function is used to achieve classification.

Claims

1. A method for processing imbalanced datasets for intrusion detection based on CVAE-GAN, characterized in that, Includes the following steps: S1: Preprocess the intrusion detection dataset; S2: Use the PCA algorithm to reduce the dimensionality of the preprocessed dataset; S3: Use the OPTICS clustering algorithm on the dimensionality-reduced data to remove outliers; S4: Apply the constructed CVAE-GAN model to expand the few attack classes in the dataset; S5: Mix the expanded data with the original data to obtain a relatively balanced intrusion detection dataset, train the classifier, and test the classification results.

2. The method for processing imbalanced datasets for intrusion detection based on CVAE-GAN according to claim 1, characterized in that, The specific preprocessing steps in step S1 are as follows: A1: Convert the string-type feature values ​​in each sample of the dataset into numerical features using one-hot encoding; A2: For the transformed sample features, the Z-Score standardization algorithm is used to transform each dimension of the sample features into a standard normal distribution, i.e., the mean is 0 and the standard deviation is 1. The Z-Score standardization calculation formula is as follows: Where x is the value of each feature in the original dataset, μ is the mean of each feature, and σ is the standard deviation of each feature.

3. The method for processing imbalanced datasets for intrusion detection based on CVAE-GAN according to claim 1, characterized in that, The specific process of dimensionality reduction in step S2 is as follows: B1: For the obtained standardized dataset features X = [x1, x2, x3, ..., x...] n ], where x i =[x 1i x 2i x 3i , ..., x mi ] T Where m is the number of samples and n is the feature dimension, the covariance matrix of matrix X is calculated using the following formula: C=E(XX T ) (2) B2: Find the eigenvalues ​​and corresponding eigenvectors of the covariance matrix C; B3: Arrange the eigenvectors into a matrix from top to bottom according to the size of their corresponding eigenvalues, and take the first k rows to form a vector matrix P; B4: Obtain the feature matrix after dimensionality reduction to k dimensions using the formula Y = PX.

4. The method for processing imbalanced datasets for intrusion detection based on CVAE-GAN according to claim 1, characterized in that, The specific process of step S3 is as follows: C1: Define an ordered queue Q and a result queue R; C2: If all the data in Y has been processed, proceed to step C7; otherwise, randomly select an unprocessed sample that is a core object from Y, put the sample into queue R, and put its directly reachable points into queue Q in ascending order of reachability distance. C3: If queue Q is empty, go to step C2; otherwise, take one sample point from it in sequence. C4: Determine whether the sample is a core object. If not, proceed to step C3; otherwise, proceed to step C5. C5: Place the sample in queue R, find all its directly density-reachable samples and place them in queue Q. If the sample is already in an ordered queue and the new reachability distance is small, update the reachability distance of the sample and reorder queue Q. C6: Proceed to step C3 until all data in Y has been processed, and obtain the sample order and reachability distance in queue R; C7: Draw a decision graph according to the sample order and reachability distance in queue R, and set a threshold ∈ i ,∈ i In R, the core distance between samples in [0, ∈] is greater than ∈ i These are then considered noise samples; C8: Filter to obtain each dataset after removing noise samples.

5. The method for processing imbalanced datasets for intrusion detection based on CVAE-GAN according to claim 4, characterized in that, The relevant definitions recorded in steps C1 to C8 are as follows: Network input parameters: ∈ — ∈-neighborhood radius, MinPts — minimum number of samples required for a given sample to become a core object in the ∈-neighborhood; the processing sample set is Y; For a given sample p, q∈Y, parameter ∈, MinPts, the relevant concepts of clustering algorithms are: ∈-neighborhood: The ∈-neighborhood of sample p represents a subset of samples in the sample set Y whose distance from p is no greater than ∈. The number of samples contained in the subset is denoted as |N|. ∈ (p)|; Core object: If the ∈-neighborhood of sample p contains at least MinPts sample points, i.e., |N ∈ If (p)|≥MinPts, then sample p is called the core object; Direct density attainability: If samples p and q satisfy q∈|N ∈ (p)|and|N ∈ If (p)|≥MinPts, then sample q is said to be directly density-reachable from sample p. Core distance: The smallest neighborhood radius that makes sample p a core object is called the core distance of p, i.e.: in, For set N ∈ (p) is the sample point that is the i-th nearest neighbor of sample p; Reachability distance: For a core sample p, the reachability distance from sample q to p is the maximum of the core distance of sample p and the actual distance between p and q, that is:

6. The method for processing imbalanced datasets for intrusion detection based on CVAE-GAN according to claim 1, characterized in that, The CVAE-GAN model in step S4 includes the following four parts: Encoder network E: maps sample x to latent vector z by learning distribution P(z|x, c); The generator network G generates samples x' by sampling from the distribution P(z|x,c) and learns the real data distribution through gradients passed from other parts of the model; Discriminator Network D: Learns to distinguish between "real" and "fake" samples; Classifier network C: posterior probability of measurement P(c|x).

7. The method for processing imbalanced datasets for intrusion detection based on CVAE-GAN according to claim 6, characterized in that, The specific process of step S4 is as follows: D1: The processed training set is redefined as X, and the one-dimensional sequence samples in it are converted into two-dimensional image structure samples x; D2: Divide the training set samples into batches of appropriate size, and execute the following steps for each batch: D2-1: Train the encoder network E of the CVAE-GAN model using sample x and its label c, and output the latent vector z; D2-2: Input the label c and the latent vector z into the generator network G to generate samples, and output the generated sample x'. D2-3: Input the original sample x and the generated sample x' into the discriminator network D, and output the discrimination result; D2-4: Input the original sample x and the generated sample x' into the classifier network C, and output the classification result; D3: Repeat step 4.2 to perform iterative training. Update the network parameters of each part of the CVAE-GAN model until the outputs of the discriminator network D and the classifier network C meet the set threshold, the training parameters of the generator G converge, and the training is completed. D4: Use the generator network G to generate the required number of generated samples for the minority class in the dataset, and mix them with the original samples to obtain a more balanced dataset.

8. The method for processing imbalanced datasets for intrusion detection based on CVAE-GAN according to claim 1, characterized in that, In step S5, a CNN network is used as a classifier. The structure of the CNN network is as follows: the first layer is a convolutional layer with 128 3×3 convolutional kernels; the second layer is a max pooling layer with a pooling window of 2×2; the third layer is a dropout layer with a parameter of 0.5; the fourth layer is a fully connected layer with 1024 neurons; and the softmax activation function is used to achieve classification.

9. The method for processing imbalanced datasets for intrusion detection based on CVAE-GAN according to claim 6, characterized in that, The training method for the CVAE-GAN model in step S5 is as follows: The CVAE-GAN model keeps the training objectives of network E, D, and C unchanged, which are respectively formulas (5)-(7). The specific training process is as follows: in, Let μ be the loss function of network E, and μ and ∈ be the mean and covariance of the latent vectors output by the encoder network. in, Let be the loss function of network D; x be the data sample; Pr be the prior distribution of sample x; Pz be the prior distribution of the latent variable z of the output of E; in, Let C be the loss function of network C; The loss function of the generator network G is modified to measure whether the generated sample x' is similar to the original sample x: Other samples within the same category: And, real samples: The final loss function of CVAE-GAN is: Where λ i This is the weighting factor.

Images