A pollution immunity-based abnormal data detection method and system
By constructing a bidirectional generative adversarial network and utilizing improved discriminator and generator optimizations, the problem of decreased detection accuracy caused by data contamination was solved, achieving efficient anomaly detection in contaminated datasets.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SUN YAT SEN UNIV
- Filing Date
- 2023-07-12
- Publication Date
- 2026-05-29
AI Technical Summary
Existing unsupervised and semi-supervised anomaly detection methods suffer from a sharp drop in detection accuracy when normal datasets are contaminated, making it difficult to effectively identify anomalous data.
A bidirectional generative adversarial network is constructed. By training a discriminator and a generator, and utilizing an improved discriminator objective function and joint optimization of the generator and encoder, the generator generates clean, normal data, and the encoder extracts hidden layer representations for discrimination. Nash equilibrium is used to achieve a probability distribution that results in clean data.
While eliminating data contamination, it improves the accuracy and performance of abnormal data detection, and effectively distinguishes between normal and abnormal data through the scoring criteria represented by the hidden layer.
Smart Images
Figure CN117093937B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of anomaly detection, and in particular to an anomaly data detection method and system based on contamination immunity. Background Technology
[0002] Anomaly detection technology has wide applications in various scenarios, including medical diagnosis, cybersecurity, and financial fraud. The goal of anomaly detection is to identify anomalous data that is significantly different from most samples or defined normal data. For example, in medical diagnosis, image anomaly detection technology can help doctors diagnose medical lesions such as skin and eye diseases more efficiently; time-series anomaly detection technology can be used in medical settings to monitor patients' heart rates and provide timely warnings of abnormalities; and in cybersecurity, sequence anomaly detection technology can be used for traffic monitoring, providing security for numerous systems in the current big data era.
[0003] Current mainstream unsupervised anomaly detection methods, such as variational autoencoders and probabilistic generative models using generative adversarial networks, learn the probability distribution of normal datasets and use the probability density value or its alternative criteria (such as reconstruction error) as the detection criterion for judging test points. There are also some semi-supervised anomaly detection algorithms, such as DeepSAD. DeepSAD works by compressing normal data into a hyperplane while simultaneously moving anomalous data away from the center of the hyperplane, thus filtering out anomalous data. Both the aforementioned unsupervised and semi-supervised anomaly detection algorithms have achieved competitive results. However, these methods all rely on the assumption that the normal dataset is not contaminated by anomalous data. Once the normal dataset is contaminated, the detection accuracy of these methods drops sharply.
[0004] In practical applications, it is almost impossible to obtain a completely pure dataset. The datasets collected in reality are often affected by noise and other factors, which contaminate the dataset and negatively impact the detection of abnormal data. Therefore, how to efficiently detect abnormal data while eliminating the influence of contamination is a technical problem that urgently needs to be solved in this field. Summary of the Invention
[0005] To address the issue that existing technologies are susceptible to contamination when detecting abnormal data, leading to a decrease in detection accuracy, this invention proposes an abnormal data detection method and system based on contamination immunity. By utilizing a bidirectional generative adversarial network to discriminate the input data, a clean probability distribution of abnormal data is obtained, ultimately ensuring the effectiveness and performance of abnormal data detection while eliminating abnormal data contamination.
[0006] According to a first aspect of the present invention, a method for detecting abnormal data based on contamination immunity is provided, comprising:
[0007] S1. Divide the dataset into a training set, an auxiliary training set, and a test set. The data in the training set and the test set include normal data and contaminated abnormal data. The data in the auxiliary training set contains only contaminated abnormal data. Normal human eye CT images in the OCT dataset are used as normal data.
[0008] S2. Construct a bidirectional generative adversarial network, which includes an encoder, a generator, and a discriminator;
[0009] S3. Input the training set and auxiliary training set into the bidirectional generative adversarial network, train the discriminator in the bidirectional generative adversarial network, and obtain the trained discriminator;
[0010] S4. Input the training set and auxiliary training set into the bidirectional generative adversarial network, and use the trained discriminator to train the generator and encoder in the bidirectional generative adversarial network to obtain the trained generator and encoder.
[0011] S5. Repeat steps S3-S4 until the bidirectional generative adversarial network reaches Nash equilibrium, then end the training and obtain the trained bidirectional generative adversarial network, which includes the optimal discriminator and the optimal generator.
[0012] S6. Input the test set into the trained bidirectional generative adversarial network, use the trained bidirectional generative adversarial network to detect contaminated and abnormal data in the test set, and calculate the performance of the bidirectional generative adversarial network based on the detection results.
[0013] S7. Use the trained bidirectional generative adversarial network for anomaly detection.
[0014] Preferably, in step S3,
[0015] S31. Input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input to the bidirectional generative adversarial network. Input x into encoder E. Encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... ,Will Input to encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation of the auxiliary training set data. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator, which restores the hidden layer representation z to a high-dimensional space to obtain the generated data. ;
[0016] S32. Construct the objective function of the discriminator D. The discriminator is used to identify whether the input data is contaminated or abnormal data. The expression is:
[0017] ,
[0018] in, The joint probability distribution of the training set, To assist in the joint probability distribution of the training set, To generate data The joint probability distribution of , where d is a parameter, is calculated as follows:
[0019] ,
[0020] in, To train the proportion of contaminated and anomalous data among all data;
[0021] S33. Establish sample pairs , and , sample pair , and Substitute into the discriminator objective function;
[0022] S34. Use gradient descent to train the discriminator in the bidirectional generative adversarial network, update the discriminator parameters, and obtain the trained discriminator.
[0023] Based on the above technical means, the encoder can be used to generate the hidden layer representation of the data, and the hidden layer representation can be input into the discriminator. The hidden layer representation can extract the features of high-dimensional spatial information, which is beneficial for the discriminator to make judgments.
[0024] Preferably, step S4 includes:
[0025] S41. Input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input to the bidirectional generative adversarial network. Input x into the encoder. The encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... , will assist training set data Input to encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator to obtain the generated data. ;
[0026] S42. Construct the objective functions for generator G and encoder E, with the following expressions:
[0027] ,
[0028] in, This represents the trained discriminator, where c is the parameter;
[0029] S43. Establish sample pairs , and , sample pair , and Substitute the objective functions of the generator and encoder;
[0030] S44. Use gradient descent to train the generator and encoder in the bidirectional generative adversarial network, update the parameters of the generator and encoder, and obtain the trained generator and encoder.
[0031] Based on the above technical means, a trained discriminator is introduced into the objective function of the generator and encoder, thereby jointly optimizing the discriminator with the generator and encoder.
[0032] Preferably, in the trained bidirectional generative adversarial network, the optimal discriminator The expression is:
[0033] ,
[0034] The probability distribution expression for the optimal generator G is:
[0035] ,
[0036] in, Let be the expected joint probability distribution.
[0037] Based on the above technical means, the optimal discriminator was enhanced by adding... Anomaly detection terms are used to constrain the assignment of low probability values to contaminated and abnormal samples, thereby detecting novel anomalies in the training set.
[0038] Preferably, in step S6, the step of detecting abnormal data in the test set includes:
[0039] S611. Input the test set data into the trained bidirectional generative adversarial network, and use the encoder in the trained bidirectional generative adversarial network to calculate the hidden layer representation of the test set data. ;
[0040] S612. Calculate each data point in the test set. Abnormal score ;
[0041] S613. Set a scoring threshold, and compare the abnormal score value of each data in the test set with the scoring threshold one by one. When the abnormal score value is greater than the scoring threshold, the data is judged as contaminated abnormal data. When the abnormal score value is less than the scoring threshold, the data is judged as normal data. After judging all the data in the test set, the judgment result is obtained.
[0042] Based on the above technical means, the hidden layer representation of the test set samples is calculated by the encoder, and the features of the test set samples are extracted through the hidden layer representation. Then, the test set samples are determined to be contaminated abnormal data according to the scoring threshold.
[0043] Preferably, in step S6, the anomaly score value The calculation expression is:
[0044] .
[0045] Preferably, in step S6, the step of calculating the performance of the bidirectional generative adversarial network includes:
[0046] S621. Calculate the precision P of the judgment result: calculate the ratio of the number of data correctly predicted as pollution anomalies to the number of data predicted as pollution anomalies in the judgment result;
[0047] S622. Calculate the recall rate R of the judgment results: Calculate the ratio of the number of correctly predicted pollution anomaly data in the judgment results to the number of actual pollution anomaly data in the test set;
[0048] S623. Calculate the F1 score of the judgment result. The calculation expression is as follows:
[0049] ,
[0050] The F1 score is used as the performance evaluation standard for the trained bidirectional generative adversarial network. The higher the F1 score, the better the performance of the bidirectional generative adversarial network.
[0051] Based on the above technical means, the recall and precision of the judgment results are calculated, and the performance is evaluated using the harmonic mean F1 score of the recall and precision. This allows for an accurate performance evaluation standard even when there is an imbalance between positive and negative samples.
[0052] According to another aspect of the present invention, an abnormal data detection system based on contamination immunity is provided, comprising:
[0053] The data partitioning module is used to partition the dataset to obtain a training set, an auxiliary training set, and a test set. The data in the training set and the test set include normal data and polluted abnormal data, while the data in the auxiliary training set contains only polluted abnormal data.
[0054] A network construction module is used to construct a bidirectional generative adversarial network, which includes an encoder, a generator, and a discriminator.
[0055] The discriminant training module is used to input the training set and auxiliary training set into the bidirectional generative adversarial network and train the discriminator in the bidirectional generative adversarial network to obtain the trained discriminator.
[0056] The generation training module is used to input the training set and auxiliary training set into the bidirectional generative adversarial network, and to train the generator and encoder in the bidirectional generative adversarial network using the trained discriminator to obtain the trained generator and encoder.
[0057] The network training module is used to repeatedly execute the discriminant training module and the generator training module until the bidirectional generative adversarial network reaches Nash equilibrium, then the training ends and a trained bidirectional generative adversarial network is obtained. The trained bidirectional generative adversarial network includes the optimal discriminator and the optimal generator.
[0058] The performance testing module is used to input a test set into the trained bidirectional generative adversarial network (BGAN), use the trained BGAN to detect contaminated and abnormal data in the test set, and calculate the performance of the BGAN based on the detection results.
[0059] The data detection module is used to detect abnormal data using the trained bidirectional generative adversarial network.
[0060] Preferably, the discrimination training module includes:
[0061] The first representation generation unit is used to input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input into the bidirectional generative adversarial network. x is input into encoder E, and encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... ,Will Input to encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation of the auxiliary training set data. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator G, which restores the hidden layer representation z to a high-dimensional space to obtain the generated data. ;
[0062] The first function construction unit is used to construct the objective function of the discriminator D. The discriminator D is used to identify whether the input data is contaminated or abnormal data, and its expression is:
[0063] ,
[0064] in, The joint probability distribution of the training set, To assist in the joint probability distribution of the training set, To generate data The joint probability distribution of , where d is a parameter, is calculated as follows:
[0065] ,
[0066] in, To train the proportion of contaminated and anomalous data among all data;
[0067] The first function calculation unit is used to establish sample pairs. , and and sample pairs , and Substitute into the discriminator objective function;
[0068] The first gradient training unit is used to train the discriminator in the bidirectional generative adversarial network using gradient descent, update the discriminator parameters, and obtain the trained discriminator.
[0069] Preferably, the training generation module includes:
[0070] The second representation generation unit is used to input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input into the bidirectional generative adversarial network. x is input into encoder E, and encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... ,Will Input encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator to obtain the generated data. ;
[0071] The second function construction unit constructs the objective functions for the generator and encoder, with the following expression:
[0072] ,
[0073] in, This represents the trained discriminator, where c is the parameter;
[0074] The second function calculation unit is used to establish sample pairs. , and , sample pair , and Substitute the objective functions of the generator and encoder;
[0075] The second gradient training unit is used to train the generator and encoder in the bidirectional generative adversarial network using gradient descent, update the parameters of the generator and encoder, and obtain the trained generator and encoder.
[0076] Compared to existing technologies, this invention proposes a method and system for anomaly detection based on contamination immunity. It constructs a bidirectional generative adversarial neural network (GAN) that differs from traditional GANs, and utilizes an improved discriminator objective function to enable it to distinguish data categories. The improved discriminator prevents the generator from generating contaminated data, ensuring only normal data is generated. The model ultimately uses the L2 norm of the implicit representation as a scoring criterion, guaranteeing that normal data receives a lower score while anomaly data receives a significantly higher score. Compared to existing unsupervised and semi-supervised methods, this method fully considers the potential contamination data in the training set and establishes corresponding mechanisms to model it. Ultimately, the model can obtain a completely pure probability distribution from the contaminated dataset, thus efficiently and accurately detecting anomalies while eliminating contamination. Attached Figure Description
[0077] Figure 1 This is a schematic diagram illustrating the steps of the abnormal data detection method based on pollution immunity proposed in this embodiment of the invention.
[0078] Figure 2 This diagram illustrates the bidirectional generative adversarial network structure proposed in this embodiment of the invention.
[0079] Figure 3 This diagram illustrates the abnormal data detection system based on contamination immunity proposed in this embodiment of the invention.
[0080] Figure 4 This diagram illustrates the discrimination training module proposed in this embodiment of the invention.
[0081] Figure 5 This diagram illustrates the generation training module proposed in this embodiment of the invention. Detailed Implementation
[0082] This invention provides an abnormal data detection method and system based on contamination immunity, which can ensure the effectiveness and performance of abnormal data detection while eliminating abnormal data contamination.
[0083] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention and are only used for illustrative purposes, and should not be construed as limiting the present patent. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. The accompanying drawings are only for illustrative purposes and should not be construed as limiting the present patent; in order to better illustrate the embodiments, some parts of the drawings may be omitted, enlarged or reduced, and do not represent actual dimensions. The description of directions such as "up" and "down" is not a limitation of the present patent; it is understandable for those skilled in the art that some well-known content may be omitted in the drawings; the description of positional relationships in the drawings are only for illustrative purposes and should not be construed as limiting the present patent.
[0084] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.
[0085] Example 1
[0086] This embodiment provides a method for detecting abnormal data based on contamination immunity. See [link to relevant documentation]. Figure 1 ,include:
[0087] S1. Divide the dataset into a training set, an auxiliary training set, and a test set. The data in the training set and the test set include normal data and polluted abnormal data. The data in the auxiliary training set contains only polluted abnormal data.
[0088] S2. Construct a bidirectional generative adversarial network (GAN), see [link / reference] Figure 2 The bidirectional generative adversarial network includes an encoder, a generator, and a discriminator;
[0089] S3. Input the training set and auxiliary training set into the bidirectional generative adversarial network, train the discriminator in the bidirectional generative adversarial network, and obtain the trained discriminator;
[0090] S4. Input the training set and auxiliary training set into the bidirectional generative adversarial network, and use the trained discriminator to train the generator and encoder in the bidirectional generative adversarial network to obtain the trained generator and encoder.
[0091] S5. Repeat steps S3-S4 until the bidirectional generative adversarial network reaches Nash equilibrium, then end the training and obtain the trained bidirectional generative adversarial network, which includes the optimal discriminator and the optimal generator.
[0092] S6. Input the test set into the trained bidirectional generative adversarial network, use the trained bidirectional generative adversarial network to detect contaminated and abnormal data in the test set, and calculate the performance of the bidirectional generative adversarial network based on the detection results.
[0093] S7. Use the trained bidirectional generative adversarial network for anomaly detection.
[0094] In step S1, considering that conventional anomaly detection methods usually assume that the training data is uncontaminated and that there are two training forms, supervised and unsupervised, the collected anomaly data can be used as auxiliary data in this example. Therefore, the method proposed in this invention can solve the anomaly detection task in the training scenario with contaminated data. Before formal implementation, the required contaminated dataset and a small amount of auxiliary anomaly data are first identified.
[0095] In this embodiment, the datasets used are the CIFAR dataset and the OCT dataset. The CIFAR dataset contains 10 categories, 50,000 training data points, and 10,000 test data points. One category is randomly selected from the 10 categories as normal data, and the other 9 categories are considered anomalous data. All 9 categories of anomalous data may contaminate the normal data. The training dataset contains 5,000 data points (all normal data), and 1,000 anomalous data points can be collected (randomly sampled from the 9 anomalous categories). The test dataset contains 10,000 data points (9,000 normal data points and 1,000 anomalous data points). The OCT dataset contains 84,495 training images. Normal human eye CT images are used as normal data, and the other 3 categories are considered anomalous data. All 3 categories of anomalous data may contaminate the normal data. The training dataset includes 36,315 CT images (26,315 normal images and 10,000 contaminated images) and 1,385 auxiliary anomalous images. The test dataset contains 1,000 images (200 normal images and 800 anomalous images).
[0096] Step S3 includes:
[0097] S31. Input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input to the bidirectional generative adversarial network. Input x into encoder E. Encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... ,Will Input to encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation of the auxiliary training set data. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator, which restores the hidden layer representation z to a high-dimensional space to obtain the generated data. ;
[0098] S32. Construct the objective function of the discriminator D. The discriminator is used to identify whether the input data is contaminated or abnormal data. The expression is:
[0099] ,
[0100] in, The joint probability distribution of the training set, To assist in the joint probability distribution of the training set, To generate data The joint probability distribution of , where d is a parameter, is calculated as follows:
[0101] ,
[0102] in, To train the proportion of contaminated and anomalous data among all data;
[0103] S33. Establish sample pairs , and , sample pair , and Substitute into the discriminator objective function;
[0104] S34. Use gradient descent to train the discriminator in the bidirectional generative adversarial network, update the discriminator parameters, and obtain the trained discriminator.
[0105] In step S3, the encoder generates a hidden layer representation of the data and inputs the hidden layer representation into the discriminator. The encoder maps the data from high-dimensional space to low-dimensional space and uses this as the feature of the data. Inputting the low-dimensional hidden layer representation into the discriminator for discrimination is more conducive to the operation of the discriminator.
[0106] Step S4 includes:
[0107] S41. Input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input to the bidirectional generative adversarial network. Input x into the encoder. The encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... , will assist training set data Input to encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator to obtain the generated data. ;
[0108] S42. Construct the objective functions for generator G and encoder E, with the following expressions:
[0109] ,
[0110] in, This represents the trained discriminator, where c is the parameter;
[0111] S43. Establish sample pairs , and , sample pair , and Substitute the objective functions of the generator and encoder;
[0112] S44. Use gradient descent to train the generator and encoder in the bidirectional generative adversarial network, update the parameters of the generator and encoder, and obtain the trained generator and encoder.
[0113] In step S4, a trained discriminator is introduced into the objective function of the generator and encoder, thereby jointly optimizing the discriminator with the generator and encoder. The trained discriminator can prevent the generator from generating contaminated data and only generate normal data, so that the model can obtain a completely pure probability distribution from the contaminated dataset.
[0114] In step S5, the bidirectional generative adversarial network (GAN) anomaly detection model uses several gradient backpropagations to reach the Nash equilibrium. The core idea of GANs originates from Nash equilibrium. The concept of Nash equilibrium is: in a game, regardless of the opponent's strategy choice, one player will always choose a certain strategy, which is called a dominant strategy. If the strategy combinations of the two players in a game constitute their respective dominant strategies, then this combination is defined as a Nash equilibrium.
[0115] In a trained bidirectional generative adversarial network, the optimal discriminator The expression is:
[0116] ,
[0117] Added to the optimal discriminator Anomaly detection terms are used to constrain the assignment of lower probability values to anomalous samples, thereby detecting novel anomalies in the training set.
[0118] The probability distribution expression for the optimal generator G is:
[0119] ,
[0120] in, Let be the expected joint probability distribution.
[0121] Example 2
[0122] Based on Example 1, another example is provided. In this example, the trained bidirectional generative adversarial network is tested using a test set, and the performance of the bidirectional generative adversarial network is evaluated based on the test results.
[0123] The steps for anomaly detection on the test set include:
[0124] S611. Input the test set data into the trained bidirectional generative adversarial network, and use the encoder in the trained bidirectional generative adversarial network to calculate the hidden layer representation of the test set data. ;
[0125] S612. Calculate each data point in the test set. Abnormal score ;
[0126] S613. Set a scoring threshold, and compare the abnormal score value of each data in the test set with the scoring threshold one by one. When the abnormal score value is greater than the scoring threshold, the data is judged as contaminated abnormal data. When the abnormal score value is less than the scoring threshold, the data is judged as normal data. After judging all the data in the test set, the judgment result is obtained.
[0127] In this step, the hidden layer representation of the test set samples is calculated by the encoder, and the features of the test set samples are extracted through the hidden layer representation. Then, based on the scoring threshold, it is determined whether the test set samples belong to contaminated abnormal data.
[0128] In step S6, the anomaly score value The calculation expression is:
[0129] .
[0130] The steps for calculating the performance of a bidirectional generative adversarial network include:
[0131] S621. Calculate the precision P of the judgment result: calculate the ratio of the number of data correctly predicted as pollution anomalies to the number of data predicted as pollution anomalies in the judgment result;
[0132] S622. Calculate the recall rate R of the judgment results: Calculate the ratio of the number of correctly predicted pollution anomaly data in the judgment results to the number of actual pollution anomaly data in the test set;
[0133] S623. Calculate the F1 score of the judgment result. The calculation expression is as follows:
[0134] ,
[0135] The F1 score is used as the performance evaluation standard for the trained bidirectional generative adversarial network. The higher the F1 score, the better the performance of the bidirectional generative adversarial network.
[0136] Since the number of positive and negative samples in the dataset is unbalanced, it is impossible to use accuracy to evaluate classification performance. Therefore, this embodiment introduces the F1 score as an evaluation criterion. The F1 score is the harmonic mean of recall and precision, which takes into account the balance between the two, thereby evaluating the model's correct classification of sample data. It can also obtain an accurate performance evaluation criterion even when the positive and negative samples are unbalanced.
[0137] Example 3
[0138] This embodiment provides an abnormal data detection system based on contamination immunity. See [link to documentation]. Figure 3 ,include:
[0139] The data partitioning module is used to partition the dataset to obtain a training set, an auxiliary training set, and a test set. The data in the training set and the test set include normal data and polluted abnormal data, while the data in the auxiliary training set contains only polluted abnormal data.
[0140] A network construction module is used to construct a bidirectional generative adversarial network, which includes an encoder, a generator, and a discriminator.
[0141] The discriminant training module is used to input the training set and auxiliary training set into the bidirectional generative adversarial network and train the discriminator in the bidirectional generative adversarial network to obtain the trained discriminator.
[0142] The generation training module is used to input the training set and auxiliary training set into the bidirectional generative adversarial network, and to train the generator and encoder in the bidirectional generative adversarial network using the trained discriminator to obtain the trained generator and encoder.
[0143] The network training module is used to repeatedly execute the discriminant training module and the generator training module until the bidirectional generative adversarial network reaches Nash equilibrium, then the training ends and a trained bidirectional generative adversarial network is obtained. The trained bidirectional generative adversarial network includes the optimal discriminator and the optimal generator.
[0144] The performance testing module is used to input a test set into the trained bidirectional generative adversarial network (BGAN), use the trained BGAN to detect contaminated and abnormal data in the test set, and calculate the performance of the BGAN based on the detection results.
[0145] The data detection module is used to detect abnormal data using the trained bidirectional generative adversarial network.
[0146] Among them, such as Figure 4 As shown, the discrimination training module includes:
[0147] The first representation generation unit is used to input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input into the bidirectional generative adversarial network. x is input into encoder E, and encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... ,Will Input to encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation of the auxiliary training set data. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator G, which restores the hidden layer representation z to a high-dimensional space to obtain the generated data. ;
[0148] The first function construction unit is used to construct the objective function of the discriminator D. The discriminator D is used to identify whether the input data is contaminated or abnormal data, and its expression is:
[0149] ,
[0150] in, The joint probability distribution of the training set, To assist in the joint probability distribution of the training set, To generate data The joint probability distribution of , where d is a parameter, is calculated as follows:
[0151] ,
[0152] in, To train the proportion of contaminated and anomalous data among all data;
[0153] The first function calculation unit is used to establish sample pairs. , and and sample pairs , and Substitute into the discriminator objective function;
[0154] The first gradient training unit is used to train the discriminator in the bidirectional generative adversarial network using gradient descent, update the discriminator parameters, and obtain the trained discriminator.
[0155] Among them, such as Figure 5 As shown, the training generation module includes:
[0156] The second representation generation unit is used to input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input into the bidirectional generative adversarial network. x is input into encoder E, and encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... ,Will Input encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator to obtain the generated data. ;
[0157] The second function construction unit constructs the objective functions for the generator and encoder, with the following expression:
[0158] ,
[0159] in, This represents the trained discriminator, where c is the parameter;
[0160] The second function calculation unit is used to establish sample pairs. , and , sample pair , and Substitute the objective functions of the generator and encoder;
[0161] The second gradient training unit is used to train the generator and encoder in the bidirectional generative adversarial network using gradient descent, update the parameters of the generator and encoder, and obtain the trained generator and encoder.
[0162] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.
Claims
1. A method for detecting abnormal data based on contamination immunity, characterized in that, Includes the following steps: S1. Divide the dataset into a training set, an auxiliary training set, and a test set. The data in the training set and the test set include normal data and contaminated abnormal data. The data in the auxiliary training set contains only contaminated abnormal data. The dataset includes the OCT dataset, and normal human eye CT images in the OCT dataset are used as normal data. S2. Construct a bidirectional generative adversarial network, which includes an encoder, a generator, and a discriminator; S3. Input the training set and auxiliary training set into the bidirectional generative adversarial network, train the discriminator in the bidirectional generative adversarial network, and obtain the trained discriminator; S4. Input the training set and auxiliary training set into the bidirectional generative adversarial network, and use the trained discriminator to train the generator and encoder in the bidirectional generative adversarial network to obtain the trained generator and encoder. S5. Repeat steps S3-S4 until the bidirectional generative adversarial network reaches Nash equilibrium, then end the training and obtain the trained bidirectional generative adversarial network, which includes the optimal discriminator and the optimal generator. S6. Input the test set into the trained bidirectional generative adversarial network, use the trained bidirectional generative adversarial network to detect contaminated and abnormal data in the test set, and calculate the performance of the bidirectional generative adversarial network based on the detection results. S7. Use the trained bidirectional generative adversarial network for anomaly detection; Step S3 includes: S31. Input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input to the bidirectional generative adversarial network. Input x into encoder E. Encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... ,Will Input to encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation of the auxiliary training set data. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator, which restores the hidden layer representation z to a high-dimensional space to obtain the generated data. ; S32. Construct the objective function of the discriminator D. The discriminator is used to identify whether the input data is contaminated or abnormal data. The expression is: ， in, The joint probability distribution of the training set, To assist in the joint probability distribution of the training set, To generate data The joint probability distribution of , where d is a parameter, is calculated as follows: ， in, To train the proportion of contaminated and anomalous data among all data; S33. Establish sample pairs , and , sample pair , and Substitute into the discriminator objective function; S34. Use gradient descent to train the discriminator in the bidirectional generative adversarial network, update the discriminator parameters, and obtain the trained discriminator; In step S6, the steps for detecting abnormal data in the test set include: S611. Input the test set data into the trained bidirectional generative adversarial network, and use the encoder in the trained bidirectional generative adversarial network to calculate the hidden layer representation of the test set data. ; S612. Calculate each data point in the test set. Abnormal score ; S613. Set a scoring threshold, and compare the abnormal score value of each data in the test set with the scoring threshold one by one. When the abnormal score value is greater than the scoring threshold, the data is judged as contaminated abnormal data. When the abnormal score value is less than the scoring threshold, the data is judged as normal data. After judging all the data in the test set, the judgment result is obtained.
2. The method for detecting abnormal data based on contamination immunity according to claim 1, characterized in that, Step S4 includes: S41. Input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input to the bidirectional generative adversarial network. Input x into the encoder. The encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... , will assist training set data Input to encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator to obtain the generated data. ; S42. Construct the objective functions for generator G and encoder E, with the following expressions: ， in, This represents the trained discriminator, where c is the parameter; S43. Establish sample pairs , and , sample pair , and Substitute the objective functions of the generator and encoder; S44. Use gradient descent to train the generator and encoder in the bidirectional generative adversarial network, update the parameters of the generator and encoder, and obtain the trained generator and encoder.
3. The method for detecting abnormal data based on contamination immunity according to claim 2, characterized in that, In a trained bidirectional generative adversarial network, the optimal discriminator The expression is: ， The probability distribution expression for the optimal generator G is: ， in, Let be the expected joint probability distribution.
4. The method for detecting abnormal data based on contamination immunity according to claim 1, characterized in that, In step S6, the anomaly score value The calculation expression is: 。 5. The abnormal data detection method based on contamination immunity according to claim 4, characterized in that, In step S6, the steps for calculating the performance of the bidirectional generative adversarial network include: S621. Calculate the precision P of the judgment result: calculate the ratio of the number of data correctly predicted as pollution anomalies to the number of data predicted as pollution anomalies in the judgment result; S622. Calculate the recall rate R of the judgment results: Calculate the ratio of the number of correctly predicted pollution anomaly data in the judgment results to the number of actual pollution anomaly data in the test set; S623. Calculate the F1 score of the judgment result. The calculation expression is as follows: ， The F1 score is used as the performance evaluation standard for the trained bidirectional generative adversarial network. The higher the F1 score, the better the performance of the bidirectional generative adversarial network.
6. An abnormal data detection system based on contamination immunity, characterized in that, include: The data partitioning module is used to partition the dataset to obtain a training set, an auxiliary training set, and a test set. The data in the training set and the test set include normal data and contaminated abnormal data, while the data in the auxiliary training set contains only contaminated abnormal data. The dataset includes the OCT dataset, and normal human eye CT images in the OCT dataset are used as normal data. A network construction module is used to construct a bidirectional generative adversarial network, which includes an encoder, a generator, and a discriminator. The discriminant training module is used to input the training set and auxiliary training set into the bidirectional generative adversarial network and train the discriminator in the bidirectional generative adversarial network to obtain the trained discriminator. The generation training module is used to input the training set and auxiliary training set into the bidirectional generative adversarial network, and to train the generator and encoder in the bidirectional generative adversarial network using the trained discriminator to obtain the trained generator and encoder. The network training module is used to repeatedly execute the discriminant training module and the generator training module until the bidirectional generative adversarial network reaches Nash equilibrium, then the training ends and a trained bidirectional generative adversarial network is obtained. The trained bidirectional generative adversarial network includes the optimal discriminator and the optimal generator. The performance testing module is used to input a test set into the trained bidirectional generative adversarial network (BGAN), use the trained BGAN to detect contaminated and abnormal data in the test set, and calculate the performance of the BGAN based on the detection results. The data detection module is used to use the trained bidirectional generative adversarial network for anomaly detection; The discrimination training module includes: The first representation generation unit is used to input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input into the bidirectional generative adversarial network. x is input into encoder E, and encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... ,Will Input to encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation of the auxiliary training set data. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator G, which restores the hidden layer representation z to a high-dimensional space to obtain the generated data. ; The first function construction unit is used to construct the objective function of the discriminator D. The discriminator D is used to identify whether the input data is contaminated or abnormal data, and its expression is: ， in, The joint probability distribution of the training set, To assist in the joint probability distribution of the training set, To generate data The joint probability distribution of , where d is a parameter, is calculated as follows: ， in, To train the proportion of contaminated and anomalous data among all data; The first function calculation unit is used to establish sample pairs. , and and sample pairs , and Substitute into the discriminator objective function; The first gradient training unit is used to train the discriminator in the bidirectional generative adversarial network using gradient descent, update the discriminator parameters, and obtain the trained discriminator.
7. The abnormal data detection system based on contamination immunity according to claim 6, characterized in that, The training generation module includes: The second representation generation unit is used to input the training set and auxiliary training set into the bidirectional generative adversarial network. Let x be the training set data sample input into the bidirectional generative adversarial network. x is input into encoder E, and encoder E maps x to the hidden layer space to obtain the hidden layer representation. Let the auxiliary training set data samples of the input bidirectional generative adversarial network be... ,Will Input encoder E, encoder E will Mapping to the hidden layer space yields the hidden layer representation. Random hidden layer representation z is obtained by random sampling from a prior Gaussian distribution. This random hidden layer representation z is then input into the generator to obtain the generated data. ; The second function construction unit constructs the objective functions for the generator and encoder, with the following expression: ， in, This represents the trained discriminator, where c is the parameter; The second function calculation unit is used to establish sample pairs. , and , sample pair , and Substitute the objective functions of the generator and encoder; The second gradient training unit is used to train the generator and encoder in the bidirectional generative adversarial network using gradient descent, update the parameters of the generator and encoder, and obtain the trained generator and encoder.