Advertisement abnormal traffic detection method, system, electronic device and readable storage medium

Abstract

The present invention discloses a method, system, electronic device and readable storage medium for detecting abnormal traffic of advertisements. The method comprises: using historical traffic data to construct and train a data reconstruction model using a VAE model; inputting the historical traffic data into the data reconstruction model to obtain first reconstruction data, calculating the error value between the historical traffic data and the first reconstruction data to obtain a first reconstruction error; clustering the first reconstruction error to divide the historical traffic data into normal traffic and abnormal traffic, determining a first threshold value according to the first reconstruction error corresponding to the normal traffic, and determining a second threshold value according to the first reconstruction error corresponding to all historical traffic data; using the data reconstruction model to perform abnormal detection on the traffic data to be detected according to the first threshold value and the second threshold value. The present invention overcomes the disadvantages of supervised learning caused by the unlabeled and unbalanced distribution of the original data set.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method, system, electronic device and readable storage medium for detecting abnormal advertising traffic based on a variational autoencoder. Background Art

[0002] In recent years, with the rapid development of Internet advertising, Internet advertising forms that closely connect brands, products and consumers have become more and more popular with advertisers, and how to prevent Internet advertising fraud has become a top priority for advertisers. Because advertising traffic reflects the efficiency and effectiveness of advertising to a certain extent, it has vital value in the Internet world. In order to make illegal profits, the advertising black industry has begun to bypass various resource restrictions and increase access frequency through various technical operations. It has created a large amount of invalid traffic through scripts, simulators, group control, domain name and App modification and crowdsourcing traffic. Advertising traffic fraud is becoming increasingly serious. With the continuous iteration and update of advertising fraud technology, the endless forms of advertising fraud have also brought greater prevention pressure to advertisers. Accurately identifying the effectiveness of traffic has become a huge challenge for the advertising marketing industry. At the same time, in recent years, artificial intelligence technology and data monitoring and storage technology have developed rapidly, and have been widely studied and applied in natural language processing, speech recognition, computer vision, target detection and other fields. Therefore, artificial intelligence technology has also begun to be applied by advertising and marketing industry personnel to advertising traffic anomaly detection. Advertising traffic anomaly detection based on artificial intelligence technology has gradually received the attention and research hotspots of researchers.

[0003] In the prior art, supervised learning models are usually used to solve the above problems. The main processes of the existing advertising traffic anomaly detection methods based on artificial intelligence technology are: data collection, feature engineering, and prediction based on machine learning algorithms. Currently, a variety of machine learning algorithms, such as artificial neural networks (ANN), support vector machines (SVM), random forests (RF), extreme gradient boosting (XGBoost), deep learning models (DL), etc., are applied to advertising traffic anomaly detection.

[0004] The existing technologies have at least the following defects, because these models are all supervised learning models. In order to obtain high-precision detection results, these supervised learning models often require a sufficient number of traffic data samples with category labels and balanced category distribution for training, but this is difficult to meet in the actual advertising industry. In actual advertising traffic monitoring, although a huge amount of traffic data is generated every day, the data that can be used for machine learning model training is very limited. Instead, most traffic data is unlabeled data. At the same time, the quantitative distribution of abnormal traffic and normal traffic data is also extremely unbalanced, that is, the number of abnormal traffic samples is far less than the number of normal traffic samples. Insufficient labeled data and unbalanced data distribution have brought challenges to advertising traffic anomaly detection methods based on artificial intelligence technology. Summary of the invention

[0005] In view of the technical problem that the above-mentioned abnormal advertising traffic detection model lacks sufficient training samples with category labels, the present invention proposes an abnormal advertising traffic detection method, system, electronic device and readable storage medium based on variational autoencoder.

[0006] In a first aspect, an embodiment of the present application provides a method for detecting abnormal advertising traffic based on a variational autoencoder, comprising:

[0007] Model building steps: Use historical traffic data to build a VAE model and train the data to reconstruct the model;

[0008] A first reconstruction error obtaining step: inputting the historical traffic data into the data reconstruction model to obtain first reconstruction data, and calculating an error value between the historical traffic data and the first reconstruction data to obtain a first reconstruction error;

[0009] Threshold determination step: dividing the historical traffic data into normal traffic and abnormal traffic by clustering the first reconstruction error, determining a first threshold according to the first reconstruction error corresponding to the normal traffic, and determining a second threshold according to the first reconstruction error corresponding to all historical traffic data;

[0010] Data detection step: using the data reconstruction model to perform anomaly detection on the flow data to be detected according to the first threshold and the second threshold.

[0011] In the above-mentioned method for detecting abnormal advertising traffic, the data detection step includes:

[0012] A second reconstruction error obtaining step: inputting the flow data to be detected into the data reconstruction model to obtain second reconstruction data, and calculating the error value between the flow data to be detected and the second reconstruction data to obtain a second reconstruction error;

[0013] Abnormal judgment step: if the second reconstruction error is less than the first threshold, the flow data to be detected is normal flow data; if the second reconstruction error is greater than the first threshold and less than or equal to the second threshold, the flow data to be detected is known abnormal flow data; if the second reconstruction error is greater than the second threshold, the flow data to be detected is unknown abnormal flow data.

[0014] The above-mentioned method for detecting abnormal advertising traffic may further include:

[0015] Data preprocessing step: cleaning and preprocessing the historical flow data and the flow data to be detected;

[0016] Data standardization step: perform feature engineering on the cleaned and preprocessed historical traffic data and the traffic data to be detected to obtain standardized historical traffic data and the traffic data to be detected.

[0017] The above-mentioned advertising abnormal traffic detection method, wherein the model building step includes: inputting the historical traffic data into the VAE model to obtain sample reconstruction data, constructing a loss function based on the MSE and KL divergence of the sample reconstruction data and the historical traffic data, and training the data reconstruction model by minimizing the loss function.

[0018] In the above-mentioned abnormal advertising traffic detection method, the data reconstruction model parameters are trained using the Adam optimization algorithm, and the data reconstruction model hyperparameters are trained using the grid search method.

[0019] In the above-mentioned abnormal advertising traffic detection method, the threshold determination step includes: determining the first threshold based on the mean and standard deviation of the first reconstruction error corresponding to the normal traffic, and determining the second threshold based on the mean and standard deviation of the first reconstruction error corresponding to all historical traffic data.

[0020] In a second aspect, an embodiment of the present application provides an advertising abnormal traffic detection system based on a variational autoencoder, comprising:

[0021] Data preprocessing unit: cleans and preprocesses historical traffic data and traffic data to be detected;

[0022] Data standardization unit: performing feature engineering on the cleaned and preprocessed historical traffic data and the traffic data to be detected to obtain standardized historical traffic data and the traffic data to be detected;

[0023] Model building unit: using the historical traffic data to build a VAE model and train the data to reconstruct the model;

[0024] A first reconstruction error obtaining unit: inputting the historical traffic data into the data reconstruction model to obtain first reconstruction data, and calculating an error value between the historical traffic data and the first reconstruction data to obtain a first reconstruction error;

[0025] A threshold determination unit: dividing the historical traffic data into normal traffic and abnormal traffic by clustering the first reconstruction errors, determining a first threshold according to the first reconstruction errors corresponding to the normal traffic, and determining a second threshold according to the first reconstruction errors corresponding to all the historical traffic data;

[0026] A data detection unit: uses the data reconstruction model to perform anomaly detection on the traffic data to be detected according to the first threshold and the second threshold.

[0027] The above-mentioned advertising abnormal traffic detection system, wherein the data detection unit includes:

[0028] A second reconstruction error obtaining module: inputting the flow data to be detected into the data reconstruction model to obtain second reconstruction data, and calculating the error value between the flow data to be detected and the second reconstruction data to obtain a second reconstruction error;

[0029] Abnormal judgment module: If the second reconstruction error is less than the first threshold, the flow data to be detected is normal flow data; if the second reconstruction error is greater than the first threshold and less than or equal to the second threshold, the flow data to be detected is known abnormal flow data; if the second reconstruction error is greater than the second threshold, the flow data to be detected is unknown abnormal flow data.

[0030] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for detecting abnormal advertising traffic as described in the first aspect above is implemented.

[0031] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method for detecting abnormal advertising traffic as described in the first aspect above.

[0032] Compared with the prior art, the advantages and positive effects of the present invention are:

[0033] 1. The present invention proposes a method for detecting abnormal advertising traffic based on a variational autoencoder, which uses an unsupervised generation model VAE to learn advertising traffic data samples. The VAE model does not require label information knowledge of the data, and mines the relationship between data by reducing the reconstruction error, thus overcoming the disadvantage of supervised learning due to the unlabeled original data set and improving data capabilities;

[0034] 2. Since the original data set has category imbalance, the present invention analyzes the reconstruction error and obtains a threshold according to the characteristics of the imbalanced data set to identify abnormal traffic and different types of abnormal traffic, thereby overcoming the disadvantages of supervised learning caused by the imbalanced distribution of the original data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 A schematic diagram of the steps of a method for detecting abnormal advertising traffic based on a variational autoencoder provided by the present invention;

[0036] Figure 2 The present invention provides Figure 1 Schematic diagram of the process of step S6;

[0037] Figure 3 A schematic flow chart of an embodiment of a method for detecting abnormal advertising traffic based on a variational autoencoder provided by the present invention;

[0038] Figure 4 A framework diagram of an abnormal advertising traffic detection system based on a variational autoencoder provided by the present invention;

[0039] Figure 5 The VAE model architecture diagram provided by the present invention;

[0040] Figure 6 A framework diagram of a computer device according to an embodiment of the present application.

[0041] Wherein, the accompanying drawings are marked as follows:

[0042] 1. Data preprocessing unit; 2. Data standardization unit; 3. Model building unit; 4. First reconstruction error acquisition unit; 5. Threshold determination unit; 6. Data detection unit; 61. Second reconstruction error acquisition module; 62. Abnormal judgment module; 81. Processor; 82. Memory; 83. Communication interface; 80. Bus. DETAILED DESCRIPTION

[0043] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0044] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0045] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0046] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0047] The present invention is described in detail below in conjunction with the various embodiments shown in the accompanying drawings, but it should be noted that these embodiments are not limitations of the present invention, and any equivalent transformations or substitutions in functions, methods, or structures made by ordinary technicians in the field based on these embodiments are all within the scope of protection of the present invention.

[0048] Before elaborating on each embodiment of the present invention in detail, the core inventive concept of the present invention is summarized and elaborated in detail through the following several embodiments.

[0049] Since VAE does not require knowledge of data label information, the present invention adopts an unsupervised generative model VAE to learn advertising traffic data samples, and identifies abnormal traffic data by judging the numerical distribution of reconstruction errors and the characteristics of the imbalanced distribution of data set categories, thereby overcoming the shortcomings of supervised learning caused by the unlabeled and unbalanced distribution of the original data set.

[0050] The abbreviations and key terms in the specific implementation are defined as follows:

[0051] ANN: Artificial Neural Network.

[0052] SVM: Support Vector Machine, support vector machine.

[0053] RF: Random Forest.

[0054] XGBoost: eXtreme Gradient Boosting, extreme gradient boosting.

[0055] DL: Deep Learning.

[0056] VAE: Artificial Neural Network, variational autoencoder.

[0057] Variational Auto-Encoders (VAE) are a deep generative model based on variational Bayesian inference. Unlike traditional autoencoders that describe latent space numerically, VAE is an unsupervised learning generative model that describes latent space in a probabilistic way. VAE can represent each feature of the data as a probability distribution, and then randomly sample from each latent state distribution according to this probability distribution to generate a vector, and finally decode this vector to generate newly generated data. The model is divided into two parts: encoder and decoder. When working, first, the encoder module of the model performs variational inference on the original input data, generates the variational probability distribution of the latent variables, and maps the input data into the mean and variance in the latent space; then, the model randomly samples from the normal distribution defined by the mean and variance; finally, the decoder restores the generated latent variable variational probability distribution to the original data approximate probability distribution, and reversely maps the sampling points in the latent space back to the original data, thereby achieving the purpose of reconstruction.

[0058] Embodiment 1:

[0059] Figure 1 The present invention provides a step diagram of a method for detecting abnormal advertising traffic based on a variational autoencoder. Figure 1 As shown, this embodiment discloses a specific implementation method of an abnormal advertising traffic detection method (hereinafter referred to as "method").

[0060] Specifically, the method disclosed in this embodiment mainly includes the following steps:

[0061] Step S1: cleaning and preprocessing the historical flow data and the flow data to be detected;

[0062] Specifically, the advertising traffic data collected by the actual monitoring system often has problems such as missing, noisy, and duplication due to data anomalies, insufficient permissions to collect data, log parsing anomalies, network packet loss and transmission anomalies, etc. Therefore, before analysis, the historical traffic data and the traffic data to be tested must first be cleaned and preprocessed, mainly including data alignment, missing value filling, deduplication, data bucketing and other processing.

[0063] Step S2: Perform feature engineering on the cleaned and preprocessed historical traffic data and the traffic data to be detected to obtain standardized historical traffic data and the traffic data to be detected.

[0064] Specifically, since the monitoring system collects many flow data variables, there is redundancy between the variables, and there are differences in their types, dimensions, and orders of magnitude, it is necessary to perform feature engineering on the flow data variables before model training, mainly including feature selection and combination, and data standardization. Feature selection and combination first uses statistical ideas to perform statistical calculations on the original flow data to obtain statistical features, and then applies the variance filtering method to quickly filter out features with small variances by calculating the variance of the features themselves. The filtered data is standardized. Data standardization uses the standard deviation standardization method to make the standardized data obey a standard normal distribution with a mean of 0 and a variance of 1.

[0065] Step S3: Use historical traffic data to build and train a data reconstruction model through a VAE model;

[0066] Among them, the VAE model architecture is as follows Figure 4 As shown, it is mainly composed of an encoder and a decoder. The encoder includes an input layer, a hidden layer, a latent space layer, and a sampling layer. The standardized data is input into the encoder through the input layer, and is mapped to the latent space layer through the hidden layer; the latent space layer contains two parameters, the mean and variance of each variable, through which a normal distribution of each variable in the latent space can be defined; the sampling layer obtains sampling variables by random sampling from the defined normal distribution. The decoder performs a reverse mapping operation on the sampled variables in the latent space to generate reconstructed variables of the original data in the original domain. Since VAE does not require knowledge of the label information of the data, and can mine the relationship between data by reducing the reconstruction error, the present invention adopts the VAE model, combined with the characteristics of the imbalanced data set, to detect abnormal advertising traffic.

[0067] The historical traffic data is input into the VAE model to obtain sample reconstruction data. The loss function is constructed according to the MSE and KL divergence of the sample reconstruction data and the historical traffic data. The data reconstruction model is trained by minimizing the above loss function. The data reconstruction model parameters are trained using the Adam optimization algorithm, and the data reconstruction model hyperparameters are trained using the grid search method.

[0068] Step S4: inputting the historical traffic data into the data reconstruction model to obtain first reconstructed data, and calculating the error value between the historical traffic data and the first reconstructed data to obtain a first reconstruction error;

[0069] Specifically, the historical traffic data is input into the trained data reconstruction model to generate the first reconstruction data corresponding thereto, and then the reconstruction error value between the two is calculated to obtain the first reconstruction error.

[0070] Step S5: dividing the historical traffic data into normal traffic and abnormal traffic by clustering the first reconstruction errors, determining a first threshold value according to the first reconstruction errors corresponding to the normal traffic, and determining a second threshold value according to the first reconstruction errors corresponding to all the historical traffic data;

[0071] Since the original data set has category imbalance, the sample size of normal traffic data is much larger than that of abnormal traffic data. Therefore, the model pays more attention to normal traffic data samples during training. This situation also makes the final VAE model more inclined to the distribution of normal traffic when reconstructing data. That is, the reconstruction error between the reconstructed data and the original data obtained by the VAE model for normal traffic samples is smaller than the reconstruction error between the reconstructed data and the original data obtained by the VAE model for abnormal traffic. At the same time, for unknown abnormal traffic data, the reconstruction error between the reconstructed data and the original data obtained by inputting it into the VAE model will be larger than the reconstruction error obtained by the data with the same distribution as the training data set. Based on this, this method proposes to determine two thresholds, the first threshold γ1 and the second threshold γ2.

[0072] First, the first reconstruction error corresponding to the historical traffic data is clustered and divided into two categories. The amount of data in one category is obviously larger than that in the other category. The category with more data is normal traffic, and the category with less data is abnormal traffic. Then, the first threshold γ1 is determined according to the mean and standard deviation of the first reconstruction error corresponding to the normal traffic, as shown in the following formula:

[0073]

[0074] in, is the mean of the first reconstruction error corresponding to normal traffic; is the standard deviation of the first reconstruction error corresponding to normal traffic.

[0075] This method determines the second threshold γ2 according to the mean and standard deviation of the first reconstruction error corresponding to all historical traffic data, as shown in the following formula:

[0076]

[0077] in, is the mean of the first reconstruction error corresponding to all historical traffic data; is the standard deviation of the first reconstruction error corresponding to all historical traffic data.

[0078] Step S6: using the data reconstruction model to perform anomaly detection on the traffic data to be detected according to the first threshold and the second threshold.

[0079] Reference Figure 2 As shown, step S6 specifically includes the following contents:

[0080] Step S61: inputting the flow data to be detected into the data reconstruction model to obtain second reconstructed data, and calculating the error value between the flow data to be detected and the second reconstructed data to obtain a second reconstruction error;

[0081] Step S62: If the second reconstruction error is less than the first threshold value γ1, the flow data to be detected is normal flow data; if the second reconstruction error is greater than the first threshold value γ1 and less than or equal to the second threshold value γ2, the flow data to be detected is known abnormal flow data; if the second reconstruction error is greater than the second threshold value γ2, the flow data to be detected is unknown abnormal flow data.

[0082] The present invention proposes a method for detecting abnormal advertising traffic based on variational autoencoder, which uses an unsupervised generation model VAE to learn advertising traffic data samples, and identifies abnormal traffic data by judging the numerical distribution of reconstruction errors and the characteristics of the imbalanced distribution of data set categories. This method not only overcomes the shortcomings of supervised learning caused by the unlabeled and unbalanced distribution of the original data set, but also can identify new abnormal traffic types and generate new traffic data through the trained VAE model.

[0083] Please refer to the following Figure 3 . Figure 3 A schematic diagram of an embodiment of a method for detecting abnormal advertising traffic based on a variational autoencoder provided by the present invention, combined with Figure 3 The specific application process of this method is as follows:

[0084] Step 1: Clean and preprocess the advertising traffic data;

[0085] The advertising traffic data collected by the actual monitoring system often has problems such as missing, noisy, and duplication due to data anomalies, insufficient data collection permissions, log parsing anomalies, network packet loss and transmission anomalies. Therefore, before analysis, it must first be cleaned and preprocessed, mainly including data alignment, missing value filling, deduplication, data bucketing and other processing.

[0086] Step 2: Perform feature engineering on traffic data;

[0087] Since the monitoring system collects many flow data variables, there is redundancy between the variables, and there are differences in their types, dimensions, and orders of magnitude, it is necessary to perform feature engineering on the flow data variables before model training, mainly including feature selection and combination, and data standardization. Feature selection and combination first uses statistical ideas to perform statistical calculations on the original data to obtain statistical features, and then applies the variance filtering method to quickly filter out features with small variances by calculating the variance of the features themselves. Data standardization uses the standard deviation standardization method to make the standardized data obey a standard normal distribution with a mean of 0 and a variance of 1.

[0088] Step 3: Use the VAE model to build an advertising traffic data reconstruction model, train it, and save it;

[0089] The VAE model is used to build an advertising traffic data reconstruction model: the model architecture is as follows Figure 4 As shown, it is mainly composed of an encoder and a decoder.

[0090] Encoder: includes input layer, hidden layer (which can be convolutional layer or perceptron layer, etc., depending on the data type), latent space layer, and sampling layer. The specific process is as follows: (i) Original data x i The input is fed into the encoder through the input layer and mapped to the latent space layer through the hidden layer; (ii) The latent space layer contains two parameters, the mean and variance of each variable, which can define a normal distribution of each variable in the latent space; (iii) The sampling layer randomly samples from the defined normal distribution to obtain the sampled variable z i .

[0091] Decoder: Perform reverse mapping (deconvolution or full connection, etc., depending on the data type) on the sampled variables in the latent space to generate reconstructed variables of the original data in the original domain

[0092] Training process: The MSE and KL divergence of the reconstructed data and the original data are used to construct the loss function for the reconstruction error. By minimizing the loss function, the model parameters are trained using the Adam optimization algorithm and the model hyperparameters are trained using the grid search method.

[0093] Step 4: Calculate the error between the original data and the reconstructed data obtained by VAE;

[0094] The original data is passed through the saved VAE model again to generate the corresponding reconstructed data, and then the reconstruction error value between the two is calculated.

[0095] Step 5: Analyze the reconstruction error, obtain the threshold according to the imbalance characteristics, build the advertising traffic anomaly detection threshold model, and distinguish between normal and abnormal traffic;

[0096] Constructing a threshold model for detecting anomalies in advertising traffic: Due to the imbalance of categories in the original data set, the sample size of normal traffic data is much larger than that of abnormal traffic data. Therefore, the model pays more attention to normal traffic data samples during training. This situation also makes the final VAE model more inclined to the distribution of normal traffic when reconstructing data. That is, the reconstruction error between the reconstructed data and the original data obtained by the VAE model for normal traffic samples is smaller than the reconstruction error between the reconstructed data and the original data obtained by the VAE model for abnormal traffic. At the same time, for unknown abnormal traffic data, the reconstruction error between the reconstructed data and the original data obtained by inputting it into the VAE model is larger than the reconstruction error obtained by the data with the same distribution as the training data set. Based on this, this method constructs a threshold model for detecting anomalies in advertising traffic, which contains two thresholds γ1 and γ2.

[0097] Anomaly detection and threshold γ1 determination: First, cluster the reconstruction error values ​​obtained in step 4 and divide them into two categories. The amount of data in one category is obviously larger than that in the other category. The category with more data is normal traffic, and the category with less data is abnormal traffic. Then, the threshold γ1 is determined for normal traffic based on the following formula:

[0098]

[0099] in, is the mean value of the reconstruction error corresponding to normal traffic; is the standard deviation of the reconstruction error corresponding to normal traffic.

[0100] Determination of threshold γ2: This method determines the threshold γ2 based on the following formula:

[0101]

[0102] in, is the mean of the reconstruction errors corresponding to all historical traffic data; is the standard deviation of the reconstruction error corresponding to all historical traffic data.

[0103] Step 6: Apply the saved advertising abnormal traffic detection model to the new traffic data to perform anomaly detection on it.

[0104] The main process of this part is: (a) preprocessing new traffic data; (b) data standardization; (c) identifying newly collected traffic based on the built and saved VAE model and the advertising traffic anomaly detection threshold model. The judgment rules are as follows: (i) If the reconstruction error between the reconstructed data obtained by the VAE model and the original data is less than or equal to γ1, it is normal traffic data; (ii) If the reconstruction error between the reconstructed data obtained by the VAE model and the original data is greater than γ1 but less than or equal to γ2, it is known abnormal traffic data; (iii) If the reconstruction error between the reconstructed data obtained by the VAE model and the original data is greater than γ2, it is unknown abnormal traffic data.

[0105] Embodiment 2:

[0106] In combination with the method for detecting abnormal advertising traffic based on a variational autoencoder disclosed in Example 1, this embodiment discloses a specific implementation example of a system for detecting abnormal advertising traffic based on a variational autoencoder (hereinafter referred to as the "system").

[0107] Reference Figure 5 As shown, the system comprises:

[0108] Data preprocessing unit 1: cleans and preprocesses historical traffic data and traffic data to be detected;

[0109] Data standardization unit 2: performing feature engineering on the cleaned and preprocessed historical traffic data and the traffic data to be detected to obtain standardized historical traffic data and the traffic data to be detected;

[0110] Model building unit 3: using the historical traffic data to build a VAE model and train the data reconstruction model;

[0111] A first reconstruction error obtaining unit 4: inputs the historical traffic data into the data reconstruction model to obtain first reconstruction data, and calculates an error value between the historical traffic data and the first reconstruction data to obtain a first reconstruction error;

[0112] Threshold determination unit 5: divides the historical traffic data into normal traffic and abnormal traffic by clustering the first reconstruction errors, determines a first threshold according to the first reconstruction errors corresponding to the normal traffic, and determines a second threshold according to the first reconstruction errors corresponding to all historical traffic data;

[0113] Data detection unit 6: performs anomaly detection on the traffic data to be detected according to the first threshold and the second threshold by using the data reconstruction model.

[0114] Specifically, the data detection unit 6 includes:

[0115] A second reconstruction error obtaining module 61: inputs the flow data to be detected into the data reconstruction model to obtain second reconstruction data, and calculates the error value between the flow data to be detected and the second reconstruction data to obtain a second reconstruction error;

[0116] Abnormal judgment module 62: If the second reconstruction error is less than the first threshold, the flow data to be detected is normal flow data; if the second reconstruction error is greater than the first threshold and less than or equal to the second threshold, the flow data to be detected is known abnormal flow data; if the second reconstruction error is greater than the second threshold, the flow data to be detected is unknown abnormal flow data.

[0117] The technical solutions of the abnormal advertising traffic detection system based on variational autoencoder disclosed in this embodiment and the abnormal advertising traffic detection method based on variational autoencoder disclosed in Embodiment 1 are the same as those in Embodiment 1, and will not be repeated here.

[0118] Embodiment three:

[0119] Combination Figure 6 As shown, this embodiment discloses a specific implementation of a computer device. The computer device may include a processor 81 and a memory 82 storing computer program instructions.

[0120] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0121] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-OnlyMemory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-OnlyMemory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0122] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0123] The processor 81 implements any one of the methods for detecting abnormal advertising traffic in the above embodiments by reading and executing computer program instructions stored in the memory 82 .

[0124] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 6 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0125] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0126] The bus 80 includes hardware, software or both, and couples the components of the computer device to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0127] In addition, in combination with the abnormal advertising traffic detection method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the abnormal advertising traffic detection methods in the above embodiment is implemented.

[0128] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0129] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for detecting abnormal advertising traffic, characterized in that: Based on variational autoencoders, including: Model building steps: Use historical traffic data to build a VAE model and train the data to reconstruct the model; A first reconstruction error obtaining step: inputting the historical traffic data into the data reconstruction model to obtain first reconstruction data, and calculating an error value between the historical traffic data and the first reconstruction data to obtain a first reconstruction error; Threshold determination step: dividing the historical traffic data into normal traffic and abnormal traffic by clustering the first reconstruction error, determining a first threshold according to the first reconstruction error corresponding to the normal traffic, and determining a second threshold according to the first reconstruction error corresponding to all historical traffic data; Data detection step: using the data reconstruction model to perform anomaly detection on the flow data to be detected according to the first threshold and the second threshold; Wherein, the data detection step includes: A second reconstruction error obtaining step: inputting the flow data to be detected into the data reconstruction model to obtain second reconstruction data, and calculating the error value between the flow data to be detected and the second reconstruction data to obtain a second reconstruction error; Abnormal judgment step: if the second reconstruction error is less than the first threshold, the flow data to be detected is normal flow data; if the second reconstruction error is greater than the first threshold and less than or equal to the second threshold, the flow data to be detected is known abnormal flow data; if the second reconstruction error is greater than the second threshold, the flow data to be detected is unknown abnormal flow data.

2. The method for detecting abnormal advertising traffic according to claim 1, characterized in that: Also includes: Data preprocessing step: cleaning and preprocessing the historical flow data and the flow data to be detected; Data standardization step: perform feature engineering on the cleaned and preprocessed historical traffic data and the traffic data to be detected to obtain standardized historical traffic data and the traffic data to be detected.

3. The method for detecting abnormal advertising traffic according to claim 1, characterized in that: The model building step includes: inputting the historical traffic data into the VAE model to obtain sample reconstruction data, building a loss function based on the MSE and KL divergence of the sample reconstruction data and the historical traffic data, and training the data reconstruction model by minimizing the loss function.

4. The method for detecting abnormal advertising traffic according to claim 3, characterized in that: The data reconstruction model parameters are trained using the Adam optimization algorithm, and the data reconstruction model hyperparameters are trained using the grid search method.

5. The method for detecting abnormal advertising traffic according to claim 1, characterized in that: The threshold determination step includes: determining a first threshold according to a mean and a standard deviation of a first reconstruction error corresponding to the normal flow, and determining a second threshold according to a mean and a standard deviation of a first reconstruction error corresponding to all historical flow data.

6. An abnormal advertising traffic detection system, characterized in that: Based on variational autoencoders, including: Data preprocessing unit: cleans and preprocesses historical traffic data and traffic data to be detected; Data standardization unit: performing feature engineering on the cleaned and preprocessed historical traffic data and the traffic data to be detected to obtain standardized historical traffic data and the traffic data to be detected; Model building unit: using the historical traffic data to build a VAE model and train the data to reconstruct the model; A first reconstruction error obtaining unit: inputting the historical traffic data into the data reconstruction model to obtain first reconstruction data, and calculating an error value between the historical traffic data and the first reconstruction data to obtain a first reconstruction error; A threshold determination unit: dividing the historical traffic data into normal traffic and abnormal traffic by clustering the first reconstruction errors, determining a first threshold according to the first reconstruction errors corresponding to the normal traffic, and determining a second threshold according to the first reconstruction errors corresponding to all the historical traffic data; A data detection unit: using the data reconstruction model to perform anomaly detection on the flow data to be detected according to the first threshold and the second threshold; Wherein, the data detection unit comprises: A second reconstruction error obtaining module: inputting the flow data to be detected into the data reconstruction model to obtain second reconstruction data, and calculating the error value between the flow data to be detected and the second reconstruction data to obtain a second reconstruction error; Abnormal judgment module: If the second reconstruction error is less than the first threshold, the flow data to be detected is normal flow data; if the second reconstruction error is greater than the first threshold and less than or equal to the second threshold, the flow data to be detected is known abnormal flow data; if the second reconstruction error is greater than the second threshold, the flow data to be detected is unknown abnormal flow data.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for detecting abnormal advertising traffic according to any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the abnormal advertising traffic detection method according to any one of claims 1 to 5 is implemented.

Images