A method and device for detecting anomaly in high-dimensional network data

Abstract

The present invention discloses a method and device for detecting anomalies in high-dimensional network data, the method comprising obtaining high-dimensional data to be detected and a normal data set; preprocessing the normal data set to obtain a preprocessed data set; using the preprocessed data set to train a preset high-dimensional network data anomaly detection model to obtain the trained high-dimensional network data anomaly detection model; using the trained high-dimensional network data anomaly detection model to process the preprocessed data set and the high-dimensional data to be detected, respectively, to obtain a normal characterization variable set, a normal latent variable set, a characterization variable to be detected, and a latent variable to be detected; processing the normal characterization variable set, the normal latent variable set, the characterization variable to be detected, and the latent variable to be detected to obtain a detection result. The method of the present invention can improve the reasoning accuracy, detect abnormal data more accurately, and improve the accuracy of high-dimensional network data anomaly detection.

Description

Technical Field

[0001] The present invention belongs to the field of abnormal data detection, and specifically relates to a high-dimensional network data abnormality detection method and device. Background Art

[0002] With the rapid development and widespread application of information technology, computers and the Internet are widely used in various fields such as finance, Internet, military and medical care, which makes the data flow in the network continue to grow and show diversified characteristics, gradually forming a large-scale high-dimensional data flow. At the same time, various attacks on computer networks are increasing and becoming more complex. If they cannot be detected in a timely and effective manner, they will seriously endanger the normal operation of the network and have a very adverse impact on people's lives and work. Therefore, in network operation and maintenance, it is of wide application value to discover abnormal data in large-scale high-dimensional data flows to determine whether an emergency has occurred.

[0003] With the enhancement of algorithms and computing power and the accumulation of massive data, data anomaly detection is expanding from traditional machine learning methods to artificial intelligence deep learning methods. At present, data anomaly detection methods based on generative models such as long short-term memory networks, variational autoencoder neural networks, and generative adversarial neural networks have emerged. Through the learning of massive data, detection methods based on generative deep learning models can intelligently "learn" the hidden patterns and implicit internal laws in the data, and are not easily affected by noise.

[0004] In the field of machine learning, the factors that generate high-dimensional raw observation data are usually considered to be concentrated in a low-dimensional manifold space in the raw data space, which is a set of latent variables that generate the raw observation data. The latent variables contain the interpretable factors of the raw observation data, and different combinations of latent variables create different observation data. For abnormal data, the relationship between the dimensions is chaotic and disordered, and does not imply a specific pattern, or the structure of the pattern is destroyed by some abnormal dimensions. Therefore, by extracting the latent variables of the raw data, it is possible to determine whether the data is abnormal.

[0005] Although generative deep learning models have the ability to "learn" latent variables, the existing generative deep learning models all judge whether the input data is abnormal by comparing the Euclid distance between the model output data and the original input data. The main shortcoming is that in high-dimensional space, the Euclid distance cannot accurately and comprehensively reflect the difference between the two data, so the detection accuracy needs to be improved. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a method and device for detecting anomalies in high-dimensional network data, which is beneficial to improving the accuracy of anomaly detection in high-dimensional network data.

[0007] In order to solve the above technical problems, an embodiment of the present invention discloses a method for detecting anomalies in high-dimensional network data, the method comprising:

[0008] S1. Obtain high-dimensional data to be detected and a normal data set; the normal data set includes N normal high-dimensional data; the dimensions of the high-dimensional data to be detected and the normal high-dimensional data are both L; and L and N are both integers greater than 1;

[0009] S2, preprocessing the normal data set to obtain a preprocessed data set; the preprocessed data set includes N preprocessed high-dimensional data; the dimension of the preprocessed high-dimensional data is L;

[0010] S3, using the preprocessed data set, processing the preset first anomaly detection model and the second anomaly detection model to obtain a third anomaly detection model;

[0011] S4, using the third anomaly detection model, respectively processing the preprocessed data set and the high-dimensional data to be detected to obtain a normal characterization variable set, a normal latent variable set, a characterization variable to be detected, and a latent variable to be detected;

[0012] S5. Process the normal characterization variable set, the normal latent variable set, the characterization variables to be detected, and the latent variables to be detected to obtain a detection result.

[0013] As an optional implementation manner, in the first aspect of the embodiment of the present invention, preprocessing the normal data set to obtain the preprocessed data set includes:

[0014] S21, initializing the first loop number p to 1;

[0015] S22, obtaining the maximum value of the p-th dimension component of all the normal high-dimensional data in the normal data set to obtain a first coefficient;

[0016] S23, obtaining the minimum value of the p-th dimension component of all the normal high-dimensional data in the normal data set to obtain a second coefficient;

[0017] S24, using the preprocessing model, updating each normal high-dimensional data in the normal data set to obtain an updated normal data set;

[0018] The preprocessing model is:

[0019]

[0020] In the formula, x p and x p *are the values ​​of the p-th dimensional components before and after each normal high-dimensional data is updated, x a and x b are the first coefficient and the second coefficient respectively;

[0021] S25, adding 1 to the value of the first cycle number;

[0022] S26, repeat S22 to S25 until the first cycle number is greater than N;

[0023] S27, setting the pre-processed data set as the updated normal data set.

[0024] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the first anomaly detection model includes a first encoder module and a decoder module;

[0025] The input end of the first encoder module inputs the preprocessed high-dimensional data; the output end of the first encoder module outputs the initial latent variable;

[0026] The input end of the decoder module is connected to the output end of the first encoder module; the output end of the decoder module outputs the reconstructed data;

[0027] The second anomaly detection model comprises a forward diffusion module, a backward generation module and a second encoder module;

[0028] The input end of the forward diffusion module is connected to the output end of the first encoder module; the output end of the forward diffusion module outputs the diffusion variable; the forward diffusion module is used to process the initial latent variable input by the first encoder module to obtain the diffusion variable;

[0029] The input end of the backward generation module is connected to the output end of the forward diffusion module; the output end of the backward generation module outputs the hidden variable;

[0030] The input end of the second encoder module is connected to the output end of the backward generation module; the output end of the second encoder module outputs the representation variable.

[0031] As an optional implementation, in the first aspect of the embodiment of the present invention, the using of the preprocessed data set to process the preset first anomaly detection model and the second anomaly detection model to obtain a third anomaly detection model includes:

[0032] S31, using the preprocessed data set, training the preset first anomaly detection model to obtain the trained first anomaly detection model;

[0033] S32, using the first encoder module of the first anomaly detection model after training, respectively process the N preprocessed high-dimensional data in the preprocessed data set to obtain N initial latent variables; the N initial latent variables constitute an initial latent variable sample set;

[0034] S33, using the initial latent variable sample set, training the preset second anomaly detection model to obtain the trained second anomaly detection model;

[0035] S34: Process the trained first anomaly detection model and the second anomaly detection model to obtain the third anomaly detection model.

[0036] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the trained first anomaly detection model and the second anomaly detection model to obtain the third anomaly detection model includes:

[0037] S341, obtaining the trained first encoder module of the first anomaly detection model to obtain a third encoder module;

[0038] S342: Obtain the trained second encoder module of the second anomaly detection model to obtain a fourth encoder module; the output end of the fourth encoder module outputs the calculated representation variable;

[0039] S343, obtaining the trained backward generation module of the second anomaly detection model to obtain a calculation generation module; the output end of the calculation generation module outputs the calculation hidden variable;

[0040] S344. Connect the output end of the third encoder module to the input end of the fourth encoder module, and connect the output end of the fourth encoder module to the calculation generation module to obtain the third anomaly detection model.

[0041] As an optional implementation, in the first aspect of the embodiment of the present invention, the use of the third anomaly detection model to process the preprocessed data set and the high-dimensional data to be detected respectively to obtain a normal characterization variable set, a normal latent variable set, a characterization variable to be detected, and a latent variable to be detected includes:

[0042] S41, initializing the current representation variable set and the current latent variable set to empty sets, and initializing the second loop number q to 1;

[0043] S42, initializing the current high-dimensional data to the qth pre-processed high-dimensional data in the pre-processed data set;

[0044] S43, inputting the current high-dimensional data into the input end of the third anomaly detection model; using the third anomaly detection model, processing the current high-dimensional data to obtain the calculation representation variables and calculation hidden variables corresponding to the current high-dimensional data;

[0045] S44, adding the calculated representation variables and the calculated latent variables corresponding to the current high-dimensional data to the current representation variable set and the current latent variable set, respectively, to obtain the updated current representation variable set and the current latent variable set;

[0046] S45, adding 1 to the value of the second cycle number;

[0047] S46, repeating S42 to S45 until the second cycle number is equal to the length N of the preprocessed data set; setting the normal characterization variable set and the normal latent variable set as the updated current characterization variable set and the current latent variable set, respectively;

[0048] S47, inputting the high-dimensional data to be detected into the input end of the third anomaly detection model; using the third anomaly detection model, processing the high-dimensional data to be detected to obtain the calculated representation variables and calculated latent variables corresponding to the high-dimensional data to be detected;

[0049] S48. Set the to-be-detected representation variable and the to-be-detected latent variable as the calculated representation variable and the calculated latent variable corresponding to the to-be-detected high-dimensional data, respectively.

[0050] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the normal characterization variable set, the normal latent variable set, the characterization variable to be detected and the latent variable to be detected to obtain a detection result includes:

[0051] S51, processing the normal representation variable set and the normal latent variable set to obtain a first mean, a second mean, a first intermediate matrix and a second intermediate matrix;

[0052] S52, using an outlier calculation model, processing the first mean, the second mean, the first intermediate matrix, the second intermediate matrix, the latent variable to be detected, and the characterization variable to be detected to obtain a first outlier and a second outlier;

[0053] The outlier calculation model is:

[0054]

[0055] In the formula, S 1 and S 2are the first abnormal value and the second abnormal value respectively, z is the latent variable to be detected, r is the characterization variable to be detected, μ 1 and μ 2 are the first mean and the second mean respectively, M 1 and M 2 are the first intermediate matrix and the second intermediate matrix respectively;

[0056] S53: Using an abnormal evaluation value calculation model, process the first abnormal value and the second abnormal value to obtain an abnormal evaluation value:

[0057] The abnormal evaluation value calculation model is:

[0058] S=αS 1 +(1-α)S 2

[0059] Where α is the preset weighting coefficient;

[0060] S54, judging the abnormal evaluation value to obtain a detection result;

[0061] When the abnormality assessment value is less than or equal to a preset abnormality threshold, determining that the detection result is normal;

[0062] When the abnormality evaluation value is greater than the abnormality threshold, the detection result is determined to be abnormal.

[0063] As an optional implementation manner, in the first aspect of the embodiment of the present invention, the processing of the normal characterization variable set and the normal latent variable set to obtain the first mean, the second mean, the first intermediate matrix and the second intermediate matrix includes:

[0064] S511, respectively calculating the means of the normal latent variable set and the normal representation variable set to obtain the first mean and the second mean;

[0065] S512, using an intermediate matrix calculation model, processing the normal latent variable set, the normal representation variable set, the first mean, and the second mean to obtain the first intermediate matrix and the second intermediate matrix;

[0066] The intermediate matrix calculation model is:

[0067]

[0068] Where M 1 and M 2 are the first intermediate matrix and the second intermediate matrix respectively, μ 1 and μ 2 are the first mean and the second mean respectively, zi is the i-th element of the normal latent variable set, r j is the j-th element in the normal representation variable set, and N is the length of the normal latent variable set and the normal representation variable set.

[0069] A second aspect of an embodiment of the present invention discloses another high-dimensional network data anomaly detection device, the device comprising:

[0070] A memory storing executable program code;

[0071] a processor coupled to the memory;

[0072] The processor calls the executable program code stored in the memory to execute part or all of the steps in the high-dimensional network data anomaly detection method disclosed in the first aspect of the embodiment of the present invention.

[0073] The third aspect of the present invention discloses a computer storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute some or all of the steps in the high-dimensional network data anomaly detection method disclosed in the first aspect of an embodiment of the present invention.

[0074] Compared with the prior art, the embodiments of the present invention have the following beneficial effects:

[0075] The present invention organically combines two generative deep learning models, the variational autoencoder model and the diffusion model, to learn the manifold space of high-dimensional data, which can significantly improve the reasoning accuracy. Through hidden variables and representation variables, the dimension of the low-dimensional manifold space of the data is effectively reduced, while the intrinsic pattern of the data can be captured deeply inside the data, thereby more accurately detecting abnormal data and improving the accuracy of anomaly detection of high-dimensional network data. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0077] Figure 1 It is a flow chart of a high-dimensional network data anomaly detection method disclosed in an embodiment of the present invention.

[0078] Figure 2 It is a structural schematic diagram of a first anomaly detection model of a high-dimensional network data anomaly detection method disclosed in an embodiment of the present invention.

[0079] Figure 3It is a structural schematic diagram of a second anomaly detection model of a high-dimensional network data anomaly detection method disclosed in an embodiment of the present invention.

[0080] Figure 4 It is a structural schematic diagram of a third anomaly detection model of a high-dimensional network data anomaly detection method disclosed in an embodiment of the present invention.

[0081] Figure 5 It is a structural schematic diagram of a high-dimensional network data anomaly detection device disclosed in an embodiment of the present invention.

[0082] Figure 6 It is a structural schematic diagram of another high-dimensional network data anomaly detection device disclosed in an embodiment of the present invention. DETAILED DESCRIPTION

[0083] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0084] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0085] In the description of the present invention, it should be noted that, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0086] Embodiment 1

[0087] See also Figure 1 , Figure 1 : is a flow chart of a method for detecting anomalies in high-dimensional network data disclosed in an embodiment of the present invention. Figure 1The described high-dimensional network data anomaly detection method is applied to the field of high-dimensional data anomaly detection, such as high-dimensional network data anomaly detection in the industrial Internet and the Internet of Things, and the embodiments of the present invention are not limited thereto. Figure 1 As shown, the method includes:

[0088] S1, obtaining high-dimensional data to be detected and a normal data set; the above normal data set includes N normal high-dimensional data; the dimensions of the above high-dimensional data to be detected and the normal high-dimensional data are both L; the above L and N are both integers greater than 1;

[0089] S2, preprocessing the normal data set to obtain a preprocessed data set; the preprocessed data set includes N preprocessed high-dimensional data; the dimension of the preprocessed high-dimensional data is L;

[0090] S3, using the preprocessed data set, processing the preset first anomaly detection model and the second anomaly detection model to obtain a third anomaly detection model;

[0091] S4. Using the third anomaly detection model, the preprocessed data set and the high-dimensional data to be detected are processed respectively to obtain a normal representation variable set, a normal latent variable set, a representation variable to be detected, and a latent variable to be detected;

[0092] S5. Process the normal representation variable set, the normal latent variable set, the to-be-detected representation variable, and the to-be-detected latent variable to obtain a detection result.

[0093] It should be noted that when the above L is greater than 3, the corresponding data is high-dimensional data, which is not limited in the embodiment of the present invention.

[0094] In an optional embodiment, the above-mentioned preprocessing of the normal data set to obtain the preprocessed data set includes:

[0095] S21, initializing the first loop number p to 1;

[0096] S22, obtaining the maximum value of the p-th dimension component of all normal high-dimensional data in the normal data set, and obtaining a first coefficient;

[0097] S23, obtaining the minimum value of the p-th dimension component of all normal high-dimensional data in the normal data set to obtain a second coefficient;

[0098] S24, using the preprocessing model, updating each normal high-dimensional data in the normal data set to obtain an updated normal data set;

[0099] The above preprocessing model is:

[0100]

[0101] In the formula, x pand x p * are the values ​​of the p-th dimension component before and after each normal high-dimensional data update, x a and x b are the first coefficient and the second coefficient respectively;

[0102] S25, adding 1 to the value of the first cycle number;

[0103] S26, repeat S22 to S25 until the first cycle number is greater than N;

[0104] S27, setting the preprocessed data set as the updated normal data set.

[0105] In another optional embodiment, Figure 2 As shown, the above-mentioned first anomaly detection model includes a first encoder module and a decoder module;

[0106] The input end of the first encoder module inputs the preprocessed high-dimensional data; the output end of the first encoder module outputs the initial latent variable;

[0107] Preferably, the first encoder module is L 1 Layer of fully connected residual network; the above L 1 Greater than 1;

[0108] The input end of the decoder module is connected to the output end of the first encoder module; the output end of the decoder module outputs the reconstructed data;

[0109] Preferably, the decoder module is an L 2 Layer of fully connected residual network; the above L 2 Greater than 1;

[0110] like Figure 3 As shown, the second anomaly detection model includes a forward diffusion module, a backward generation module and a second encoder module;

[0111] The input end of the forward diffusion module is connected to the output end of the first encoder module; the output end of the forward diffusion module outputs the diffusion variable; the forward diffusion module is used to process the initial latent variable input by the first encoder module to obtain the diffusion variable;

[0112] The input end of the backward generation module is connected to the output end of the forward diffusion module; the output end of the backward generation module outputs the hidden variable;

[0113] Preferably, the backward generation module is an L 4 Layer of fully connected residual network; the above L 4 Greater than 1;

[0114] The input end of the second encoder module is connected to the output end of the backward generation module; the output end of the second encoder module outputs the representation variable;

[0115] Preferably, the second encoder module is an L 3 Layer of fully connected residual network; the above L 3 Greater than 1.

[0116] In yet another optional embodiment, the initial latent variable input by the first encoder module is processed to obtain a diffusion variable, including:

[0117] A1, preset total number of steps T, first scale factor and second scale factor; the above T is an integer greater than 1;

[0118] A2. Initialize the diffusion variable and the current step number t to the initial latent variable and 1 respectively;

[0119] A3. Randomly initialize intermediate random variables;

[0120] Preferably, the intermediate random variable satisfies Gaussian distribution;

[0121] A4. Using the scale coefficient calculation model, the current number of steps, the total number of steps, the first scale coefficient and the second scale coefficient are processed to obtain the current scale coefficient;

[0122] The above scale coefficient calculation model is:

[0123]

[0124] In the formula, ε t , ε 1 , ε T are the current scale coefficient, the first scale coefficient, and the second scale coefficient respectively; T and t are the total number of steps and the current number of steps respectively;

[0125] A5. Use the diffusion variable to update the model, update the diffusion variable, and obtain the updated diffusion variable;

[0126] The above diffusion variable update model is

[0127]

[0128] In the formula, z t-1 and z t are the diffusion variables before and after the update, ε t is the current scale coefficient, τ is the intermediate random variable;

[0129] A6, add 1 to the current step value;

[0130] A7. Repeat A4 to A6 until the current number of steps is equal to T, and obtain the diffusion variable.

[0131] In yet another optional embodiment, the above-mentioned use of the preprocessed data set to process the preset first anomaly detection model and the second anomaly detection model to obtain a third anomaly detection model includes:

[0132] S31, using the preprocessed data set, training a preset first anomaly detection model to obtain a trained first anomaly detection model;

[0133] S32, using the trained first encoder module of the first anomaly detection model, respectively process the N preprocessed high-dimensional data in the preprocessed data set to obtain N initial latent variables; the above N initial latent variables constitute an initial latent variable sample set;

[0134] S33, using the initial latent variable sample set, training the preset second anomaly detection model to obtain a trained second anomaly detection model;

[0135] S34: Process the trained first anomaly detection model and the second anomaly detection model to obtain a third anomaly detection model.

[0136] In yet another optional embodiment, the preprocessed data set is used to train the preset first anomaly detection model to obtain the trained first anomaly detection model, including:

[0137] S311, randomly extract S from the preprocessed data set 1 preprocess the high-dimensional data to obtain the first small batch sample set; the above S 1 is the preset first sampling number;

[0138] S312, using the first encoder module, respectively 1 Preprocess the high-dimensional data to obtain S 1 Initial latent variables;

[0139] S313, using the decoder module, respectively 1 The initial latent variables are processed to obtain S 1 Reconstruction data;

[0140] S314, using the first generation value calculation model, processing the reconstructed data and the initial latent variables to obtain the first generation value;

[0141] The above first-generation value calculation model is:

[0142]

[0143] Where, L 1is the first generation value, E is the mean of the calculated vector elements, For S 1 The vector of reconstructed data, p(z) is the S of the standard normal distribution 1 A vector of randomly sampled values, For S 1 A vector of initial latent variables;

[0144] It should be noted that The reconstruction data of any number in The initial latent variables with the same number in are calculated;

[0145] S315, updating parameters and calculating gradient values ​​of the first anomaly detection model to obtain an updated first anomaly detection model and a first gradient value;

[0146] Preferably, the parameter update and gradient value calculation of the first anomaly detection model are performed using the Adam algorithm, and its expression is:

[0147]

[0148] In the formula, is the first gradient value calculated by the Adam algorithm, and are the parameters of the first encoder unit before and after the update, Φ and Φ * are the parameters of the decoder module before and after the update respectively;

[0149] S316, repeat N 1 Execute S311 to S315 again to obtain the trained first anomaly detection model; the above N 1 The preset first training number.

[0150] In another optional embodiment, the above-mentioned initial latent variable sample set is used to train the preset second anomaly detection model to obtain the trained second anomaly detection model, including:

[0151] S331, randomly select S from the initial latent variable sample set 2 Initial hidden variables, get the second small batch sample set; the above S 2 is the preset second sampling number;

[0152] S332, using the forward diffusion module, respectively 2 The initial latent variables are processed to obtain S 2 diffusion variables;

[0153] S333, using the backward generation module, respectively 2 diffusion variables are processed to obtain S2 Hidden variables;

[0154] S334, using the second encoder module, respectively 2 hidden variables are processed to obtain S 2 Characteristic variables;

[0155] S335, to S 2 The distance measurement is calculated by using the representation variables to obtain S 2 A distance measure;

[0156] S336, using the second generation value calculation model, the second small batch sample set, S 2 hidden variables and S 2 The distance measure DSW is processed to obtain the second generation value;

[0157] The above second-generation value calculation model is:

[0158] L 2 =E[z 0 -μ 2 ]-SW

[0159] Where, L 2 is the second generation value, E is the mean of the calculated vector elements, z 0 S is the second small batch of samples 2 The vector of initial latent variables, μ is S 2 A vector of hidden variables, SW is S 2 A vector of distance measures;

[0160] It should be noted that any numbered element in SW and μ is calculated from the initial latent variable corresponding to the number in the second mini-batch sample set;

[0161] S337, updating parameters and calculating gradient values ​​of the second anomaly detection model to obtain an updated second anomaly detection model and a second gradient value;

[0162] Preferably, the parameter update and gradient value calculation of the second anomaly detection model are performed using the Adam algorithm, and its expression is:

[0163]

[0164] In the formula, The second gradient value calculated by the Adam algorithm, θ and θ * are the parameters of the second encoder module before and after the update, η * and η are the parameters of the backward generation module before and after the update, respectively;

[0165] S338, repeat N2 Execute S331 to S337 again to obtain the trained second anomaly detection model; the above N 2 The second training number is preset.

[0166] In yet another optional embodiment, the above-mentioned S 2 The distance measurement is calculated by using the representation variables to obtain S 2 A distance measure is used to perform the following operations on each representation variable:

[0167] S3351, respectively, characterize the variable q θ Perform Randon transformation with any standard normal distribution p(r) to obtain the first variable Rq θ (τ, ω) and the second variable Rp(r)(τ, ω); the specific expression of Randon transform is:

[0168]

[0169] Where δ is the unit pulse function, M is the preset number of sampling times, and the first variable and the second variable are both functions of τ and ω, where τ∈R, ω∈S d-1 , S d-1 is a d-dimensional unit spherical space, where d is a preset integer less than L; r m For the mth time from S 2 A vector consisting of d randomly sampled representation variables among the representation variables; is the vector of sample values ​​obtained by sampling p(r) d times for the mth time;

[0170] S3352. Using the distance measurement calculation model, the first variable and the second variable are processed to obtain a distance measurement value. The calculation expression is:

[0171]

[0172] It should be noted that the above distance measurement, based on the characteristics of manifold space, uses the Wasserstein distance measurement to more accurately measure the "distance" between probability distributions and improve the accuracy of detection.

[0173] In yet another optional embodiment, the above-mentioned processing of the trained first anomaly detection model and the second anomaly detection model to obtain a third anomaly detection model includes:

[0174] S341, obtaining a first encoder module of a trained first anomaly detection model to obtain a third encoder module;

[0175] S342, obtaining the trained second encoder module of the second anomaly detection model to obtain a fourth encoder module; the output end of the fourth encoder module outputs the calculated representation variable;

[0176] S343, obtaining the backward generation module of the trained second anomaly detection model to obtain a calculation generation module; the output end of the calculation generation module outputs the calculation hidden variable;

[0177] S344, connect the output end of the third encoder module to the input end of the fourth encoder module, and connect the output end of the fourth encoder module to the calculation generation module, so as to obtain Figure 4 The third anomaly detection model shown.

[0178] It should be noted that the input end of the third encoder module is configured as the input end of the third anomaly detection model; the output end of the calculation generation module is configured as the output end of the third anomaly detection model.

[0179] In another optional embodiment, the third anomaly detection model is used to process the preprocessed data set and the high-dimensional data to be detected, respectively, to obtain a normal characterization variable set, a normal latent variable set, a characterization variable to be detected, and a latent variable to be detected, including:

[0180] S41, initializing the current representation variable set and the current latent variable set to empty sets, and initializing the second loop number q to 1;

[0181] S42, initializing the current high-dimensional data to the qth pre-processed high-dimensional data in the pre-processed data set;

[0182] S43, inputting the current high-dimensional data into the input end of the third anomaly detection model; using the third anomaly detection model, processing the current high-dimensional data to obtain the calculation representation variables and calculation hidden variables corresponding to the current high-dimensional data;

[0183] S44, adding the calculated representation variables and the calculated latent variables corresponding to the current high-dimensional data to the current representation variable set and the current latent variable set, respectively, to obtain an updated current representation variable set and current latent variable set;

[0184] S45, adding 1 to the value of the second cycle number;

[0185] S46, repeat S42 to S45 until the second cycle number is equal to the length N of the preprocessed data set; set the normal representation variable set and the normal latent variable set as the updated current representation variable set and the current latent variable set, respectively;

[0186] It should be noted that the above normal representation variable set and normal latent variable set are equal to the length of the preprocessed data set, both containing N elements;

[0187] S47, inputting the high-dimensional data to be detected into the input end of the third anomaly detection model; using the third anomaly detection model, processing the high-dimensional data to be detected to obtain the calculated representation variables and the calculated latent variables corresponding to the high-dimensional data to be detected;

[0188] S48. Set the characterization variable to be detected and the latent variable to be detected as the calculated characterization variable and the calculated latent variable corresponding to the high-dimensional data to be detected, respectively.

[0189] In yet another optional embodiment, the above processing of the normal characterization variable set, the normal latent variable set, the characterization variable to be detected and the latent variable to be detected to obtain the detection result includes:

[0190] S51, processing the normal representation variable set and the normal latent variable set to obtain a first mean, a second mean, a first intermediate matrix, and a second intermediate matrix;

[0191] S52, using an outlier calculation model, processing the first mean, the second mean, the first intermediate matrix, the second intermediate matrix, the latent variable to be detected, and the characterization variable to be detected to obtain a first outlier and a second outlier;

[0192] The above outlier calculation model is:

[0193]

[0194] In the formula, S 1 and S 2 are the first and second outliers respectively, z is the latent variable to be detected, r is the representation variable to be detected, μ 1 and μ 2 are the first mean and the second mean respectively, M 1 and M 2 are respectively the first intermediate matrix and the second intermediate matrix;

[0195] S53, using the abnormal evaluation value calculation model, processing the first abnormal value and the second abnormal value to obtain an abnormal evaluation value:

[0196] The above abnormal evaluation value calculation model is:

[0197] S=αS 1 +(1-α)S 2

[0198] Where α is the preset weighting coefficient;

[0199] Preferably, the value of α is between 0 and 1;

[0200] S54, judging the abnormal evaluation value and obtaining the detection result;

[0201] When the abnormality assessment value is less than or equal to the preset abnormality threshold, the detection result is determined to be normal;

[0202] When the abnormality evaluation value is greater than the abnormality threshold, the detection result is determined to be abnormal.

[0203] In yet another optional embodiment, the above processing of the normal characterization variable set and the normal latent variable set to obtain the first mean, the second mean, the first intermediate matrix and the second intermediate matrix includes:

[0204] S511, respectively calculating the means of a normal latent variable set and a normal representation variable set to obtain a first mean and a second mean;

[0205] S512, using the intermediate matrix calculation model, processing the normal latent variable set, the normal representation variable set, the first mean, and the second mean to obtain a first intermediate matrix and a second intermediate matrix;

[0206] The above intermediate matrix calculation model is:

[0207]

[0208] Where M 1 and M 2 are the first intermediate matrix and the second intermediate matrix, μ 1 and μ 2 are the first mean and the second mean, respectively, z i is the i-th element of the normal latent variable set, r j is the jth element in the normal representation variable set, and N is the length of the normal latent variable set and the normal representation variable set.

[0209] It can be seen that the high-dimensional network data anomaly detection method described in the embodiment of the present invention organically combines the two generative deep learning models of the variational autoencoder model and the diffusion model to learn the manifold space of high-dimensional data, which can significantly improve the reasoning accuracy; through hidden variables and representation variables, the dimension of the low-dimensional manifold space of the data is effectively reduced, and the intrinsic pattern of the data can be deeply captured inside the data, thereby more accurately detecting abnormal data and improving the accuracy of detection.

[0210] Embodiment 2

[0211] See also Figure 5 , Figure 5 Schematic diagram of a high-dimensional network data anomaly detection device disclosed in an embodiment of the present invention. Figure 5 The high-dimensional network data anomaly detection device described is applied to the field of high-dimensional data anomaly detection, such as high-dimensional network data anomaly detection in the industrial Internet and the Internet of Things, and the embodiments of the present invention are not limited thereto. Figure 5As shown, the high-dimensional network data anomaly detection device includes:

[0212] The data acquisition and preprocessing module 201 is used to acquire the high-dimensional data to be detected and the normal data set, and preprocess the normal data set to obtain the preprocessed data set;

[0213] A model training module 202 is used to process the preset first anomaly detection model and the second anomaly detection model using the preprocessed data set to obtain a third anomaly detection model;

[0214] The model calculation module 203 is used to use the third anomaly detection model to process the preprocessed data set and the high-dimensional data to be detected, respectively, to obtain a normal characterization variable set, a normal latent variable set, a characterization variable to be detected, and a latent variable to be detected;

[0215] The detection module 204 is used to process the normal characterization variable set, the normal latent variable set, the characterization variables to be detected and the latent variables to be detected to obtain a detection result.

[0216] It can be seen that the high-dimensional network data anomaly detection method described in the embodiment of the present invention organically combines the two generative deep learning models of the variational autoencoder model and the diffusion model to learn the manifold space of high-dimensional data, which can significantly improve the reasoning accuracy; through hidden variables and representation variables, the dimension of the low-dimensional manifold space of the data is effectively reduced, and the intrinsic pattern of the data can be captured deeply inside the data, thereby more accurately detecting abnormal data and improving the accuracy.

[0217] Embodiment 3

[0218] See also Figure 6 , Figure 6 is a schematic diagram of the structure of another high-dimensional network data anomaly detection device disclosed in an embodiment of the present invention. Figure 6 The high-dimensional network data anomaly detection device described can be applied to the field of high-dimensional data anomaly detection, such as high-dimensional network data anomaly detection in the industrial Internet and the Internet of Things, and the embodiments of the present invention are not limited thereto. Figure 6 As shown, the high-dimensional network data anomaly detection device may include the following parts:

[0219] A memory 301 storing executable program codes;

[0220] a processor 302 coupled to the memory 301;

[0221] The processor 302 calls the executable program code stored in the memory 301 to execute the steps in the high-dimensional network data anomaly detection method described in the first embodiment.

[0222] Embodiment 4

[0223] An embodiment of the present invention discloses a computer-readable storage medium storing a computer program for electronic data exchange, wherein the computer program enables a computer to execute the steps of the high-dimensional network data anomaly detection method described in the first embodiment.

[0224] The device embodiments described above are only illustrative, wherein the modules described as separate components may or may not be physically separated, and the components displayed as modules may or may not be physical modules, i.e., they may be located in one place, or they may be distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative work.

[0225] Through the specific description of the above embodiments, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the above technical solution can be essentially or partly contributed to the prior art in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, and the storage medium includes a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable rewritable read-only memory (EEPROM), a compact disc (CD-ROM) or other optical disc storage, a magnetic disk storage, a magnetic tape storage, or any other computer-readable medium that can be used to carry or store data.

[0226] Finally, it should be noted that the high-dimensional network data anomaly detection method and device disclosed in the embodiments of the present invention disclose only the preferred embodiments of the present invention, which are only used to illustrate the technical solution of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features therein can be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A high-dimensional network data anomaly detection method, characterized in that: include: S1. Obtain the high-dimensional data to be detected and the normal data set; The normal data set includes N normal high-dimensional data; The dimensions of the high-dimensional data to be detected and the normal high-dimensional data are both L; Both L and N are integers greater than 1; S2, preprocessing the normal data set to obtain a preprocessed data set; The preprocessed data set includes N preprocessed high-dimensional data; The dimension of the preprocessed high-dimensional data is L; S3. Using the preprocessed data set, training a preset first anomaly detection model to obtain the trained first anomaly detection model; The first anomaly detection model comprises a first encoder module and a decoder module; The input end of the first encoder module inputs the preprocessed high-dimensional data; the output end of the first encoder module outputs the initial latent variable; The input end of the decoder module is connected to the output end of the first encoder module; the output end of the decoder module outputs the reconstructed data; S4, using the first encoder module of the trained first anomaly detection model to process the N preprocessed high-dimensional data in the preprocessed data set respectively to obtain N initial latent variables; The N initial latent variables constitute an initial latent variable sample set; S5. Using the initial latent variable sample set, training a preset second anomaly detection model to obtain a trained second anomaly detection model; The second anomaly detection model comprises a forward diffusion module, a backward generation module and a second encoder module; The input end of the forward diffusion module is connected to the output end of the first encoder module; the output end of the forward diffusion module outputs the diffusion variable; The forward diffusion module is used to process the initial latent variables input by the first encoder module to obtain diffusion variables; The input end of the backward generation module is connected to the output end of the forward diffusion module; the output end of the backward generation module outputs the hidden variable; The input end of the second encoder module is connected to the output end of the backward generation module; the output end of the second encoder module outputs the representation variable; S6. Obtain the trained first encoder module of the first anomaly detection model to obtain a third encoder module; S7, obtaining the trained second encoder module of the second anomaly detection model to obtain a fourth encoder module; the output end of the fourth encoder module outputs the calculated representation variable; S8, obtaining the trained backward generation module of the second anomaly detection model to obtain a calculation generation module; the output end of the calculation generation module outputs the calculation hidden variable; S9, connecting the output end of the third encoder module to the input end of the fourth encoder module, and connecting the output end of the fourth encoder module to the calculation generation module to obtain a third anomaly detection model; S10, using the third anomaly detection model, respectively processing the preprocessed data set and the high-dimensional data to be detected to obtain a normal characterization variable set, a normal latent variable set, a characterization variable to be detected, and a latent variable to be detected; S11. Process the normal characterization variable set, the normal latent variable set, the characterization variables to be detected, and the latent variables to be detected to obtain a detection result.

2. The high-dimensional network data anomaly detection method according to claim 1, characterized in that: The preprocessing of the normal data set to obtain a preprocessed data set includes: S21, initializing the first loop number p to 1; S22, obtaining the maximum value of the p-th dimension component of all the normal high-dimensional data in the normal data set to obtain a first coefficient; S23, obtaining the minimum value of the p-th dimension component of all the normal high-dimensional data in the normal data set to obtain a second coefficient; S24, using the preprocessing model, updating each normal high-dimensional data in the normal data set to obtain an updated normal data set; The preprocessing model is: In the formula, x p and are the values ​​of the p-th dimensional components before and after each normal high-dimensional data is updated, x a and x b are the first coefficient and the second coefficient respectively; S25, adding 1 to the value of the first cycle number; S26, repeat S22 to S25 until the first cycle number is greater than N; S27, setting the pre-processed data set as the updated normal data set.

3. The high-dimensional network data anomaly detection method according to claim 1, characterized in that: The method of using the third anomaly detection model to process the preprocessed data set and the high-dimensional data to be detected respectively to obtain a normal characterization variable set, a normal latent variable set, a characterization variable to be detected, and a latent variable to be detected includes: S101, initializing the current representation variable set and the current latent variable set to empty sets, and initializing the second loop number q to 1; S102, initializing the current high-dimensional data to the qth pre-processed high-dimensional data in the pre-processed data set; S103, inputting the current high-dimensional data into the input end of the third anomaly detection model; using the third anomaly detection model, processing the current high-dimensional data to obtain calculation representation variables and calculation latent variables corresponding to the current high-dimensional data; S104, adding the calculated representation variables and the calculated latent variables corresponding to the current high-dimensional data to the current representation variable set and the current latent variable set, respectively, to obtain the updated current representation variable set and the current latent variable set; S105, adding 1 to the value of the second cycle number; S106, repeating S102 to S105 until the second cycle number is equal to the length N of the preprocessed data set; setting the normal characterization variable set and the normal latent variable set as the updated current characterization variable set and the current latent variable set, respectively; S107, inputting the high-dimensional data to be detected into the input end of the third anomaly detection model; using the third anomaly detection model, processing the high-dimensional data to be detected to obtain the calculated representation variables and calculated latent variables corresponding to the high-dimensional data to be detected; S108. Set the to-be-detected representation variable and the to-be-detected latent variable as the calculated representation variable and the calculated latent variable corresponding to the to-be-detected high-dimensional data, respectively.

4. The high-dimensional network data anomaly detection method according to claim 1, characterized in that: The processing of the normal characterization variable set, the normal latent variable set, the characterization variable to be detected and the latent variable to be detected to obtain a detection result includes: S111, processing the normal representation variable set and the normal latent variable set to obtain a first mean, a second mean, a first intermediate matrix, and a second intermediate matrix; S112, using an outlier calculation model, processing the first mean, the second mean, the first intermediate matrix, the second intermediate matrix, the latent variable to be detected, and the characterization variable to be detected to obtain a first outlier and a second outlier; The outlier calculation model is: Wherein, S1 and S2 are the first abnormal value and the second abnormal value respectively, z is the latent variable to be detected, r is the characterization variable to be detected, μ1 and μ2 are the first mean and the second mean respectively, M1 and M2 are the first intermediate matrix and the second intermediate matrix respectively; S113: Using an abnormal evaluation value calculation model, process the first abnormal value and the second abnormal value to obtain an abnormal evaluation value: The abnormal evaluation value calculation model is: S=αS1+(1-α)S2 Where α is the preset weighting coefficient; S114, judging the abnormal evaluation value to obtain a detection result; When the abnormality assessment value is less than or equal to a preset abnormality threshold, determining that the detection result is normal; When the abnormality evaluation value is greater than the abnormality threshold, the detection result is determined to be abnormal.

5. The high-dimensional network data anomaly detection method according to claim 4, characterized in that: The processing of the normal characterization variable set and the normal latent variable set to obtain a first mean, a second mean, a first intermediate matrix and a second intermediate matrix includes: S1111. Calculate the means of the normal latent variable set and the normal representation variable set respectively to obtain the first mean and the second mean; S1112. Using an intermediate matrix calculation model, process the normal latent variable set, the normal representation variable set, the first mean, and the second mean to obtain the first intermediate matrix and the second intermediate matrix; The intermediate matrix calculation model is: Wherein, M1 and M2 are the first intermediate matrix and the second intermediate matrix, μ1 and μ2 are the first mean and the second mean, respectively, z i is the i-th element of the normal latent variable set, r j is the j-th element in the normal representation variable set, and N is the length of the normal latent variable set and the normal representation variable set.

6. A high-dimensional network data anomaly detection device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the high-dimensional network data anomaly detection method as described in any one of claims 1-5.

7. A computer storage medium, characterized in that: The computer storage medium stores computer instructions, which, when called, are used to execute the high-dimensional network data anomaly detection method as described in any one of claims 1 to 5.

Images