A segmented data detection method and corresponding apparatus

Abstract

The present application relates to the technical field of network security, and provides a segmented data detection method and a corresponding device. First, the data to be detected is obtained, and the data to be detected is vectorized to obtain a corresponding target encoding vector. Then, the target encoding vector is processed to obtain an anomaly score value of the data to be detected. Then, the anomaly score value is compared with a segmentation threshold value, and according to the comparison result, it is determined that the data to be detected is normal data or suspicious data. If the data to be detected is suspicious data, the encoding vector is input into a classification model, and the classification model is used to determine whether the data to be detected is normal data or malicious data. The present application uses a two-stage detection scheme to detect data, quickly filters out normal data, and then detects the remaining small amount of suspicious data through a classification model to obtain a higher detection accuracy. The present application can quickly determine whether the data to be detected is malicious data, and can also ensure the detection accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network security, in particular to a segmented data detection method and a corresponding device. BACKGROUND

[0002] With the continuous development of the Internet, people's work and life has been inseparable from the benefits brought by the network. Criminals in order to steal information for profit, are more and more active on the Internet. They carry out network attacks through various ways, including SQL injection (Structured Query Language), XSS attack (Cross Site Scripting), phishing website, etc. Therefore, if the traffic data can be detected and distinguished, and the malicious traffic data is found to be warned and intercepted, the attacker will not be easy to attack successfully, and the user loss will be greatly reduced. In daily work and life, malicious traffic data is generally less, and there are mainly normal traffic data. However, if the malicious traffic data is not detected and intercepted in time, it may cause user loss. In view of this situation, the traffic data needs to be detected quickly to ensure the detection efficiency and improve the user experience. At the same time, the detection accuracy should be ensured to reduce the loss caused by misjudgment.

[0003] At present, the traffic data detection method mainly detects in one step, and the detection method includes a traditional machine learning method based on SVM (Support Vector Machine) and a deep learning method based on a neural network model. The deep learning method is more used because of its high detection accuracy, but the detection is time-consuming, and the real-time detection efficiency of a large amount of traffic data is low.

[0004] Therefore, it is urgent to overcome the defects of the prior art in the technical field. SUMMARY

[0005] The technical problem to be solved by the present application is to provide a segmented data detection method to solve the problem of low detection efficiency when a large amount of traffic data is detected in real time in the prior art.

[0006] The present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a segmented data detection method, comprising:

[0008] Obtaining to-be-detected data, and vectorizing the to-be-detected data to obtain a corresponding target encoding vector;

[0009] Processing the target encoding vector to obtain an abnormal score value of the to-be-detected data;

[0010] comparing the abnormal score value with a segmentation threshold value, and determining, according to a comparison result, whether the to-be-detected data is normal data or suspicious data;

[0011] If the to-be-detected data is suspicious data, inputting the target encoding vector into a classification model, and determining, by the classification model, whether the to-be-detected data is normal data or malicious data.

[0012] Further, the training method of the classification model comprises:

[0013] obtaining training data, wherein the training data is divided into normal samples and malicious samples, and the training data is vectorized to obtain corresponding training encoding vectors;

[0014] training the classification model by using the training encoding vectors until a loss value of the classification model is reduced to an expected value.

[0015] Further, the training data is divided into normal samples and malicious samples, and the normal samples and the malicious samples correspond to different class labels;

[0016] The calculation process of the loss value of the classification model comprises:

[0017] calculating an original variance and an original mean value of the training encoding vectors;

[0018] obtaining a class label corresponding to the training encoding vectors, taking the original mean value of the training encoding vectors as a base number, and taking the class label as a power to obtain a target mean value;

[0019] calculating a divergence loss according to the original mean value, the target mean value, and the original variance;

[0020] determining a loss value according to the divergence loss.

[0021] Further, the determining of the loss value according to the divergence loss comprises:

[0022] reconstructing the original variance and the original mean value into a reconstructed vector, calculating a reconstruction error according to the reconstructed vector and a corresponding training encoding vector;

[0023] determining a category to which the reconstructed vector belongs by using a binary classification layer of the classification model, and calculating a binary classification cross-entropy loss;

[0024] taking a sum of the divergence loss, the reconstruction error, and the binary classification cross-entropy loss as the loss value.

[0025] Further, the processing of the target encoding vector to obtain an abnormal score value of the to-be-detected data comprises:

[0026] For each feature dimension of the target coding vector, determine the value range of data in the feature dimension;

[0027] Divide the value range of data into K consecutive intervals, and calculate the frequency of data falling into each interval to establish a data histogram for each feature dimension;

[0028] According to the data histogram, calculate the probability density of the corresponding feature dimension, and take the sum of the probability densities of all feature dimensions as the anomaly score value of the data to be detected.

[0029] Further, the acquisition process of the segmentation threshold is:

[0030] Obtain training data, divide the training data into normal samples and malicious samples, and vectorize all training data to obtain corresponding training coding vectors;

[0031] Calculate the score results of each normal sample and each malicious sample;

[0032] Determine the segmentation threshold according to the score of the normal sample and the score of the malicious sample.

[0033] Further, the segmentation threshold is determined according to the score of the normal sample and the score of the malicious sample, including:

[0034] Take the median of the anomaly score values of the normal samples as A;

[0035] In the anomaly score values of the malicious samples, take the anomaly score values ranked in the preset position in size order as B, and set the segmentation threshold to the smaller value of A and B.

[0036] Further, the acquisition of the data to be detected and the vectorization of the data to be detected to obtain the corresponding target coding vector includes:

[0037] Iterate through the sample library and construct a character dictionary according to the sample library;

[0038] Iterate through the data to be detected to obtain the characters of the data to be detected;

[0039] Calculate the number of times each character of the data to be detected appears in the data to be detected and the number of times it appears in the sample library;

[0040] According to the number of times each character appears in the data to be detected and the number of times it appears in the sample library, generate the target coding vector of the data to be detected.

[0041] Further, the classification model comprises an encoder, a decoder and a binary classification layer, if the to-be-detected data is suspicious data, the target encoding vector is input into the classification model, the target encoding vector is processed by the classification model to determine whether the to-be-detected data is normal data or malicious data, comprising:

[0042] The target encoding vector is encoded into the mean and variance of the latent variable by the encoder to determine the latent variable;

[0043] The latent variable is reconstructed into a reconstructed vector by the decoder;

[0044] The category of the reconstructed vector is determined by the binary classification layer to determine whether the corresponding to-be-detected data is normal data or malicious data.

[0045] In a second aspect, the present application also provides a segmented data detection device for implementing the segmented data detection method of the first aspect, the device comprising: at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions executable by the at least one processor, and the instructions are used to complete the segmented data detection method of the first aspect after being executed by the processor.

[0046] The present application uses a two-stage detection scheme to detect data, first vectorizes the to-be-detected data to obtain a corresponding target encoding vector, then scores the target encoding vector to obtain an abnormal score value of the to-be-detected data, then quickly filters out normal data with a large proportion from a large amount of to-be-detected data according to the abnormal score value and a segmentation threshold, and then detects a small part of suspicious data with high accuracy by using the classification model. This method can quickly determine whether the to-be-detected data is malicious data, and can also ensure the detection accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained from these drawings without creative labor.

[0048] Figure 1 is a flowchart of a segmented data detection method provided by an embodiment of the present application;

[0049] Figure 2 is a flowchart of a training method of a classification model in a segmented data detection method provided by an embodiment of the present application;

[0050] Figure 3 is a method for calculating a loss value in step 202 in the embodiment of the present application Figure 2 is a flowchart of the method for calculating a loss value in step 202 in the embodiment of the present application

[0051] Figure 4 is a method for calculating a loss value in step 202 in the embodiment of the present application Figure 2 is a flowchart of step 304 in the embodiment of the present application

[0052] Figure 5 is a method for calculating a loss value in step 202 in the embodiment of the present application Figure 1 is a flowchart of step 102 in the embodiment of the present application

[0053] Figure 6 is a method for calculating a loss value in step 202 in the embodiment of the present application Figure 1 is a flowchart of step 103 in the embodiment of the present application

[0054] Figure 7 is a method for calculating a loss value in step 202 in the embodiment of the present application Figure 1 is a flowchart of step 101 in the embodiment of the present application

[0055] Figure 8 is a schematic diagram of the architecture of a classification model of a segmented data detection method according to the embodiment of the present application

[0056] Figure 9 is a method for calculating a loss value in step 202 in the embodiment of the present application Figure 1 is a flowchart of step 104 in the embodiment of the present application

[0057] Figure 10 is a schematic diagram of the technical flow of a segmented data detection method according to the embodiment of the present application

[0058] Figure 11 is a schematic diagram of the architecture of a segmented data detection device according to the embodiment of the present application DETAILED DESCRIPTION

[0059] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0060] In the description of the present application, the terms "inner", "outer", "longitudinal", "transverse", "upper", "lower", "top", "bottom" and the like indicate the orientation or positional relationship shown in the drawings, and are only used to facilitate the description of the present application and do not require the present application to be constructed and operated in a particular orientation, therefore should not be understood as a limitation on the present application.

[0061] In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0062] Example 1:

[0063] Embodiment 1 of the present invention provides a segmented data detection method, combined with Figure 1 The process steps of the segmented data detection method include:

[0064] Step 101: Obtain the data to be detected and vectorize the data to be detected to obtain the corresponding target encoding vector.

[0065] The data to be detected can be URL data (Uniform Resource Locator), excluding the specific content within the webpage. The URL data is a string composed of various characters. In this embodiment, the TF-IDF algorithm (Term Frequency–Inverse Document Frequency) can be used to vectorize the data to be detected, generating specific statistical data from the data to be detected as the corresponding target encoding vector.

[0066] Of course, in other embodiments, the data to be detected may also be other types of data.

[0067] Step 102: Process the target encoding vector to obtain the anomaly score value of the data to be detected.

[0068] Specifically, an anomaly score value is calculated for the target encoding vector, and this anomaly score value is used as the anomaly score value for the corresponding data to be detected. This anomaly score value is a predicted value obtained after calculation using a specific algorithm.

[0069] Step 103: Compare the abnormal score with the segmentation threshold, and determine whether the data to be detected is normal data or suspicious data based on the comparison result.

[0070] The higher the anomaly score, the higher the degree of anomaly in the corresponding data to be detected; the segmentation threshold is derived from multiple sample data in the sample library; when the anomaly score is less than the segmentation threshold, the corresponding data to be detected is determined to be normal data, and when the anomaly score is not less than the segmentation threshold, the corresponding data to be detected is determined to be suspicious data.

[0071] Step 104: If the data to be detected is suspicious, the target encoding vector is input into the classification model, and the classification model determines whether the data to be detected is normal or malicious.

[0072] The plurality of sample data in the sample library is used as training data, deep learning is performed based on a neural network model, and then the classification model is obtained. In this embodiment, the classification model is trained based on a conditional variational autoencoder and a binary classification layer. The construction process of the classification model will be described in detail below.

[0073] The present application uses a two-stage detection scheme to detect data. First, the to-be-detected data is vectorized to obtain the corresponding target encoding vector. Then, the target encoding vector is scored to obtain the abnormal score value of the to-be-detected data. Then, according to the abnormal score value and the segmentation threshold, the normal data with a large proportion is quickly screened out from a large amount of to-be-detected data. Then, the remaining small part of suspicious data is detected with high accuracy by the classification model. This method can quickly determine whether the to-be-detected data is malicious data, and can also ensure the detection accuracy.

[0074] In this embodiment, the classification model is trained based on the training data in the sample library before being put into use. Figure 2 In order to determine whether the to-be-detected data is normal data or malicious data by the classification model, the classification model needs to be trained based on the training data in the sample library before being put into use. The training method of the classification model includes:

[0075] Step 201: Obtain training data, which is divided into normal samples and malicious samples, and vectorize all training data to obtain corresponding training encoding vectors.

[0076] The training data is known as normal samples or malicious samples. The training encoding vector includes a plurality of feature dimensions, and one feature dimension contains corresponding data for describing one feature of the training encoding vector.

[0077] Step 202: Train the classification model by using the training encoding vectors until the loss value of the classification model is reduced to an expected value.

[0078] The greater the loss value, the less accurate the judgment result of the classification model on the training data. During the training process of the classification model, the loss value gradually decreases to the expected value, so that the classification model can accurately determine whether the to-be-detected data is normal data or malicious data.

[0079] In order to improve the fitting effect of the classification model and the training data, and improve the detection accuracy of the training speed of the classification model, Figure 3 The training data is divided into normal samples and malicious samples, and the class labels corresponding to the normal samples and the malicious samples are different. In step 202, the calculation process of the loss value of the classification model includes:

[0080] Step 301: Calculate the original variance and the original mean of the training encoding vector.

[0081] Since the training encoding vector includes multiple feature dimensions, multiple original variances and multiple original means of the training encoding vector need to be calculated for each feature dimension. Based on the distribution represented by the original variances and the original means, a hidden variable is sampled, which also includes multiple dimensions. The classification model outputs a reconstruction variable according to the hidden variable.

[0082] Step 302: Obtain the category label corresponding to the training encoding vector, and obtain a target mean by taking the category label as the power and the original mean corresponding to the training encoding vector as the base number.

[0083] In a preferred embodiment, the category label of the normal sample is marked as 0, and the category label of the malicious sample is marked as 1. For example, For the original mean, the target mean of the normal sample is = 1; and the target mean of the malicious sample is = Here, 0 and 1 are category labels, and their values have no actual meaning, and can also be other values. However, taking 0 and 1 is more convenient for constructing a loss function and performing calculations.

[0084] Step 303: Calculate a divergence loss according to the original mean, the target mean, and the original variance.

[0085] Since the mathematical expectation of the classification model P(Z|X) obeys a normal distribution, the KL divergence (Kullback-Leibler divergence, abbreviated as KL divergence) is used to describe the loss of the classification model during training. In this embodiment, the category label is also introduced into the training process of the classification model. Compared with not introducing the category label and training the classification model in an unsupervised manner, the significance of introducing the category label is to improve the fitting degree of the classification model to all training data in the case of clearly training data being malicious samples or normal samples, and to improve the accuracy of the classification model in detecting the to-be-detected data. At this time, the calculation formula of the divergence loss is:

[0086]

[0087] Where d is the dimension of the hidden variable, is the original mean, is the original variance, is the target mean, and Y is the category label.

[0088] Step 304: Determine a loss value according to the divergence loss.

[0089] The loss value is determined by the divergence loss and other losses.

[0090] To reasonably set the loss value of the classification model, the divergence loss and the reconstruction error are combined. Figure 4 The step 304 specifically includes:

[0091] Step 3041: reconstruct the original variance and the original mean into a reconstruction vector, and calculate a reconstruction error according to the reconstruction vector and a corresponding training encoding vector.

[0092] The original variance and the original mean are first used to generate a hidden vector Z The reconstruction vector is then generated from the hidden vector The training encoding vector is denoted as X， The reconstruction error is denoted by .

[0093] Step 3042: determine the category of the reconstruction vector by a binary classification layer of the classification model, and calculate a binary cross-entropy loss.

[0094] The binary classification layer is a fully connected binary classification layer of Softmax (normalized exponential), which determines the category of the reconstruction vector and calculates the binary cross-entropy loss according to the determination result.

[0095] Step 3043: take the sum of the divergence loss, the reconstruction error and the binary cross-entropy loss as a loss value.

[0096] The error generated when the classification model encodes the training encoding vector and then decodes it into a reconstruction vector is measured by the divergence loss and the reconstruction error. The error of determining whether the corresponding training data is normal data or malicious data by the reconstruction vector is measured by the binary cross-entropy loss. The loss value covers several important steps in the training process of the classification model, so that the classification model can accurately determine whether the to-be-tested data is malicious data after training.

[0097] To quickly obtain the anomaly score value of the to-be-tested data and quickly filter out normal data, the divergence loss and the binary cross-entropy loss are combined. Figure 5 The specific process steps of step 102 include:

[0098] Step 1021: for each feature dimension of the target encoding vector, determine the value range of the data in the feature dimension.

[0099] The target encoding vector contains multiple feature dimensions, and different feature dimensions contain data describing different feature dimensions of the target encoding vector. It is assumed that the feature dimensions are independent of each other, and the independence assumption is made for all feature dimensions.

[0100] Step 1022: dividing the value range of the data into K continuous intervals, calculating the frequency of the data falling into each interval to establish the data histogram of each feature dimension.

[0101] Wherein, the data histogram of the feature dimension is established based on the frequency of the data falling into different intervals, for example, the value range of the data is divided into 10 continuous intervals, in order to ensure the uniformity of the feature weight of different feature dimensions, the highest height of the data histogram of each feature dimension is set to 1 by using the normalization method.

[0102] Step 1023: calculating the probability density of the corresponding feature dimension according to the data histogram, and taking the sum of the probability densities of all feature dimensions as the anomaly score value of the data to be detected.

[0103] Wherein, the height of each interval in the data histogram is data density estimation, and then the probability density of a data histogram is obtained, that is, the probability density of the corresponding feature dimension is obtained, and finally the sum of the probability densities of all data histograms can be obtained, and it is taken as the anomaly score value of the data to be detected. The above method can quickly calculate the anomaly score value of the data to be detected, and the calculation formula of the anomaly score value is as follows:

[0104]

[0105] Wherein, d is the total number of feature dimensions, represents the probability density of the i-th feature dimension of the i-th data to be detected. P i

[0106] In order to quickly filter out the normal data with a large proportion in a large number of data to be detected, a suitable segmentation threshold needs to be set. In step 103, combined with Figure 6 , the acquisition process of the segmentation threshold is as follows:

[0107] Step 1031: obtaining training data, dividing the training data into normal samples and malicious samples, and vectorizing all training data to obtain corresponding training encoding vectors.

[0108] Wherein, the implementation of this step 1031 is the same as step 201, except that the operations performed after obtaining the training encoding vectors are different. The training encoding vectors obtained by step 1031 are used to determine the segmentation threshold based on the training encoding vectors; the training encoding vectors obtained by step 201 are used to train the classification model.

[0109] ​​Step 1032: Calculate the score results of each of the normal samples and each of the malicious samples according to the training encoding vector.

[0110] Wherein, the abnormal score value of the training data needs to be calculated according to the training encoding vector, and the specific calculation process of the abnormal score value is shown in Figure 5 The specific flow steps of step 102 are shown. For details of the process of calculating the score value, please refer to step 102, that is, for each feature dimension of the training encoding vector, determine the value range of the data in the feature dimension; divide the value range of the data into K continuous intervals, calculate the frequency of the data falling into each interval to establish the data histogram of each feature dimension; calculate the probability density of the corresponding feature dimension according to the data histogram, and take the sum of the probability densities of all feature dimensions as the abnormal score value of the sample corresponding to the training encoding vector.

[0111] Step 1033: Determine the segmentation threshold according to the score situation of the normal samples and the score situation of the malicious samples.

[0112] Wherein, the abnormal score value of the normal sample will be lower than the abnormal score value of the malicious sample, and the reasonable segmentation threshold can be determined by combining the abnormal score values of the two.

[0113] In order to prevent malicious data from mixing while ensuring that most of the normal data is filtered out when filtering normal data from a large amount of to-be-detected data, the specific flow steps of step 1033 include:

[0114] Take the median of the abnormal score values of the normal samples as A; in the abnormal score values of the malicious samples, take the abnormal score value ranked in the preset position in the order of size as B; and set the segmentation threshold to the smaller value of A and B.

[0115] In this embodiment, the abnormal score values of the malicious samples are sorted in descending order, and the abnormal score value located at the last 5% position is selected as B, for example, the abnormal scores of 10000 malicious samples are sorted in descending order, and the abnormal score value ranked at the 9500th position is taken as B.

[0116] Wherein, the reason for taking the median of the abnormal score values of the normal samples as A is that in a large number of normal samples, the median is not affected by the maximum and minimum extreme values, and the variation of part of the data has no effect on the median, so the central tendency of the abnormal score values of the normal samples can be described by the median. In the statistical sense, the event with an occurrence rate lower than 5% is considered as a small probability, that is, the probability of the score value of the normal sample exceeding B is 5%, so the abnormal score value located at the last 5% position in the abnormal score of the malicious sample is selected as B.

[0117] The smaller value between A and B means that the smaller value between A and B is at least A, that is, the segmentation threshold value will not exceed the median of the anomaly score value of the normal sample, so that the screening standard is relatively loose, and it can be ensured that at most 50% of the normal data is screened out in a large amount of to-be-detected data; at the same time, when B is less than A, that is, when the score of the last 5% of the anomaly score value of the malicious sample is less than the median of the normal sample, it is indicated that it is difficult to distinguish between the last 5% of the malicious sample and the normal sample by the anomaly score value, and therefore, B is used as the segmentation threshold value, and the screening standard is slightly tightened, so that 50% or less of the normal data is screened out from a large amount of to-be-detected data; for the remaining suspicious data, the classification model is used for judgment again. The setting method of the segmentation threshold value simultaneously considers the accuracy and speed of screening a large amount of to-be-detected data, reduces the probability of misjudging malicious data as normal data, and improves the speed of screening the detection data.

[0118] In order to quickly obtain the target encoding vector, and to do the pre-work of detecting and judging the to-be-detected data, in combination with Figure 7 Step 101 specifically includes:

[0119] Step 1011: traversing the sample library, and constructing a character dictionary according to the sample library.

[0120] The character dictionary is established in the process of training the classification model, and needs to be established only once. The character dictionary contains all characters appearing in the sample library, including lowercase letters, uppercase letters, numbers, special characters, etc. At the same time, the character dictionary statistically contains the number of times each character appears in the sample library.

[0121] The character dictionary can be understood as being obtained by arranging all characters appearing in the sample library in a predetermined order.

[0122] Step 1012: traversing the to-be-detected data, and obtaining characters of the to-be-detected data.

[0123] The to-be-detected data is traversed to obtain all characters appearing in the to-be-detected data. For example, the to-be-detected data is URL data, specifically a string, and the URL data is traversed to obtain specific characters in the string.

[0124] Step 1013: calculating the number of times each character of the to-be-detected data appears in the to-be-detected data and the number of times each character appears in the sample library.

[0125] The number of occurrences of each character in the sample library and the number of occurrences of each character in the sample library can be used to measure the importance of the character in the to-be-detected data.

[0126] Step 1014: generating a target encoding vector of the to-be-detected data according to the number of occurrences of each character in the to-be-detected data and the number of occurrences of each character in the sample library.

[0127] The target encoding vector contains multiple dimensions, and the number of occurrences of each character in the to-be-detected data and the number of occurrences of each character in the sample library are stored in the target encoding vector. The length of the target encoding vector is consistent with the length of the character dictionary, so as to unify the size of the target encoding vector in different to-be-detected data and facilitate subsequent calculation. At this time, the number of occurrences of the character that has not occurred is set to 0.

[0128] In actual application scenarios, the target encoding vector is obtained in the same way as the training encoding vector, and the corresponding data sources are different, which will not be described here.

[0129] In this embodiment, in combination with Figure 8 and Figure 9 The classification model includes an encoder, a decoder, and a binary classification layer. In order to accurately detect the suspicious data preliminarily screened, the step 104 includes:

[0130] Step 1041: encoding the target encoding vector into the mean and variance of the latent variable by the encoder to determine the latent variable.

[0131] The target encoding vector needs to be processed in multiple feature dimensions to generate a latent variable with multiple dimensions.

[0132] Step 1042: reconstructing the latent variable into a reconstructed vector by the decoder.

[0133] In expectation, the reconstructed vector should tend to be consistent with the corresponding target encoding vector.

[0134] Step 1043: determining the category of the reconstructed vector by the binary classification layer to determine whether the corresponding to-be-detected data is normal data or malicious data.

[0135] After the binary classification layer is determined, the final detection result of the to-be-detected data is obtained. When the to-be-detected data is detected by the above steps, the detection rate is relatively slow, but the detection accuracy is high.

[0136] In general, in this embodiment, first, the normal data is screened out by the quick screening of the segmentation threshold, and in this stage, no specific analysis is performed on each data, but only one judgment is performed according to the abnormal score, and the data is divided into normal data and suspicious data. This method is faster, but it cannot effectively screen out the data at the normal boundary or abnormal boundary (i.e., suspicious data). In order to further subdivide the suspicious data, the suspicious data is input into the classification model, and the classification model is used to analyze the suspicious data in detail to determine the category to which it belongs.

[0137] Embodiment 2

[0138] In combination Figure 10 Based on the foregoing embodiment 1, another segmented data detection method is provided in this embodiment. First, the segmentation threshold is determined according to a large amount of training data, and the classification model is trained by the training data. After the training vectors are obtained by vectorizing all the training data, the abnormal score of all the training data is calculated based on the training vector encoding by using the abnormal score algorithm, and the segmentation threshold is obtained accordingly. Then, a conditional variational auto-encoding model is constructed as the classification model, and the classification model is trained by the training vector encoding. After the training is completed, the model structure and model weight value of the classification model are saved. In the detection of the to-be-detected data, the target vector encoding is obtained by vectorizing the to-be-detected data, the abnormal score of the to-be-detected data is calculated based on the target vector encoding, and the to-be-detected data with an abnormal score less than the segmentation threshold is judged as normal data. The to-be-detected data with an abnormal score not less than the segmentation threshold is regarded as suspicious data, and the target encoding vector of the suspicious data is detected by the classification model to determine whether the suspicious data is normal data or malicious data.

[0139] In this embodiment, a two-stage detection scheme is used to detect malicious data. First, an algorithm with general accuracy but high efficiency is used for detection, and a relatively loose threshold is set to allow normal data with high confidence to be screened out in advance. The remaining suspicious data is then subjected to a second detection, and the detection rate of the second detection is relatively low but the detection accuracy is high. In general, for a large amount of data on the network, the segmented data detection method provided by the present application can ensure both detection accuracy and detection efficiency.

[0140] Embodiment 3

[0141] Based on the segmented data detection methods provided in the foregoing embodiments 1 and 2, a segmented data detection device is provided in this embodiment. The device includes at least one processor and a memory, which are connected through a data bus. The memory stores instructions that can be executed by the at least one processor. After the instructions are executed by the processor, the distributed slicing method described in the embodiments is completed.

[0142] As Figure 11 shown, it is a schematic diagram of the architecture of the segmented data detection device of the embodiment of the present application. The content recommendation device based on the human state of the embodiment includes one or more processors 21 and a memory 22. Among them, Figure 11 In the embodiment, the processor 21 is taken as an example.

[0143] The processor 21 and the memory 22 can be connected through a bus or other means, Figure 11 In the embodiment, the connection through the bus is taken as an example.

[0144] The memory 22 is a kind of non-volatile computer readable storage medium, which can be used to store non-volatile software programs and non-volatile computer executable programs, such as the segmented data detection method in embodiment 1. The processor 21 executes the segmented data detection method by running the non-volatile software programs and instructions stored in the memory 22.

[0145] The memory 22 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 22 can optionally include a memory remotely arranged with respect to the processor 21, and these remote memories can be connected to the processor 21 through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0146] The program instructions / modules are stored in the memory 22, and when executed by the one or more processors 21, the segmented data detection method in embodiment 1 is executed, for example, the above-described Figure 1 each step shown.

[0147] It is worth noting that the information interaction, execution process, etc. between the modules and units in the above-mentioned device and system are based on the same concept as the processing method embodiments of the present application, and the specific content can be referred to the description in the method embodiments of the present application, which will not be described here.

[0148] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the embodiments can be completed by a program instructing the relevant hardware, and the program can be stored in a computer readable storage medium, which can include a read only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0149] The above merely describes preferred embodiments of the present application, and is not used to limit the present application, any modification, equivalent replacement and improvement within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A method of segmental data detection, characterized by, The method comprises the following steps: obtaining to-be-detected data, and vectorizing the to-be-detected data to obtain a corresponding target encoding vector; processing the target encoding vector to obtain an anomaly score value of the to-be-detected data; comparing the anomaly score value with a segmentation threshold value, and determining whether the to-be-detected data is normal data or suspicious data according to a comparison result; if the to-be-detected data is suspicious data, inputting the target encoding vector into a classification model to determine whether the to-be-detected data is normal data or malicious data through the classification model; the classification model comprises an encoder, a decoder and a binary classification layer, the target encoding vector is encoded into a mean value and a variance of a latent variable through the encoder to determine the latent variable, the latent variable is reconstructed into a reconstructed vector through the decoder, and a category to which the reconstructed vector belongs is determined through the binary classification layer to determine whether the corresponding to-be-detected data is normal data or malicious data.

2. The method of claim 1, wherein, The training method of the classification model comprises the following steps: obtaining training data, the training data is divided into normal samples and malicious samples, and all the training data is vectorized to obtain corresponding training encoding vectors; training the classification model through the training encoding vectors until a loss value of the classification model is reduced to an expected value.

3. The method of claim 2, wherein, The training data is divided into normal samples and malicious samples, and the normal samples and the malicious samples correspond to different category labels; the calculation process of the loss value of the classification model comprises the following steps: calculating an original variance and an original mean value of a training encoding vector; obtaining a category label corresponding to the training encoding vector, taking the original mean value of the training encoding vector as a base number, and taking the category label as a power to obtain a target mean value; calculating a divergence loss according to the original mean value, the target mean value and the original variance; determining a loss value according to the divergence loss.

4. The method of claim 3, wherein, The determination of the loss value according to the divergence loss comprises the following steps: reconstructing the original variance and the original mean value into a reconstructed vector, calculating a reconstruction error according to the reconstructed vector and a corresponding training encoding vector; determining a category to which the reconstructed vector belongs through a binary classification layer of the classification model, and calculating a binary classification cross-entropy loss; taking a sum of the divergence loss, the reconstruction error and the binary classification cross-entropy loss as the loss value.

5. The method of claim 1, wherein, The processing of the target encoding vector to obtain the anomaly score value of the to-be-detected data comprises the following steps: for each feature dimension of the target encoding vector, determining a value range of data in the feature dimension; dividing the value range of data into K continuous intervals, calculating a frequency of data falling into each interval to establish a data histogram of each feature dimension; calculating a probability density of a corresponding feature dimension according to the data histogram, and taking a sum of probability densities of all feature dimensions as the anomaly score value of the to-be-detected data.

6. The method of claim 1, wherein, The acquisition process of the segmentation threshold value comprises the following steps: obtaining training data, dividing the training data into normal samples and malicious samples, and vectorizing all the training data to obtain corresponding training encoding vectors; calculating score results of each normal sample and each malicious sample according to the training encoding vectors; Determine the segmentation threshold according to the score condition of the normal sample and the score condition of the malicious sample.

7. The method of claim 6, wherein, The determining the segmentation threshold according to the score condition of the normal sample and the score condition of the malicious sample comprises: Taking the median of the abnormal score values of the normal sample as A; In the abnormal score values of the malicious sample, taking the abnormal score values ranked in the preset position in size order as B, and setting the segmentation threshold as the smaller one of A and B.

8. The method of claim 1-7, wherein, The obtaining the to-be-detected data and vectorizing the to-be-detected data to obtain the corresponding target encoding vector comprises: Traverse the sample library to construct a character dictionary according to the sample library; Traverse the to-be-detected data to obtain the characters of the to-be-detected data; Calculate the number of times each character of the to-be-detected data appears in the to-be-detected data and the number of times each character appears in the sample library; Generate the target encoding vector of the to-be-detected data according to the number of times each character appears in the to-be-detected data and the number of times each character appears in the sample library.

9. A segmented data detection apparatus, characterized by, The device comprises at least one processor and a memory, the at least one processor and the memory are connected through a data bus, the memory stores instructions executable by the at least one processor, and the instructions are used to complete the segmented data detection method in any one of claims 1-8 after being executed by the processor.

Images