Data stream anomaly detection method and related equipment

Abstract

The embodiment of the invention provides a data flow anomaly detection method and related equipment. The data flow anomaly detection method comprises the steps of training an anomaly detection model according to a data flow feature sample set to obtain a trained anomaly detection model and a reference feature vector, and taking the trained anomaly detection model as a target anomaly detection model; inputting the real-time data stream features into a target anomaly detection model to obtain real-time feature vectors; obtaining a judgment result about whether the real-time data flow is abnormal or not according to the real-time feature vector; according to the real-time feature vector and the reference feature vector, determining whether to accumulate and update real-time data stream sample data, the real-time data stream sample data including the reference feature vector; and according to the real-time data stream sample data, re-training the anomaly detection model, and updating the target anomaly detection model. According to the embodiment of the invention, real-time updating can be carried out according to the flow change, and the accuracy of system anomaly detection is effectively improved when the streaming flow data generated in real time is processed.

Description

Technical Field

[0001] The present application relates to the field of computer and communication technology, and in particular to a data flow anomaly detection method and related equipment. Background Art

[0002] In recent years, technologies such as big data, cloud computing, and machine learning have developed by leaps and bounds. All industries have become more intelligent, and the resulting data asset security issues have become increasingly prominent. The traditional supervised learning-based methods used in network traffic detection are difficult to adapt to the ever-changing attack methods. After a period of use, there will be data drift problems, resulting in a decrease in model accuracy and failure to meet the needs of real-time monitoring. Summary of the invention

[0003] The embodiments of the present application provide a data flow anomaly detection method and related equipment, which can at least to some extent overcome the problem that traditional neural network models cannot meet the needs of real-time monitoring.

[0004] Other features and advantages of the present application will become apparent from the following detailed description, or may be learned in part by the practice of the present application.

[0005] According to one aspect of an embodiment of the present application, a data stream anomaly detection method is provided, including: training an anomaly detection model according to a data stream feature sample set to obtain a trained anomaly detection model and a benchmark feature vector, and using the trained anomaly detection model as a target anomaly detection model; inputting real-time data stream features into the target anomaly detection model to obtain a real-time feature vector; obtaining a judgment result on whether the real-time data stream is abnormal according to the real-time feature vector; determining whether to accumulate and update real-time data stream sample data according to the real-time feature vector and the benchmark feature vector, the real-time data stream sample data including the benchmark feature vector; and retraining the anomaly detection model according to the real-time data stream sample data to update the target anomaly detection model.

[0006] In some embodiments of the present application, determining whether to accumulate and update the real-time data stream sample data based on the real-time feature vector and the baseline feature vector specifically includes: determining the degree of abnormality based on the real-time feature vector and the baseline feature vector; determining whether to accumulate and update the real-time data stream sample data based on the degree of abnormality.

[0007] In some embodiments of the present application, the real-time data stream sample data also includes sample data stream features, and determining whether to accumulate and update the real-time data stream sample data based on the abnormality level specifically includes: if the abnormality level is a first abnormality level, updating the first storage space, and storing the real-time feature vector as a baseline feature vector in the first storage space; if the abnormality level is a second abnormality level, updating the second storage space, and storing the real-time data stream features as sample data stream features in the second storage space.

[0008] In some embodiments of the present application, the anomaly detection model is retrained based on the real-time data stream sample data, and the target anomaly detection model is updated, which specifically includes: determining whether the second storage space meets the retraining conditions; if the second storage space meets the retraining conditions, the anomaly detection model is retrained, and the target anomaly detection model is updated.

[0009] In some embodiments of the present application, if the second storage space meets the retraining conditions, the anomaly detection model is retrained and the target anomaly detection model is updated, specifically including: if the sample data stream features in the second storage space reach a predetermined storage number, and the proportion of judgment results of real-time data stream anomalies obtained within a predetermined time is less than a predetermined proportion threshold, then all sample data stream features in the second storage space are used as data stream feature samples to constitute a feature sample set; according to the feature sample set, the anomaly detection model is retrained and the target anomaly detection model is updated.

[0010] In some embodiments of the present application, the anomaly detection model is retrained according to the feature sample set to update the target anomaly detection model, which specifically includes: retraining the anomaly detection model according to the feature sample set to obtain a new anomaly detection model; if the new anomaly detection model meets the self-update condition, updating the target anomaly detection model to the new anomaly detection model.

[0011] In some embodiments of the present application, before inputting the real-time data stream features into the target anomaly detection model to obtain the real-time feature vector, the method includes: extracting real-time preliminary features from the real-time data stream; performing feature expansion on the real-time preliminary features to obtain real-time expanded features; and obtaining the real-time data stream features based on the real-time preliminary features and the real-time expanded features.

[0012] In some embodiments of the present application, the real-time preliminary features include address features, port features, protocol features, byte features, data packet features and time features; the extended features include geographic location features, window features and attack features; the feature expansion of the real-time preliminary features to obtain real-time extended features specifically includes: determining geographic location features based on the address features and the time features; determining window features based on the byte features, the data packet features and the time features; determining attack features based on the protocol features and the time features.

[0013] In some embodiments of the present application, the real-time data stream feature is obtained according to the real-time preliminary feature and the real-time extended feature, specifically including: performing missing value processing on the real-time preliminary feature and the real-time extended feature to obtain the real-time data stream feature.

[0014] In some embodiments of the present application, the anomaly detection model is trained according to the data stream feature sample set to obtain a trained anomaly detection model and a baseline feature vector, and the trained anomaly detection model is used as a target anomaly detection model, specifically including: obtaining a data stream feature sample set, wherein the data stream feature sample set includes multiple data stream sample features, and each of the data stream feature samples is a white sample; inputting the data stream feature samples one by one into the anomaly detection model for training to obtain a trained anomaly detection model, and using the trained anomaly detection model as the target anomaly detection model; inputting the data stream feature samples one by one into the target anomaly detection model to obtain a baseline feature vector.

[0015] In some embodiments of the present application, the anomaly detection model includes an encoder and a decoder, and the data stream feature samples are input one by one into the anomaly detection model for training to obtain a trained anomaly detection model, and the trained anomaly detection model is used as the target anomaly detection model, specifically including: inputting the data stream feature samples one by one into the encoder to obtain feature vector samples; inputting the feature vector samples into the decoder to obtain data stream reconstruction features; and updating the parameters of the encoder and the decoder according to the data stream reconstruction features and the data stream feature samples until a predetermined end condition is reached, and the training is terminated to obtain a trained encoder, which is the trained anomaly detection model.

[0016] According to one aspect of an embodiment of the present application, a data flow anomaly detection device is provided, and the data flow anomaly detection device includes: a pre-training module, which is used to train an anomaly detection model according to a data flow feature sample set to obtain a trained anomaly detection model and a benchmark feature vector, and use the trained anomaly detection model as a target anomaly detection model; a feature input module, which is used to input real-time data flow features into the target anomaly detection model to obtain a real-time feature vector; a result judgment module, which is used to obtain a judgment result on whether the real-time data flow is abnormal according to the real-time feature vector; a data accumulation module, which is used to determine whether to accumulate and update real-time data flow sample data according to the real-time feature vector and the benchmark feature vector, and the real-time data flow sample data includes the benchmark feature vector; a re-training module, which is used to re-train the anomaly detection model according to the real-time data flow sample data, and update the target anomaly detection model.

[0017] In some embodiments of the present application, the data accumulation module specifically includes: an abnormality degree submodule, used to determine the abnormality degree based on the real-time feature vector and the baseline feature vector; an accumulation update submodule, used to determine whether to accumulate and update the real-time data stream sample data based on the abnormality degree.

[0018] In some embodiments of the present application, the real-time data stream sample data also includes sample data stream features, and the accumulation and update submodule specifically includes: a first exception unit, which is used to update the first storage space if the abnormality level is a first abnormality level, and store the real-time feature vector as a baseline feature vector in the first storage space; and a second exception unit, which is used to update the second storage space if the abnormality level is a second abnormality level, and store the real-time data stream features as sample data stream features in the second storage space.

[0019] In some embodiments of the present application, the retraining module specifically includes:

[0020] A condition judgment submodule, used for judging whether the second storage space meets the retraining condition;

[0021] The model updating submodule is used to retrain the anomaly detection model and update the target anomaly detection model if the second storage space meets the retraining condition.

[0022] In some embodiments of the present application, the model update submodule specifically includes: a sample reconstruction unit, which is used to use all sample data stream features in the second storage space as data stream feature samples to form a feature sample set if the sample data stream features in the second storage space reaches a predetermined storage number and the proportion of judgment results of real-time data stream anomalies obtained within a predetermined time is less than a predetermined proportion threshold; a model reconstruction unit, which is used to retrain the anomaly detection model according to the feature sample set and update the target anomaly detection model.

[0023] In some embodiments of the present application, the model reconstruction unit is specifically used to perform: retraining the anomaly detection model according to the feature sample set to obtain a new anomaly detection model; if the new anomaly detection model meets the self-update condition, updating the target anomaly detection model to the new anomaly detection model.

[0024] In some embodiments of the present application, the data flow anomaly detection device also includes: a feature extraction module, which is used to extract real-time preliminary features from the real-time data flow; a feature expansion module, which is used to expand the real-time preliminary features to obtain real-time expanded features; and a feature determination module, which is used to obtain the real-time data flow features based on the real-time preliminary features and the real-time expanded features.

[0025] In some embodiments of the present application, the real-time preliminary features include address features, port features, protocol features, byte features, data packet features and time features; the extended features include geographic location features, window features and attack features; the feature extension module specifically includes: a location feature submodule, used to determine the geographic location features based on the address features and the time features; a window feature submodule, used to determine the window features based on the byte features, the data packet features and the time features; an attack feature submodule, used to determine the attack features based on the protocol features and the time features.

[0026] In some embodiments of the present application, the feature determination module is specifically used to execute: performing missing value processing on the real-time preliminary feature and the real-time extended feature to obtain the real-time data stream feature.

[0027] In some embodiments of the present application, the pre-training module specifically includes: a sample acquisition submodule, used to acquire a data stream feature sample set, wherein the data stream feature sample set contains multiple data stream sample features, and each of the data stream feature samples is a white sample; a model training submodule, used to input the data stream feature samples one by one into the anomaly detection model for training, to obtain a trained anomaly detection model, and to use the trained anomaly detection model as a target anomaly detection model; a vector determination submodule, used to input the data stream feature samples one by one into the target anomaly detection model to obtain a baseline feature vector.

[0028] In some embodiments of the present application, the anomaly detection model includes an encoder and a decoder, and the model training submodule specifically includes: a sample input unit, used to input the data stream feature samples into the encoder one by one to obtain feature vector samples; a feature reconstruction unit, used to input the feature vector samples into the decoder to obtain data stream reconstruction features; a parameter updating unit, used to update the parameters of the encoder and the decoder according to the data stream reconstruction features and the data stream feature samples until a predetermined end condition is reached, the training is terminated, and a trained encoder is obtained, and the trained encoder is a trained anomaly detection model.

[0029] According to one aspect of an embodiment of the present application, a computer-readable medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data flow anomaly detection method as described in the above embodiment is implemented.

[0030] According to one aspect of an embodiment of the present application, an electronic device is provided, comprising: one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the data flow anomaly detection method as described in the above embodiments.

[0031] In the technical solutions provided in some embodiments of the present application, the anomaly detection model is first pre-trained using a data stream feature sample set to obtain a target anomaly detection model and a benchmark feature vector. In the process of detecting data stream anomalies, the feature vector of the real-time data stream is compared with the benchmark feature vector. The anomaly detection model is retrained based on the feature vector of the real-time data stream and the benchmark feature vector to achieve real-time update of the target anomaly detection model, so that it can adapt to the real-time changes of the data stream and meet the needs of real-time monitoring. Compared with the neural network model of the prior art, the anomaly detection model in the embodiment of the present application can be updated in real time according to traffic changes, effectively improving the accuracy of system anomaly detection when processing real-time streaming traffic data.

[0032] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] The drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0034] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0035] Figure 2 A schematic diagram of a process for data flow anomaly detection provided in an embodiment of the present application is shown.

[0036] Figure 3 A flow chart of a self-updating system for data flow anomaly detection provided by an embodiment of the present application is shown.

[0037] Figure 4 A schematic structural diagram of a dual storage space model provided in an embodiment of the present application is shown.

[0038] Figure 5 Shown according to Figure 2 A specific implementation flow chart of step S100 in data flow anomaly detection shown in the corresponding embodiment.

[0039] Figure 6 A flow chart of another data flow anomaly detection method provided in an embodiment of the present application is shown.

[0040] Figure 7 Shown according to Figure 2 A specific implementation flowchart of step S400 in the data flow anomaly detection method shown in the corresponding embodiment.

[0041] Figure 8 Shown according to Figure 7 A specific implementation flowchart of step S420 in the data flow anomaly detection method shown in the corresponding embodiment.

[0042] Fig. 9 A schematic diagram of the structure of a data flow anomaly detection device provided in an embodiment of the present application is shown.

[0043] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this application will be more comprehensive and complete and fully convey the concept of example embodiments to those skilled in the art.

[0045] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. can be adopted. In other cases, known methods, devices, realizations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0046] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0047] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0048] Figure 1 A schematic diagram of an exemplary system architecture to which the technical solution of the embodiments of the present application can be applied is shown.

[0049] like Figure 1 As shown, the system architecture may include terminal devices (such as Figure 1 The embodiment of the present invention is a schematic diagram of a mobile phone 101, a tablet computer 102, a portable computer 103, a desktop computer, etc., a network 104, and a server 105. The network 104 is used to provide a medium for a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as a wired communication link, a wireless communication link, etc.

[0050] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. According to the implementation requirements, there may be any number of terminal devices, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

[0051] The user can use the terminal device to interact with the server 105 through the network 104 to receive or send messages, etc. The server 105 can be a server that provides various services. For example, the user uses the terminal device 103 (or the terminal device 101 or 102) to upload a data stream feature sample set and a real-time data stream or a real-time data stream feature to the server 105. The server 105 can use the data stream feature sample set to pre-train the anomaly detection model to obtain the target anomaly detection model and the benchmark feature vector, and then obtain the real-time data stream feature in the real-time data stream or directly input the real-time data stream feature into the target anomaly detection model to obtain the real-time feature vector; obtain the judgment result of whether the real-time data stream is abnormal according to the real-time feature vector; re-train the anomaly detection model according to the real-time feature vector and the benchmark feature vector, and update the target anomaly detection model.

[0052] It should be noted that the data flow anomaly detection method provided in the embodiment of the present application is generally executed by the server 105, and accordingly, the data flow anomaly detection device is generally arranged in the server 105. However, in other embodiments of the present application, the terminal device may also have similar functions as the server, so as to execute the data flow anomaly detection scheme provided in the embodiment of the present application.

[0053] The implementation details of the technical solution of the embodiment of the present application are described in detail below:

[0054] Figure 2 A flow chart of a data flow anomaly detection method according to an embodiment of the present application is shown. The data flow anomaly detection method can be executed by a server, which can be Figure 1 Refer to the server shown in . Figure 2 As shown, the data flow anomaly detection method at least includes:

[0055] Step S100, training an anomaly detection model according to a data stream feature sample set to obtain a trained anomaly detection model and a reference feature vector, and using the trained anomaly detection model as a target anomaly detection model.

[0056] Step S200, inputting the real-time data stream features into the target anomaly detection model to obtain a real-time feature vector.

[0057] Step S300: obtaining a judgment result on whether the real-time data stream is abnormal according to the real-time feature vector.

[0058] Step S400: determining whether to accumulate and update real-time data stream sample data according to the real-time feature vector and the reference feature vector, wherein the real-time data stream sample data includes the reference feature vector.

[0059] Step S500: retrain the anomaly detection model based on the real-time data stream sample data to update the target anomaly detection model.

[0060] In the embodiments of the present application, reference can be made to Figure 3 and Figure 4 First, the anomaly detection model is pre-trained using a data stream feature sample set. After obtaining a target anomaly detection model and a benchmark feature vector, in the process of detecting data stream anomalies, the feature vector of the real-time data stream is compared with the benchmark feature vector, and it is determined whether to accumulate and update the real-time data stream sample data according to the feature vector of the real-time data stream and the benchmark feature vector, and then the anomaly detection model is re-trained according to the real-time data stream sample data to achieve real-time update of the target anomaly detection model, so that it can adapt to the real-time changes of the data stream and meet the needs of real-time monitoring. Compared with the neural network model of the prior art, the anomaly detection model in the embodiment of the present application can be updated in real time according to the flow changes, and effectively improves the accuracy of system anomaly detection when processing real-time streaming traffic data.

[0061] In step S100, some data stream feature samples are first used to pre-train the anomaly detection model to obtain a pre-trained anomaly detection model. The training of the model can be performed in a supervised manner or in an unsupervised manner, which is not limited in this application.

[0062] Specifically, in some embodiments, the specific implementation of step S100 can be found in Figure 5 . Figure 5 is based on Figure 2 The detailed description of step S100 in the data flow anomaly detection method shown in the corresponding embodiment, in the data flow anomaly detection method, step S100 may include the following steps:

[0063] Step S110 , obtaining a data stream feature sample set, wherein the data stream feature sample set includes a plurality of data stream sample features, and each of the data stream feature samples is a white sample.

[0064] Step S120: input the data stream feature samples one by one into the anomaly detection model for training to obtain a trained anomaly detection model, and use the trained anomaly detection model as a target anomaly detection model.

[0065] Step S130: input the data stream feature samples one by one into the target anomaly detection model to obtain a reference feature vector.

[0066] In the embodiment of the present application, the anomaly detection model is trained in an unsupervised manner, and the data stream feature samples in the data stream feature sample set are all white samples, and the data stream feature samples themselves are used as supervision for training. At the same time, after the target anomaly detection model is trained, the data stream feature samples are input into the target anomaly detection model one by one to obtain a reference feature vector, which will be used as a reference for comparison with the feature vector obtained from the real-time data stream feature during anomaly detection, so as to realize real-time update of the anomaly detection model.

[0067] In step S110, the data stream feature samples in the data stream feature sample set are all white samples, and the data stream feature samples themselves are used as supervision for training.

[0068] In step S120, the data stream feature samples are input into the anomaly detection model one by one for training until the training ends. The training ends and a trained anomaly detection model is obtained. The trained anomaly detection model can be put into use as the model actually used in the anomaly detection process (i.e., the target anomaly detection model) and updated in real time during use.

[0069] Specifically, in some embodiments, the specific implementation of step S120 can refer to the following embodiments. Figure 2 The detailed description of step S120 in the data flow anomaly detection method shown in the corresponding embodiment, in the data flow anomaly detection method, the anomaly detection model includes an encoder and a decoder, and step S120 may include the following steps:

[0070] The data stream feature samples are input into the encoder one by one to obtain feature vector samples.

[0071] The feature vector samples are input into the decoder to obtain data stream reconstruction features.

[0072] According to the data stream reconstruction features and the data stream feature samples, the parameters of the encoder and the decoder are updated until a predetermined end condition is reached, and the training is terminated to obtain a trained encoder, which is a trained anomaly detection model.

[0073] In an embodiment of the present application, the anomaly detection model can be trained using an encoder-decoder structure, that is, the data stream features are first encoded by the encoder to obtain a feature vector, and then the feature vector encoded by the encoder is reconstructed by the decoder to obtain a reconstructed data stream feature, that is, a reconstructed data stream feature, and finally the data stream reconstructed feature and the data stream feature sample are compared, and the parameters of the encoder and the decoder are updated, and this cycle is repeated until a predetermined end condition is reached, the training is terminated, and a trained encoder is obtained, and the trained encoder is the trained anomaly detection model.

[0074] In other embodiments, the training of the anomaly detection model can also be performed through a single model, that is, the data stream feature samples are input into the anomaly detection model one by one to obtain feature vector samples, and then the result of whether the data stream is abnormal is determined based on the feature vector samples. The parameters of the anomaly detection model are updated based on the result of whether the data stream is abnormal. This cycle is repeated until a predetermined end condition is reached, the training is terminated, and a trained encoder is obtained. The trained encoder is the trained anomaly detection model.

[0075] In some embodiments, the above-mentioned updating of parameters of the encoder and the decoder according to the data stream reconstruction feature and the data stream feature sample until a predetermined end condition is reached, and the training is terminated to obtain a trained encoder, specifically includes:

[0076] Determining feature loss according to the data stream reconstruction feature and the data stream feature sample;

[0077] According to the feature loss, parameters of the encoder and the decoder are updated until a predetermined end condition is reached, and the training is terminated to obtain a trained encoder.

[0078] The predetermined end condition may be that the feature loss converges or the feature order is less than a predetermined loss threshold (eg, 0.001).

[0079] In some other embodiments, the above-mentioned updating of parameters of the encoder and the decoder according to the data stream reconstruction feature and the data stream feature sample until a predetermined end condition is reached, and the training is terminated to obtain a trained encoder, specifically includes:

[0080] If in the data stream feature sample set, only no more than a predetermined number of data stream feature samples are reconstructed by the encoder-decoder and the data stream feature features obtained are consistent with the data stream feature samples, then the encoder and the decoder are parameter updated;

[0081] If in the data stream feature sample set, more than a predetermined number of data stream feature samples are reconstructed by the encoder-decoder and the data stream reconstructed features are consistent with the data stream feature samples, then the predetermined end condition is met, the training is ended, and a trained encoder is obtained.

[0082] In step S130, after the target anomaly detection model is trained, the data stream feature samples are input into the target anomaly detection model one by one to obtain a baseline feature vector, which will be used as a benchmark for comparison with the feature vector obtained from the real-time data stream features during anomaly detection, so as to achieve real-time update of the anomaly detection model.

[0083] In step S200, the real-time data stream features are obtained by various processing of the preliminary features and the extended features. The specific method of obtaining the real-time data stream features can refer to the following embodiments. In some embodiments of the present application, Figure 6 As shown, before step S200, the method includes:

[0084] Step S600, extracting real-time preliminary features from the real-time data stream.

[0085] Step S700, performing feature expansion on the real-time preliminary feature to obtain a real-time expanded feature.

[0086] Step S800, obtaining the real-time data stream feature according to the real-time preliminary feature and the real-time extended feature.

[0087] In an embodiment of the present application, real-time preliminary features are first extracted from the previous data stream, and then feature expansion is performed based on the real-time preliminary features to obtain real-time expanded features. Finally, the real-time preliminary features and the real-time expanded features are finally processed to obtain real-time data stream features.

[0088] In step S600, the real-time preliminary features are initial features that can be directly obtained from the real-time data stream, such as address features, port features, protocol features, byte features, data packet features and time features, etc., which can be directly extracted from the quintuple of the data packet in the data stream, wherein the address features may include the source IP address and the destination IP address, the port features may include the source port and the destination port, the protocol features may include various transmission protocols used by the data stream, the byte features may include features related to the number of bytes in the data stream, etc., the data packet features may include features related to the data packets in the data stream, etc., and the time features are time-related features such as the time sequence corresponding to the real-time data stream.

[0089] After directly extracting the above real-time preliminary features from the real-time data stream, it is also necessary to clean the above features to remove the incomplete data stream after log parsing. This data stream data is incomplete and cannot be used for anomaly detection, which is meaningless for subsequent detection. For example, the IP address is empty, the total amount of payload bytes transmitted by the data stream is empty, or the total amount of data packets in the data stream is empty.

[0090] In step S700, the real-time extended features include geographic location features, window features, and attack features, and the specific acquisition method thereof may refer to the following embodiments.

[0091] Specifically, in some embodiments, the specific implementation of step S700 can refer to the following embodiments. Figure 6 The detailed description of step S700 in the data flow anomaly detection method shown in the corresponding embodiment, in the data flow anomaly detection method, the real-time preliminary features include address features, port features, protocol features, byte features, data packet features and time features; the extended features include geographic location features, window features, attack features, and step S700 may include the following steps:

[0092] A geographic location feature is determined according to the address feature and the time feature.

[0093] Window characteristics are determined according to the byte characteristics, the data packet characteristics and the time characteristics.

[0094] An attack feature is determined according to the protocol feature and the time feature.

[0095] In the embodiment of the present application, the geographic location feature is obtained from the address feature and the time feature. Specifically, the IP address, as the key initial feature of the traffic data, can be obtained by combining the IP with the geographic location database. According to the database information, the continent, longitude and latitude, country, province, city, ASN and other information of the IP can be obtained. The features such as the number of value categories in the window of 'source country', 'source city', 'source inside and outside' and whether a mutation occurs are obtained by combining the time window statistics, which are the geographic location features.

[0096] Considering that there are a large number of intranet IPs (victim environment) in the dataset, the geographic location “intranet” can be marked as a separate category during the intranet IP processing.

[0097] The window feature is obtained from byte features, packet features and time features. Specifically, according to the correlation between the behaviors between flows, the numerical features ['packets', 'bytes', 'packet_avg_bytes'] (where 'bytes' is the total number of bytes transmitted by the data flow, 'packets' is the total number of packets, and 'packet_avg_bytes' is the average number of bytes per packet) can be expanded with a sliding window, such as extracting the numerical window features by aggregation and rolling according to the destination port and destination IP address, to obtain the above-mentioned window features.

[0098] The above-mentioned window characteristics may include the mean, median, maximum value, minimum value, variance, standard deviation, sum, covariance, skewness, kurtosis, standard score = x-(window mean) / (window standard deviation), coefficient of variation = (window standard deviation) / (window mean), x / mean / std, etc. of various numerical data (number of bytes, number of data packets, etc.).

[0099] Compared with the real-time preliminary features which only have the necessary five-tuple, timestamp identification information, and numerical features which only contain the number of packets and bytes, window features can well learn the fine-grained information within the stream due to the existence of behavioral correlations between streams.

[0100] As for the attack features, they are obtained from the protocol features and time features. Specifically, the number of flows in a certain time window can be aggregated by combining the TCP protocol grouping, the average interval time of flows, the number of flow packets and the number of bytes, etc., so as to expand the feature set that can reflect the attack behavior according to the common intrusion attack traffic characteristics, that is, the attack features. Since normal traffic has certain traffic characteristics and rules, it is very effective to introduce mutation features related to attack characteristics.

[0101] The above-mentioned embodiment starts from the available necessary fields and builds a set of feature expansion methods from multiple perspectives, making full use of limited time information, spatial information and statistical information. The dynamic features are expanded in the form of sliding windows and spatial geographic location information and mutation features that conform to common attack characteristics are introduced, which increases the characterization ability of traffic. The rich features can effectively capture the characteristics of traffic data to help the subsequent anomaly detection model achieve better performance.

[0102] In step S800, missing value processing is performed on the real-time preliminary features and the real-time extended features, that is, real-time data stream features are obtained. Specific missing value processing methods may include infinite value processing, character encoding, log normalization, and other processing methods.

[0103] In step S300, after obtaining the real-time feature vector, the real-time feature vector can be analyzed to obtain a judgment result of whether the real-time data stream is abnormal. The specific method can be to output the judgment result through a classification layer after passing through multiple convolution layers and pooling layers.

[0104] In step S400, after obtaining the real-time feature vector, not only a judgment result is obtained based on it, but also the real-time feature vector and the benchmark feature vector are compared to determine whether to accumulate and update the real-time data stream sample data, and then determine whether to update the anomaly detection model in real time.

[0105] Specifically, in some embodiments, the specific implementation of step S400 can be found in Figure 7 . Figure 7 is based on Figure 2 The detailed description of step S400 in the data flow anomaly detection method shown in the corresponding embodiment, in the data flow anomaly detection method, step S400 may include the following steps:

[0106] Step S410: determining the abnormality degree according to the real-time feature vector and the reference feature vector.

[0107] Step S420: Determine whether to accumulate and update the real-time data stream sample data according to the abnormality level.

[0108] In the embodiment of the present application, the anomaly detection model is retrained according to the degree of anomaly between the real-time feature vector and the reference feature vector to update the target anomaly detection model.

[0109] In step S410, the benchmark feature vectors may be stored in a storage space. When there are too many benchmark feature vectors stored in the storage space, a predetermined number of benchmark feature vectors closest to the real-time feature vector may be selected for comparison with the real-time feature vector. The comparison may be performed by calculating the vector distance.

[0110] The degree of abnormality may be expressed by numerical values ​​such as scores or percentages, or by levels, and this application does not limit this.

[0111] Taking the score as an example, the real-time feature vector is compared with a predetermined number of reference feature vectors closest to the real-time feature vector to obtain an abnormal score.

[0112] Taking the level as an example, the real-time feature vector is compared with a predetermined number of reference feature vectors closest to the real-time feature vector to obtain the abnormal level.

[0113] In step S420, according to different abnormality levels, it is determined whether to accumulate and update the real-time data stream sample data. The specific situation can be referred to the following embodiment.

[0114] Specifically, in some embodiments, the specific implementation of step S420 can be found in Figure 8 . Figure 8 is based on Figure 7 The detailed description of step S420 in the data flow anomaly detection method shown in the corresponding embodiment, in the data flow anomaly detection method, the reference feature vector is stored in the first storage space, and step S420 may include the following steps:

[0115] Step S422: If the abnormality level is the first abnormality level, the first storage space is updated, and the real-time feature vector is stored in the first storage space as a reference feature vector.

[0116] Step S424: If the abnormality level is the second abnormality level, the second storage space is updated, and the real-time data stream feature is stored as a sample data stream feature in the second storage space.

[0117] In an embodiment of the present application, a first storage space is used to store feature vectors, such as a baseline feature vector. A second storage space is used to store data stream features, such as sample data stream features. Under different abnormality levels, different operations are performed to store real-time feature vectors in different storage spaces, or directly discard them. Specifically, if the abnormality level is a first abnormality level, the first storage space is updated, and the real-time feature vector is stored in the first storage space as a baseline feature vector. If the abnormality level is a second abnormality level, the second storage space is updated, and the real-time data stream features are stored in the second storage space as sample data stream features.

[0118] In an embodiment of the present application, when the abnormality degree is a level, the first abnormality degree and the second abnormality degree correspond to different abnormality levels. For example, the first abnormality degree corresponds to the first abnormality level, and when it is at the first abnormality level, the similarity between the real-time feature vector and the reference feature vector is the highest among all abnormality levels. The second abnormality degree corresponds to the second abnormality level, and when it is at the second abnormality level, the similarity between the real-time feature vector and the reference feature vector is the second highest among all abnormality levels, second only to the first abnormality level. That is, as the abnormality level increases, the similarity between the real-time feature vector and the reference feature vector gradually decreases, and the possibility of abnormality gradually increases.

[0119] When the abnormality level is a score, the first abnormality level and the second abnormality level correspond to different score value intervals. For example, the first abnormality level corresponds to the value interval score(Z t )<α, that is, when the abnormal score score(Z t) is less than the super parameter α, it is the first abnormality level; the value interval corresponding to the second abnormality level is α <score(Z t )<β, that is, when the abnormal score score(Z t ) is between the super parameter α and the super parameter β, it is the second abnormality level. t ) is smaller, the more similar the real-time feature vector is to the benchmark feature vector, and the lower the possibility of anomaly. The hyperparameter α and hyperparameter β can be used to define the minimum quantile threshold or set according to the data of a certain period of time.

[0120] That is, in the above embodiment, when the abnormality level is the lowest first abnormality level, there is the highest similarity between the real-time feature vector and the baseline feature vector. At this time, in order to maintain the real-time and advanced nature of the first storage space, the first storage space is updated, and the real-time feature vector is stored in the first storage space as the baseline feature vector, that is, the baseline feature vector is updated in real time to avoid the problem of data drift caused by always using the same baseline feature vector.

[0121] In some embodiments, the first storage space is an explicit memory module for storing a normal data baseline and adopting a first-in-first-out memory replacement policy mechanism.

[0122] When the abnormality level is a lower second abnormality level, there is a higher degree of similarity between the real-time feature vector and the baseline feature vector. At this time, the second storage space is updated, and the real-time data stream features are stored as sample data stream features in the second storage space to capture the characteristics of the real-time white sample in real time and update the model when the data drifts.

[0123] In step S500, after the real-time data stream sample data is continuously updated until the update condition is met, the judgment result accumulates the white flow data of the new stage in real time, that is, the real-time data stream sample data is used to update the anomaly detection model in real time. The specific real-time update method can refer to the following embodiment.

[0124] Specifically, in some embodiments, the specific implementation of step S500 can refer to the following embodiments. Figure 8 The detailed description of step S500 in the data flow anomaly detection method shown in the corresponding embodiment, in the data flow anomaly detection method, step S500 may include the following steps:

[0125] It is determined whether the second storage space meets the retraining condition.

[0126] If the second storage space meets the retraining condition, the anomaly detection model is retrained and the target anomaly detection model is updated. In an embodiment of the present application, if the sample data stream features stored in the second storage space meet the retraining condition, the anomaly detection model is retrained and the target anomaly detection model is updated.

[0127] Specifically, if the number of sample data stream features in the second storage space reaches a predetermined storage number, and the proportion of judgment results of real-time data stream anomalies obtained within a predetermined time is less than a predetermined proportion threshold, all sample data stream features in the second storage space are used as data stream feature samples to form a feature sample set.

[0128] The anomaly detection model is retrained according to the feature sample set, and the target anomaly detection model is updated.

[0129] In an embodiment of the present application, when the sample data stream features in the second storage space reach a certain number, and only a small number of abnormal data streams are detected in real time, the target anomaly detection model can be updated. Among them, the predetermined storage number of sample data stream features in the second storage space that needs to be reached is less than the number of data stream feature samples in the data stream feature sample set pre-trained in step S100. The predetermined time is a unit time, which can be within a time window. The predetermined ratio threshold can be specifically determined according to the length of the predetermined time and the amount of traffic in the unit time, for example, 0.01.

[0130] After the above conditions are met, the sample data stream features in the second storage space are used as data stream feature samples to form a feature sample set. According to the feature sample set, the anomaly detection model is retrained and the target anomaly detection model is updated. At the same time, the admission thresholds, super parameters α and super parameters β of the first storage space and the second storage space after the refitting model are also updated.

[0131] Specifically, the above-mentioned retraining of the anomaly detection model according to the feature sample set and updating of the target anomaly detection model specifically includes:

[0132] According to the feature sample set, the anomaly detection model is retrained to obtain a new anomaly detection model.

[0133] If the new anomaly detection model satisfies the self-update condition, the target anomaly detection model is updated to the new anomaly detection model.

[0134] In an embodiment of the present application, the specific method of retraining the anomaly detection model according to the feature sample set to obtain a new anomaly detection model can refer to the method of pre-training the anomaly detection model in step S100, which will not be described in detail in this embodiment.

[0135] When updating the anomaly detection model, the new anomaly detection model needs to be tested. Only when the new anomaly detection model meets the self-update conditions, that is, when it is determined whether it is better than the existing anomaly detection model in all aspects, especially timeliness, the target anomaly detection model is updated to the above new anomaly detection model. Otherwise, the model is rolled back and the update is invalid.

[0136] After the target anomaly detection model is updated to the new anomaly detection model, the second storage space will be cleared and wait for the next update to avoid too frequent updates, resulting in waste of resources and reduced efficiency of data flow monitoring. The first storage space can also be cleared or retained.

[0137] In some embodiments, the first storage space will be cleared. At this time, each sample in the feature sample set used to train the above-mentioned updated target anomaly detection model can be input into the updated target anomaly detection model to obtain a new benchmark feature vector and store it in the first storage space.

[0138] The following describes an apparatus embodiment of the present application, which can be used to execute the data flow anomaly detection method in the above-mentioned embodiment of the present application. For details not disclosed in the apparatus embodiment of the present application, please refer to the above-mentioned embodiment of the data flow anomaly detection method of the present application.

[0139] In the above-mentioned embodiment, the streaming abnormal traffic detection algorithm based on the dual memory module intelligently solves the problem of data drift through automatic memory update and model automatic update and fallback mechanism, effectively improves the accuracy of system anomaly detection when processing real-time streaming traffic data, and has broad application prospects.

[0140] Fig. 9 A block diagram of a data flow anomaly detection device according to an embodiment of the present application is shown.

[0141] Reference Fig. 9 As shown, according to an embodiment of the present application, a data flow anomaly detection device 900 includes: a pre-training module 910, a feature input module 920, a result judgment module 930, a data accumulation module 940, and a re-training module 950.

[0142] A pre-training module 910 is used to train an anomaly detection model according to a data stream feature sample set to obtain a trained anomaly detection model and a benchmark feature vector, and use the trained anomaly detection model as a target anomaly detection model; a feature input module 920 is used to input real-time data stream features into the target anomaly detection model to obtain a real-time feature vector; a result judgment module 930 is used to obtain a judgment result on whether the real-time data stream is abnormal according to the real-time feature vector; a data accumulation module 940 is used to determine whether to accumulate and update real-time data stream sample data according to the real-time feature vector and the benchmark feature vector, and the real-time data stream sample data includes the benchmark feature vector; a re-training module 950 is used to re-train the anomaly detection model according to the real-time data stream sample data to update the target anomaly detection model.

[0143] In some embodiments of the present application, the data accumulation module specifically includes: an abnormality degree submodule, used to determine the abnormality degree based on the real-time feature vector and the baseline feature vector; an accumulation update submodule, used to determine whether to accumulate and update the real-time data stream sample data based on the abnormality degree.

[0144] In some embodiments of the present application, the real-time data stream sample data also includes sample data stream features, and the accumulation and update submodule specifically includes: a first exception unit, which is used to update the first storage space if the abnormality level is a first abnormality level, and store the real-time feature vector as a baseline feature vector in the first storage space; and a second exception unit, which is used to update the second storage space if the abnormality level is a second abnormality level, and store the real-time data stream features as sample data stream features in the second storage space.

[0145] In some embodiments of the present application, the retraining module specifically includes:

[0146] A condition judgment submodule, used for judging whether the second storage space meets the retraining condition;

[0147] The model updating submodule is used to retrain the anomaly detection model and update the target anomaly detection model if the second storage space meets the retraining condition.

[0148] In some embodiments of the present application, the model update submodule specifically includes: a sample reconstruction unit, which is used to use all sample data stream features in the second storage space as data stream feature samples to form a feature sample set if the sample data stream features in the second storage space reaches a predetermined storage number and the proportion of judgment results of real-time data stream anomalies obtained within a predetermined time is less than a predetermined proportion threshold; a model reconstruction unit, which is used to retrain the anomaly detection model according to the feature sample set and update the target anomaly detection model.

[0149] In some embodiments of the present application, the model reconstruction unit is specifically used to perform: retraining the anomaly detection model according to the feature sample set to obtain a new anomaly detection model; if the new anomaly detection model meets the self-update condition, updating the target anomaly detection model to the new anomaly detection model.

[0150] In some embodiments of the present application, the data flow anomaly detection device also includes: a feature extraction module, which is used to extract real-time preliminary features from the real-time data flow; a feature expansion module, which is used to expand the real-time preliminary features to obtain real-time expanded features; and a feature determination module, which is used to obtain the real-time data flow features based on the real-time preliminary features and the real-time expanded features.

[0151] In some embodiments of the present application, the real-time preliminary features include address features, port features, protocol features, byte features, data packet features and time features; the extended features include geographic location features, window features and attack features; the feature extension module specifically includes: a location feature submodule, used to determine the geographic location features based on the address features and the time features; a window feature submodule, used to determine the window features based on the byte features, the data packet features and the time features; an attack feature submodule, used to determine the attack features based on the protocol features and the time features.

[0152] In some embodiments of the present application, the feature determination module is specifically used to execute: performing missing value processing on the real-time preliminary feature and the real-time extended feature to obtain the real-time data stream feature.

[0153] In some embodiments of the present application, the pre-training module specifically includes: a sample acquisition submodule, used to acquire a data stream feature sample set, wherein the data stream feature sample set contains multiple data stream sample features, and each of the data stream feature samples is a white sample; a model training submodule, used to input the data stream feature samples one by one into the anomaly detection model for training, to obtain a trained anomaly detection model, and to use the trained anomaly detection model as a target anomaly detection model; a vector determination submodule, used to input the data stream feature samples one by one into the target anomaly detection model to obtain a baseline feature vector.

[0154] In some embodiments of the present application, the anomaly detection model includes an encoder and a decoder, and the model training submodule specifically includes: a sample input unit, used to input the data stream feature samples into the encoder one by one to obtain feature vector samples; a feature reconstruction unit, used to input the feature vector samples into the decoder to obtain data stream reconstruction features; a parameter updating unit, used to update the parameters of the encoder and the decoder according to the data stream reconstruction features and the data stream feature samples until a predetermined end condition is reached, the training is terminated, and a trained encoder is obtained, and the trained encoder is a trained anomaly detection model.

[0155] Compared with the prior art, the above-mentioned embodiments, on the one hand, start from the available necessary fields, combine multiple angles to build a set of feature expansion methods, and make full use of limited time information, spatial information and statistical information. The dynamic features are expanded in the form of sliding windows and the spatial geographic location information and mutation features that conform to common attack characteristics are introduced, which increases the ability to characterize traffic. The rich features can effectively capture the characteristics of traffic data to help subsequent anomaly detection models obtain better performance; on the other hand, the streaming anomaly traffic detection algorithm based on dual memory modules intelligently solves the problem of data drift through automatic memory update and model automatic update and fallback mechanisms, effectively improving the accuracy of system anomaly detection when processing real-time streaming traffic data, and has broad application prospects.

[0156] Fig.10 A schematic diagram of the structure of a computer system suitable for implementing an electronic device of an embodiment of the present application is shown.

[0157] It should be noted that Fig.10 The computer system of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0158] like Fig.10As shown, the computer system includes a central processing unit (CPU) 1801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1802 or the program loaded from the storage part 1808 to the random access memory (RAM) 1803, such as executing the method described in the above embodiment. In RAM 1803, various programs and data required for system operation are also stored. CPU 1801, ROM 1802 and RAM 1803 are connected to each other through bus 1804. Input / output (I / O) interface 1805 is also connected to bus 1804.

[0159] The following components are connected to the I / O interface 1805: an input section 1806 including a keyboard, a mouse, etc.; an output section 1807 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to the I / O interface 1805 as needed. A removable medium 1811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1810 as needed so that a computer program read therefrom is installed into the storage section 1808 as needed.

[0160] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication section 1809, and / or installed from a removable medium 1811. When the computer program is executed by a central processing unit (CPU) 1801, various functions defined in the system of the present application are executed.

[0161] It should be noted that the computer-readable medium shown in the embodiment of the present application may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by an instruction execution system, device or device or used in combination with it. In the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a computer-readable computer program is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0162] The flowchart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present application. Wherein, each box in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and the above-mentioned module, program segment, or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0163] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. The names of these units do not, in some cases, constitute limitations on the units themselves.

[0164] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or may exist independently without being assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the above embodiment.

[0165] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0166] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0167] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the embodiments disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application.

[0168] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A data flow anomaly detection method, characterized in that: The data flow anomaly detection method comprises: According to the data stream feature sample set, the anomaly detection model is trained to obtain a trained anomaly detection model and a reference feature vector, and the trained anomaly detection model is used as a target anomaly detection model; Inputting the real-time data stream features into the target anomaly detection model to obtain a real-time feature vector; Obtaining a judgment result on whether the real-time data stream is abnormal according to the real-time feature vector; Determining whether to accumulate and update real-time data stream sample data according to the real-time feature vector and the reference feature vector, wherein the real-time data stream sample data includes the reference feature vector; The anomaly detection model is retrained according to the real-time data stream sample data, and the target anomaly detection model is updated.

2. The data flow anomaly detection method according to claim 1, characterized in that: The determining whether to accumulate and update the real-time data stream sample data according to the real-time feature vector and the reference feature vector specifically includes: Determining the degree of abnormality according to the real-time feature vector and the reference feature vector; According to the abnormality degree, it is determined whether to accumulate and update the real-time data stream sample data.

3. The data flow anomaly detection method according to claim 2, characterized in that: The real-time data stream sample data also includes sample data stream features, and determining whether to accumulate and update the real-time data stream sample data according to the abnormality degree specifically includes: If the abnormality degree is the first abnormality degree, the first storage space is updated, and the real-time feature vector is stored in the first storage space as a reference feature vector; If the abnormality level is the second abnormality level, the second storage space is updated, and the real-time data stream feature is stored in the second storage space as a sample data stream feature.

4. The data flow anomaly detection method according to claim 3, characterized in that: The retraining of the anomaly detection model according to the real-time data stream sample data to update the target anomaly detection model specifically includes: Determining whether the second storage space meets the retraining condition; If the second storage space meets the retraining condition, the anomaly detection model is retrained to update the target anomaly detection model.

5. The data flow anomaly detection method according to claim 1, characterized in that: Before inputting the real-time data stream features into the target anomaly detection model to obtain the real-time feature vector, the method includes: Extracting real-time preliminary features from real-time data streams; Performing feature expansion on the real-time preliminary feature to obtain a real-time expanded feature; The real-time data stream feature is obtained according to the real-time preliminary feature and the real-time extended feature.

6. The data flow anomaly detection method according to claim 5, characterized in that: The real-time preliminary features include address features, port features, protocol features, byte features, data packet features and time features; the extended features include geographic location features, window features and attack features; The step of performing feature expansion on the real-time preliminary feature to obtain the real-time expanded feature specifically includes: Determining geographic location features according to the address features and the time features; Determining a window feature according to the byte feature, the data packet feature, and the time feature; An attack feature is determined according to the protocol feature and the time feature.

7. The data flow anomaly detection method according to claim 1, characterized in that: The method of training the anomaly detection model according to the data stream feature sample set to obtain a trained anomaly detection model and a reference feature vector, and using the trained anomaly detection model as a target anomaly detection model, specifically includes: Acquire a data stream feature sample set, wherein the data stream feature sample set includes a plurality of data stream sample features, and each of the data stream feature samples is a white sample; Inputting the data stream feature samples one by one into the anomaly detection model for training to obtain a trained anomaly detection model, and using the trained anomaly detection model as a target anomaly detection model; The data stream feature samples are input into the target anomaly detection model one by one to obtain a reference feature vector.

8. A data flow anomaly detection device, characterized in that: The data flow anomaly detection device comprises: A pre-training module is used to train an anomaly detection model according to a data stream feature sample set to obtain a trained anomaly detection model and a reference feature vector, and use the trained anomaly detection model as a target anomaly detection model; A feature input module, used to input the real-time data stream features into the target anomaly detection model to obtain a real-time feature vector; A result judgment module, used for obtaining a judgment result on whether the real-time data stream is abnormal according to the real-time feature vector; A data accumulation module, used to determine whether to accumulate and update real-time data stream sample data according to the real-time feature vector and the reference feature vector, wherein the real-time data stream sample data includes the reference feature vector; A retraining module is used to retrain the anomaly detection model according to the real-time data stream sample data and update the target anomaly detection model.

9. A computer readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the data flow anomaly detection method according to any one of claims 1 to 7 is implemented.

10. An electronic device, characterized in that: include: one or more processors; A storage device, used to store one or more programs, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the data flow anomaly detection method according to any one of claims 1 to 7.

Images