Digital object abnormal log detection method and device

Abstract

The invention relates to the technical field of digital networking, and discloses a digital object abnormal log detection method and device. The method is applied to the Internet of Things and comprises the following steps: acquiring an original log data set, and preprocessing the original log data set to generate an initial sample set; based on the abnormal log data in the initial sample set, adopting an adversarial generation algorithm to generate supplementary abnormal samples according to a first proportion, and adding the supplementary abnormal samples into the initial sample set; performing up-sampling and random sampling on the initial sample set to obtain a target sample set; training a target detection model based on the target sample set; the target detection model comprises at least one of the following network architectures: a neural network or a random forest network; using the trained target detection model to perform anomaly detection on log data generated when a user accesses a digital object to obtain a detection result; the detection result is normal or abnormal. By adopting the method, the accuracy of identifying the abnormal log data can be improved.

Description

Technical Field

[0001] The present application relates to the field of digital networking technology, and in particular to a method and device for detecting abnormal logs of digital objects. Background Art

[0002] The digital network is a virtual data network built on the Internet. Through open software architecture and standardized protocols, it can efficiently connect various data platforms and systems, realize the interconnection and interoperability of heterogeneous, heterogeneous and heterogeneous data, and thus form a data space with "data interconnection, on-demand scheduling, intra-domain autonomy, and inter-domain collaboration". Enterprises or institutions store externally accessible files in the form of digital objects in a distributed manner in the digital object warehouse of the cloud server. Users request access to the digital object warehouse to obtain files. Compared with the traditional way of accessing enterprise servers to obtain files, this greatly improves access efficiency and can well cope with the access operations of a large number of users.

[0003] When users access digital objects, corresponding log data will be generated. The log data can be used to monitor abnormal access, facilitate statistics and tracing by management personnel, and ensure the safe maintenance of files or services.

[0004] The amount of access log data in the Internet of Things is huge. Faced with such a large amount of access data, it is crucial to effectively monitor user access anomalies. In the current log data anomaly monitoring solution, the long short-term memory network (LSTM) is mainly used to detect anomalies in log data and obtain abnormal logs for subsequent auditing and analysis. However, due to the uneven distribution of positive and negative samples in the captured log data and the complex and diverse characteristics of the log data, the trained long short-term memory network has a strong dependence on the training samples, poor model generalization, and difficulty in capturing abnormal features, which leads to the problem of low recognition rate of abnormal log data and high frequency of false alarms. Summary of the invention

[0005] In view of this, the present application aims to propose a method and device for detecting abnormal logs of digital objects to improve the accuracy of identifying abnormal log data.

[0006] To achieve the above purpose, the technical solution of this application is as follows: A first aspect of an embodiment of the present application provides a method for detecting abnormal logs of digital objects, which is applied to a digital network. The method includes: Acquire an original log data set, pre-process the original log data set, and generate an initial sample set; the original log data set includes: normal log data and abnormal log data generated by users accessing the target digital object within a specified time period; Based on the abnormal log data in the initial sample set, an adversarial generation algorithm is used to generate supplementary abnormal samples according to a first ratio, and the supplementary abnormal samples are added to the initial sample set; Performing upsampling and random sampling on the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set; A target detection model is trained based on the target sample set; the target detection model includes at least one of the following network architectures: a neural network or a random forest network; The trained target detection model is used to perform anomaly detection on the log data generated by the user accessing the digital object to obtain a detection result; the detection result is normal or abnormal.

[0007] Optionally, obtain the original log data set, including: Acquire normal log data and abnormal log data generated by accessing the target digital object within the specified time period as first log data; Obtain normal log data and abnormal log data from a public data set as second log data; The first log data and the second log data are combined to obtain the original log data set.

[0008] Optionally, preprocessing the original log data set to generate an initial sample set includes: Delete invalid data and missing data; Convert character data into unique values ​​in an encoded format; Add a judgment label to each original log data set; the judgment label is normal or abnormal; Convert time information into timestamp; Extracting part of the data in the original log data set by random sampling to obtain intermediate data; The intermediate data is normalized to obtain the initial sample set.

[0009] Optionally, upsampling and random sampling are performed on the initial sample set to obtain a target sample set, including: Dividing the initial sample set into original training samples and original test samples according to a preset ratio; Using an oversampling algorithm, upsampling the original training sample to generate a first synthetic sample; Using an oversampling algorithm, upsampling the original test sample to generate a second synthetic sample; Randomly sampling the original training samples and the first synthetic samples respectively, and using the obtained samples as the training set; The original test sample set and the second synthetic samples are randomly sampled respectively, and the obtained samples are used as the test set.

[0010] Optionally, training a target detection model based on the target sample set includes: Using the target sample set to train a first model; the first model is pre-built based on a neural network; The target sample set is used to train a second model; the second model is pre-constructed based on a random forest network; Using the trained target detection model, anomaly detection is performed on the log data generated by the user's access to the digital object to obtain the detection result, specifically including: using at least one of the first model and the second model to perform anomaly detection on the log data generated by the user's access to the digital object to obtain the detection result.

[0011] Optionally, the digital object abnormal log detection method further includes: When the first model and the second model are used to perform anomaly detection on log data generated by a user accessing a digital object, and a detection result of the first model is different from a detection result of the second model, determining that the log data is abnormal; The log data is added to a first database, and a first prompt message is generated; the first prompt message is used to prompt that the log data needs a secondary analysis.

[0012] According to a second aspect of an embodiment of the present application, a digital object abnormality log detection device is provided, which is used to implement the digital object abnormality log detection method provided in the first aspect of the embodiment of the present application, and the device includes: The preprocessing module is configured to obtain an original log data set, preprocess the original log data set, and generate an initial sample set; the original log data set includes: normal log data and abnormal log data generated by users accessing the target digital object within a specified time period; A generation module is configured to generate supplementary abnormal samples according to a first ratio using an adversarial generation algorithm based on the abnormal log data in the initial sample set, and add the supplementary abnormal samples to the initial sample set; up-sample and randomly sample the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set; A training module is configured to train a target detection model based on the target sample set; the target detection model includes at least one of the following network architectures: a neural network or a random forest network; The detection module is configured to use the trained target detection model to perform anomaly detection on the log data generated by the user's access to the digital object to obtain a detection result; the detection result is normal or abnormal.

[0013] Optionally, the generating module is configured to perform upsampling and random sampling on the initial sample set to obtain a target sample set, specifically comprising: Dividing the initial sample set into original training samples and original test samples according to a preset ratio; Using an oversampling algorithm, upsampling the original training sample to generate a first synthetic sample; Using an oversampling algorithm, upsampling the original test sample to generate a second synthetic sample; Randomly sampling the original training samples and the first synthetic samples respectively, and using the obtained samples as the training set; The original test sample set and the second synthetic samples are randomly sampled respectively, and the obtained samples are used as the test set.

[0014] Optionally, the training module is configured to train a target detection model based on the target sample set, specifically comprising: using the target sample set to train a first model; the first model is pre-built based on a neural network; using the target sample set to train a second model; the second model is pre-built based on a random forest network; The detection module includes a pluggable first detection submodule and a second detection submodule; the first detection submodule is configured to use the trained first model to perform anomaly detection on the log data generated by the user's access to the digital object, and obtain a first judgment result; the first judgment result is normal or abnormal; the second detection submodule is configured to use the trained second model to perform anomaly detection on the log data generated by the user's access to the digital object, and obtain a second judgment result; the second judgment result is normal or abnormal; the detection result is obtained according to the first judgment result and / or the second judgment result.

[0015] Optionally, the detection module is further configured to perform anomaly detection on log data generated by a user accessing a digital object using the first model and the second model, and if the detection result of the first model is different from the detection result of the second model, determine that the log data is abnormal; The device also includes: The prompt module is configured to add the log data to a first database and generate first prompt information; the first prompt information is used to prompt that the log data needs secondary analysis.

[0016] Using the digital object abnormal log detection method provided by the present application, first obtain the original log data set, and pre-process the original log data therein to generate an initial sample set. The original log data set includes normal log data and abnormal log data. For the abnormal log data in the original log data set, an adversarial generation algorithm is used to generate supplementary abnormal samples, thereby increasing the number of abnormal samples in the initial sample set. Further, the initial sample set is upsampled and randomly sampled to obtain a target sample set, and the target sample set is divided into a training set and a test set for training a target calibration model. In the present application, the target detection model adopts a pluggable architecture, including at least one of a neural network or a random forest network. The trained target detection model is used to perform anomaly detection on the log data to obtain a detection result.

[0017] This application uses an adversarial generation algorithm to generate supplementary abnormal samples, thereby expanding the number of abnormal log data in the training sample set, solving the problem of low generalization of the trained model due to uneven positive and negative samples, and improving the accuracy of the model in identifying abnormal log data. In addition, this application builds a pluggable target detection model based on a neural network and a random forest network, reducing the model's dependence on training samples and further improving the model's generalization ability and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 is a flow chart of a method for detecting abnormal logs of digital objects proposed in one embodiment of the present application; Figure 2 is a schematic diagram of the process of training a target detection model in one embodiment of the present application; Figure 3 It is a schematic diagram of a digital object abnormal log detection device proposed in an embodiment of the present application. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0021] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0022] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0023] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0024] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application may be combined with each other.

[0025] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0026] Figure 1 1 is a flow chart of a method for detecting abnormal logs of digital objects proposed in one embodiment of the present application. Figure 1 As shown, the method is applied to a digital network, including: S1: obtaining an original log data set, preprocessing the original log data set, and generating an initial sample set; the original log data set includes: normal log data and abnormal log data generated by users accessing a target digital object within a specified time period; S2: Based on the abnormal log data in the initial sample set, generate supplementary abnormal samples according to a first ratio using an adversarial generation algorithm, and add the supplementary abnormal samples to the initial sample set; S3: performing upsampling and random sampling on the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set; S4: training a target detection model based on the target sample set; the target detection model includes at least one of the following network architectures: a neural network or a random forest network; S5: Using the trained target detection model, perform anomaly detection on the log data generated by the user accessing the digital object to obtain a detection result; the detection result is normal or abnormal.

[0027] In the existing solution, a detection model built with a single LSTM network is used to detect user log data. However, due to the large difference between positive and negative samples in log data and the data dependence of the LSTM network, it is difficult for the detection model to capture abnormal features. When predicting, it tends to predict the majority class (normal log) and ignore the minority class (abnormal log), resulting in low accuracy of the detection results.

[0028] In this embodiment, on the one hand, a target sample set for training the target detection model is constructed. In view of the imbalance problem of the positive and negative training samples in the existing scheme, this scheme uses an adversarial generation algorithm to generate supplementary abnormal samples in the process of constructing the target sample set, which are negative samples in the target sample set, thereby alleviating the problem of decreased model generalization ability caused by the imbalance of the positive and negative sample ratios.

[0029] In the process of building the target sample set, the obtained original log data set is first preprocessed, including supplementing or deleting blank data in the captured original log data, etc., to ensure that the log data in the obtained initial sample set is complete.

[0030] Then, an adversarial generation algorithm is used to generate supplementary abnormal samples in accordance with the first ratio to expand the number of abnormal samples in the initial sample set. Generally, when the ratio of positive to negative samples is close to 1:100 or greater, it is considered that the data set is significantly unbalanced. In this case, the model may tend to over-learn normal samples and fail to effectively capture the characteristics of abnormal samples. In this embodiment, in order to solve the problem of imbalance between positive and negative samples, a smaller positive and negative sample ratio (for example, negative sample: positive sample = 1:10 or 1:20) is used to ensure that the model can fully learn the characteristics of abnormal samples and improve the performance of the model. In practical applications, the specific positive and negative sample ratio can be fine-tuned according to demand, and this is not limited in this embodiment.

[0031] In this embodiment, the first ratio is set to 10%, that is, the GAN (Generative Adversarial Networks) algorithm is used to generate supplementary abnormal samples that are 10% of the number of abnormal log data in the initial sample set. For example, there are 2000 log data in the initial sample set, of which 200 are abnormal log data. According to the first ratio of 10%, 20 supplementary abnormal samples (abnormal log data) are generated by the GAN algorithm. Figure 2 FIG. 1 is a flow chart of training a target detection model in an embodiment of the present application. Figure 2As shown, in one embodiment, for the initial sample set, DANS (Distributed Adversarial Networks) is used to generate supplementary abnormal samples to expand the number of negative samples in the initial sample set. DANS is a special adversarial generative network (GAN) structure.

[0032] Then, the expanded initial sample set is subjected to SMOTE upsampling and random sampling to obtain the target sample set, which is used to iteratively train the target detection model.

[0033] In this embodiment, on the other hand, a target detection model for log data detection is constructed. In view of the fact that the existing scheme uses a single LSTM network to detect log data anomalies, there is a strong data dependency, such as Figure 2 As shown in the figure, in this embodiment, a pluggable target detection model is built based on the LSTM network and the random forest network. Users can select at least one of the two networks as the target detection model to detect abnormal log data. Each algorithm will generate a corresponding detection result, which will be integrated into the final detection result for user auditing.

[0034] In this embodiment, a pluggable target detection model is built based on a variety of network structures, and users can select a variety of algorithms to detect user log data, so as to better capture abnormal features. At the same time, when constructing training samples, an adversarial generation algorithm is used to generate a variety of abnormal data to expand abnormal samples, reduce the impact of the imbalance of positive and negative sample ratios on the model, improve the generalization ability of the model, and thus improve the accuracy and reliability of detection.

[0035] As an implementation of the present application, obtaining an original log data set includes: Acquire normal log data and abnormal log data generated by accessing the target digital object within the specified time period as first log data; Obtain normal log data and abnormal log data from a public data set as second log data; The first log data and the second log data are combined to obtain the original log data set.

[0036] In one embodiment, the original log data is obtained by data crawling. The access log data of the target digital object is crawled according to a determined time period to obtain the first log data. The target digital object can be set according to the actual application scenario. For example, the target digital object can be accessible data that is publicly available within the enterprise. In addition, in order to make the characteristics of the log data more diverse, in this embodiment, a part of the log data is also obtained from the public data set as the second log data. The public data set can be selected according to the actual application scenario, and this is not limited in this application. In this embodiment, the CMU-CERT data set is selected as the source of the second log data.

[0037] After obtaining the first log data and the second log data, the first log data and the second log data are merged to obtain an original log data set. In this embodiment, a variety of log data are obtained from different data sources to generate an original log data set for subsequent training of a target detection model to improve the model's ability to capture abnormal features, thereby improving the model's accuracy in identifying abnormal log data.

[0038] As an implementation manner of the present application, preprocessing the original log data set to generate an initial sample set includes: Delete invalid data and missing data; Convert character data into unique values ​​in an encoded format; Add a judgment label to each original log data set; the judgment label is normal or abnormal; Convert time information into timestamp; Extracting part of the data in the original log data set by random sampling to obtain intermediate data; The intermediate data is normalized to obtain the initial sample set.

[0039] In one embodiment, preprocessing the original data set specifically includes the following steps: (1) Import dependency packages: Before starting preprocessing, import dependency packages in advance. Dependency packages are program packages used to provide the software operating environment required by this solution; (2) Data cleaning: Identify and delete invalid data or missing data (data with missing data) in the original log data set; (3) Data format conversion: convert the character data in the original data set into unique values ​​in the encoding format; (4) Adding labels: According to the classification rules of the data set, add an error column to the abnormal log data and assign values ​​(labels). Some labels of the abnormal log data come from the original labels of the log data, and some labels come from the data labels based on the rules; (5) Convert timestamp: Convert the Datetime in the log data to a timestamp for easy calculation; (6) Random sampling: Randomly sample the original data set to obtain intermediate data; (7) Data normalization: Convert the absolute values ​​in the intermediate data into relative values, complete data normalization, and obtain the initial sample set.

[0040] As an implementation manner of the present application, upsampling and random sampling are performed on the initial sample set to obtain a target sample set, including: Dividing the initial sample set into original training samples and original test samples according to a preset ratio; Using an oversampling algorithm, upsampling the original training sample to generate a first synthetic sample; Using an oversampling algorithm, upsampling the original test sample to generate a second synthetic sample; Randomly sampling the original training samples and the first synthetic samples respectively, and using the obtained samples as the training set; The original test sample set and the second synthetic samples are randomly sampled respectively, and the obtained samples are used as the test set.

[0041] In one embodiment, after the GAN algorithm is used to generate supplementary abnormal samples and expand the initial sample set, the SMOTE (Synthetic Minority Oversampling) algorithm is used to sequentially perform upsampling and random sampling operations on the initial sample set to obtain a target sample set. Specifically, before upsampling, the initial sample set must be divided. In this embodiment, the training set and the test set are divided from the initial sample set at a ratio of 8:2, and the SMOTE upsampling and random sampling operations are performed on the training set and the test set, respectively.

[0042] SMOTE upsampling increases the number of minority samples by synthesizing new minority samples (i.e., abnormal log data) to achieve a more balanced ratio of positive and negative samples. In SMOTE upsampling, for each minority sample, its K neighboring samples are selected, and then new synthetic samples are generated based on the differences between these neighboring samples. The specific operation is to randomly select a position on the line between two adjacent minority samples, interpolate according to the weight of this position, and generate a new synthetic sample. SMOTE upsampling can help the model better learn the characteristics of minority samples and improve the prediction ability for minority classes. It alleviates the sample imbalance problem by expanding the minority samples in the training set. Specifically, the basic steps of the SMOTE algorithm are as follows: (1) Calculate the K nearest neighbors of each minority class sample (i.e., abnormal log data) through the nearest neighbor sampling algorithm; (2) Randomly select N samples from the K nearest neighbors for random linear interpolation. The k parameter in the algorithm is set to 5 by default, indicating the number of neighbors used to calculate the nearest neighbors; the random_state parameter is used to set the random seed to ensure the repeatability of the results. In this embodiment, the random_state parameter setting range is [40-50]; (3) Construct new minority class samples; (4) Merge the new samples with the original data to generate a new training set.

[0043] After SMOTE upsampling, the number of negative samples in the initial sample set is further expanded.

[0044] Next, the initial sample set is further randomly sampled to balance the ratio of positive and negative samples. After SMOTE upsampling, since the synthesized samples (abnormal log data) are added to the initial sample set, there may be too many minority class samples (negative samples). In order to avoid overfitting the model to minority class samples, random sampling is required, that is, a portion of samples are randomly selected from the samples synthesized by upsampling in the initial data set and the original samples in the initial data set to generate the target sample set. It is worth noting that the random sampling operation is the same as the SMOTE upsampling operation, and is also performed on the training set and the test set separately.

[0045] In this embodiment, the number of abnormal log data in the initial sample set is further expanded through the SMOTE upsampling operation, thereby further alleviating the problem of poor model generalization caused by the imbalance of positive and negative samples. Furthermore, the ratio of positive and negative samples in the sample set is made closer to balance through random sampling operations, avoiding the model's dependence on minority class samples due to excessive minority class samples in the subsequent training process, thereby further improving the generalization ability of the model. The combination of SMOTE upsampling and random sampling can effectively process unbalanced data sets and improve the performance of the model on minority class samples.

[0046] As an implementation manner of the present application, training a target detection model based on the target sample set includes: Using the target sample set to train a first model; the first model is pre-built based on a neural network; The target sample set is used to train a second model; the second model is pre-constructed based on a random forest network; Using the trained target detection model, anomaly detection is performed on the log data generated by the user's access to the digital object to obtain the detection result, specifically including: using at least one of the first model and the second model to perform anomaly detection on the log data generated by the user's access to the digital object to obtain the detection result.

[0047] In one embodiment, a target detection model is built using a neural network and / or a random forest network. In this embodiment, the target detection model is a pluggable architecture, which mainly includes two model structures, the first model is based on an LSTM network (neural network), and the second model is based on a random forest network. Specifically, the network structure of the first model is a recurrent neural network based on the LSTM architecture, including: an input layer, two hidden layers and an output layer. When constructing the network of the first model, first create a sequential model model through the Sequential() function, and then use the model.add() method to add neural network layers layer by layer. In this embodiment, the LSTM network includes two superimposed hidden layers, each layer containing 100 nodes.

[0048] The network structure of the second model builds a classifier based on a random forest network, which uses 100 different decision trees (i.e. n_estimators=100).

[0049] The target detection network adopts a pluggable architecture. By integrating the first model and the second model, the LSTM algorithm and the random forest algorithm are combined to give full play to the respective advantages of the two networks. The LSTM network controls the inflow and outflow of information by designing different gating structures, and has better long-term memory and better gradient fluidity, so that it can effectively handle long-term dependencies in the sequence; random forest is an integrated learning method that constructs multiple decision trees and combines their results to perform classification and regression analysis. It is more robust to noise and outliers, which helps to reduce the risk of overfitting.

[0050] When the user selects both the first model and the second model to detect abnormal log data, the detection result output by the first model and the detection result output by the second model are integrated to obtain the final detection result.

[0051] In this embodiment, the first model and the second model are designed to be pluggable to give full play to the respective advantages of the two models and improve the overall performance of the target detection model. This pluggable architecture can give full play to the advantages of different algorithms, and at the same time combine SMOTE upsampling and random sampling to solve the problem of unbalanced positive and negative ratios of sample data, which not only improves the flexibility of the target detection model, but also enables the model to perform well on different types of data sets.

[0052] In one embodiment, in addition to using the LSTM network, the first model may also use other types of neural network structures according to the needs of the actual application scenario. For example, an autoencoder may be used as the first model.

[0053] As an implementation of the present application, the digital object abnormal log detection method further includes: When the first model and the second model are used to perform anomaly detection on log data generated by a user accessing a digital object, and a detection result of the first model is different from a detection result of the second model, determining that the log data is abnormal; The log data is added to a first database, and a first prompt message is generated; the first prompt message is used to prompt that the log data needs a secondary analysis.

[0054] In one embodiment, when the user selects both the first model and the second model to detect abnormal log data, two detection results are obtained respectively. When the two detection results are integrated, if the detection result of the first model is inconsistent with the detection result of the second model, the final detection result of the integrated output is abnormal. On this basis, since the output results of the two models are inconsistent, the user needs to further perform secondary analysis on the log data in the subsequent tracing or auditing stage. Therefore, in this embodiment, a first database is pre-constructed to store abnormal log data that requires secondary analysis, so that users can regularly and uniformly process abnormal log data that requires secondary analysis.

[0055] When the detection results output by the two models are inconsistent, the log data is added to the first database, and a first prompt message is generated and displayed on the user interface. The first prompt message can be presented in the form of a log to prompt the user that the current abnormal log data needs to be analyzed again.

[0056] Based on the same inventive concept, an embodiment of the present application provides a digital object abnormal log detection device. Figure 3 , Figure 3 FIG. 1 is a schematic diagram of a digital object abnormality log detection device 100 proposed in an embodiment of the present application. Figure 3 As shown, the device comprises: The preprocessing module 101 is configured to obtain an original log data set, preprocess the original log data set, and generate an initial sample set; the original log data set includes: normal log data and abnormal log data generated by users accessing the target digital object within a specified time period; The generating module 102 is configured to generate supplementary abnormal samples according to a first ratio using an adversarial generation algorithm based on the abnormal log data in the initial sample set, and add the supplementary abnormal samples to the initial sample set; perform upsampling and random sampling on the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set; The training module 103 is configured to train a target detection model based on the target sample set; the target detection model includes at least one of the following network architectures: a neural network or a random forest network; The detection module 104 is configured to use the trained target detection model to perform anomaly detection on the log data generated by the user accessing the digital object to obtain a detection result; the detection result is normal or abnormal.

[0057] As an implementation manner of the present application, the preprocessing module 101 is configured to obtain an original log data set, specifically including: Acquire normal log data and abnormal log data generated by accessing the target digital object within the specified time period as first log data; Obtain normal log data and abnormal log data from a public data set as second log data; The first log data and the second log data are combined to obtain the original log data set.

[0058] As an implementation manner of the present application, the preprocessing module 101 is configured to preprocess the original log data set to generate an initial sample set, specifically including: Delete invalid data and missing data; Convert character data into unique values ​​in an encoded format; Add a judgment label to each original log data set; the judgment label is normal or abnormal; Convert time information into timestamp; Extracting part of the data in the original log data set by random sampling to obtain intermediate data; The intermediate data is normalized to obtain the initial sample set.

[0059] As an implementation manner of the present application, the generating module 102 is configured to perform upsampling and random sampling on the initial sample set to obtain a target sample set, specifically including: Dividing the initial sample set into original training samples and original test samples according to a preset ratio; Using an oversampling algorithm, upsampling the original training sample to generate a first synthetic sample; Using an oversampling algorithm, upsampling the original test sample to generate a second synthetic sample; Randomly sampling the original training samples and the first synthetic samples respectively, and using the obtained samples as the training set; The original test sample set and the second synthetic samples are randomly sampled respectively, and the obtained samples are used as the test set.

[0060] As an implementation mode of the present application, the training module 103 is configured to train a target detection model based on the target sample set, specifically including: using the target sample set to train a first model; the first model is pre-built based on a neural network; using the target sample set to train a second model; the second model is pre-built based on a random forest network; The detection module 104 includes a pluggable first detection submodule and a second detection submodule; the first detection submodule is configured to use the trained first model to perform anomaly detection on the log data generated by the user's access to the digital object, and obtain a first judgment result; the first judgment result is normal or abnormal; the second detection submodule is configured to use the trained second model to perform anomaly detection on the log data generated by the user's access to the digital object, and obtain a second judgment result; the second judgment result is normal or abnormal; the detection result is obtained according to the first judgment result and / or the second judgment result.

[0061] As an implementation mode of the present application, the detection module 104 is further configured to perform anomaly detection on the log data generated by the user accessing the digital object using the first model and the second model, and if the detection result of the first model is different from the detection result of the second model, determine that the log data is abnormal; The device also includes: The prompt module is configured to add the log data to a first database and generate first prompt information; the first prompt information is used to prompt that the log data needs secondary analysis.

[0062] Based on the same inventive concept, an embodiment of the present application provides a readable storage medium on which a computer program is stored. When the program is executed by a processor, the steps in the digital object abnormal log detection method described in any of the above embodiments of the present application are implemented.

[0063] Based on the same inventive concept, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes, the steps in the digital object anomaly log detection method described in any of the above embodiments of the present application are implemented.

[0064] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0065] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0066] For the method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the order of the actions described, because according to the present application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and components involved are not necessarily required by the present application.

[0067] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, devices, or computer program products. Therefore, the embodiments of the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the embodiments of the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0068] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the methods, terminal devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0069] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing terminal device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device so that a series of operating steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, thereby providing instructions for executing on the computer or other programmable terminal device to implement the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0071] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0072] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0073] The above is a detailed introduction to the digital object anomaly log detection method and device provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for detecting abnormal logs of digital objects, characterized in that: Applied to digital networking, including: Acquire an original log data set, pre-process the original log data set, and generate an initial sample set; the original log data set includes: normal log data and abnormal log data generated by users accessing the target digital object within a specified time period; Based on the abnormal log data in the initial sample set, an adversarial generation algorithm is used to generate supplementary abnormal samples according to a first ratio, and the supplementary abnormal samples are added to the initial sample set; Performing upsampling and random sampling on the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set; A target detection model is trained based on the target sample set; the target detection model includes at least one of the following network architectures: a neural network or a random forest network; The trained target detection model is used to perform anomaly detection on the log data generated by the user accessing the digital object to obtain a detection result; the detection result is normal or abnormal.

2. The method for detecting abnormal logs of digital objects according to claim 1, characterized in that: Get the original log data set, including: Acquire normal log data and abnormal log data generated by accessing the target digital object within the specified time period as first log data; Obtain normal log data and abnormal log data from a public data set as second log data; The first log data and the second log data are combined to obtain the original log data set.

3. The method for detecting abnormal logs of digital objects according to claim 1, characterized in that: The original log data set is preprocessed to generate an initial sample set, including: Delete invalid data and missing data; Convert character data into unique values ​​in an encoded format; Add a judgment label to each original log data set; the judgment label is normal or abnormal; Convert time information into timestamp; Extracting part of the data in the original log data set by random sampling to obtain intermediate data; The intermediate data is normalized to obtain the initial sample set.

4. The method for detecting abnormal logs of digital objects according to claim 1, characterized in that: The initial sample set is upsampled and randomly sampled to obtain a target sample set, including: Dividing the initial sample set into original training samples and original test samples according to a preset ratio; Using an oversampling algorithm, upsampling the original training sample to generate a first synthetic sample; Using an oversampling algorithm, upsampling the original test sample to generate a second synthetic sample; Randomly sampling the original training samples and the first synthetic samples respectively, and using the obtained samples as the training set; The original test sample set and the second synthetic samples are randomly sampled respectively, and the obtained samples are used as the test set.

5. The method for detecting abnormal logs of digital objects according to claim 1, characterized in that: Training a target detection model based on the target sample set includes: Using the target sample set to train a first model; the first model is pre-built based on a neural network; The target sample set is used to train a second model; the second model is pre-constructed based on a random forest network; Using the trained target detection model, anomaly detection is performed on the log data generated by the user's access to the digital object to obtain the detection result, specifically including: using at least one of the first model and the second model to perform anomaly detection on the log data generated by the user's access to the digital object to obtain the detection result.

6. The method for detecting abnormal logs of digital objects according to claim 5, characterized in that: Also includes: When the first model and the second model are used to perform anomaly detection on log data generated by a user accessing a digital object, and a detection result of the first model is different from a detection result of the second model, determining that the log data is abnormal; Adding the log data to a first database and generating first prompt information; The first prompt information is used to prompt that the log data needs secondary analysis.

7. A digital object abnormal log detection device, characterized in that: Used to implement the method according to any one of claims 1 to 6, comprising: The preprocessing module is configured to obtain an original log data set, preprocess the original log data set, and generate an initial sample set; the original log data set includes: normal log data and abnormal log data generated by users accessing the target digital object within a specified time period; A generation module is configured to generate supplementary abnormal samples according to a first ratio using an adversarial generation algorithm based on the abnormal log data in the initial sample set, and add the supplementary abnormal samples to the initial sample set; up-sample and randomly sample the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set; A training module is configured to train a target detection model based on the target sample set; the target detection model includes at least one of the following network architectures: a neural network or a random forest network; The detection module is configured to use the trained target detection model to perform anomaly detection on the log data generated by the user's access to the digital object to obtain a detection result; the detection result is normal or abnormal.

8. The digital object abnormality log detection device according to claim 7, characterized in that: The generation module is configured to perform upsampling and random sampling on the initial sample set to obtain a target sample set, specifically including: Dividing the initial sample set into original training samples and original test samples according to a preset ratio; Using an oversampling algorithm, upsampling the original training sample to generate a first synthetic sample; Using an oversampling algorithm, upsampling the original test sample to generate a second synthetic sample; Randomly sampling the original training samples and the first synthetic samples respectively, and using the obtained samples as the training set; The original test sample set and the second synthetic samples are randomly sampled respectively, and the obtained samples are used as the test set.

9. The digital object abnormality log detection device according to claim 7, characterized in that: The training module is configured to train a target detection model based on the target sample set, specifically comprising: using the target sample set to train a first model; the first model is pre-built based on a neural network; using the target sample set to train a second model; the second model is pre-built based on a random forest network; The detection module includes a pluggable first detection submodule and a second detection submodule; the first detection submodule is configured to use the trained first model to perform anomaly detection on the log data generated by the user's access to the digital object, and obtain a first judgment result; the first judgment result is normal or abnormal; the second detection submodule is configured to use the trained second model to perform anomaly detection on the log data generated by the user's access to the digital object, and obtain a second judgment result; the second judgment result is normal or abnormal; the detection result is obtained according to the first judgment result and / or the second judgment result.

10. The digital object abnormality log detection device according to claim 9, characterized in that: The detection module is further configured to perform anomaly detection on log data generated by a user accessing a digital object using the first model and the second model, and if the detection result of the first model is different from the detection result of the second model, determine that the log data is abnormal; The device also includes: The prompt module is configured to add the log data to a first database and generate first prompt information; the first prompt information is used to prompt that the log data needs secondary analysis.

Images