Digital Object Exception Log Detection Method and Device

Abstract

The present application relates to the technical field of digital networking, and discloses a method and device for detecting abnormal logs of digital objects. The method is applied to a digital network and includes: obtaining an original log data set, preprocessing the original log data set to generate an initial sample set; based on the abnormal log data in the initial sample set, using an adversarial generation algorithm to generate supplementary abnormal samples according to a first ratio and adding them to the initial sample set; performing upsampling and random sampling on the initial sample set to obtain a target sample set; training a target detection model based on the target sample set; the target detection model includes at least one of the following network architectures: a neural network or a random forest network; using the trained target detection model to perform abnormal detection on the log data generated by a user accessing a digital object to obtain a detection result; the detection result is normal or abnormal. Using this method can improve the accuracy of identifying abnormal log data.

Description

Technical Field

[0001] The present application relates to the technical field of digital networking, and particularly to a method and device for detecting abnormal logs of digital objects. Background Art

[0002] The digital networking is a virtual data network built on the Internet. Through an open software architecture and standardized protocols, it can efficiently connect various data platforms and systems, realize the interconnection and interoperability of heterogeneous, remote, and heterogeneous-master data, and thus form a data space with the characteristics of "data interconnection, on-demand scheduling, intra-domain autonomy, and inter-domain collaboration". Enterprises or institutions distribute and store externally accessible files in the digital object warehouse of a cloud server in the form of digital objects. Users request access to the digital object warehouse to obtain files. Compared with the traditional way of accessing enterprise servers to obtain files, the access efficiency is greatly improved, and it can well handle the access operations of a large number of users.

[0003] When a user accesses a digital object, corresponding log data will be generated. The log data can be used for abnormal access monitoring, facilitating managers to conduct statistics and traceability, and ensuring the security maintenance of files or services.

[0004] The amount of access log data in digital networking is huge. Facing such a large amount of access data, how to effectively monitor abnormal user access is crucial. In the current abnormal monitoring scheme of log data, the long short-term memory network (LSTM, Long Short-Term Memory) is mainly used to detect abnormal log data and obtain abnormal logs for subsequent auditing and analysis. However, due to the problem of uneven distribution of positive and negative samples in the captured log data, and the complex and diverse characteristics of log data, the trained long short-term memory network is highly dependent on training samples, the model generalization ability is poor, it is difficult to capture abnormal features, and thus the model has problems such as low recognition rate of abnormal log data and high false error reporting frequency. Summary of the Invention

[0005] In view of this, the present application aims to propose a method and device for detecting abnormal logs of digital objects to improve the accuracy of identifying abnormal log data.

[0006] To achieve the above object, the technical solution of the present application is as follows:

[0007] The first aspect of the embodiment of the present application provides a method for detecting abnormal logs of digital objects, which is applied to digital networking. The method includes:

[0008] Obtain an original log data set, preprocess the original log data set to generate an initial sample set; the original log data set includes: normal log data and abnormal log data generated by a user accessing a target digital object within a specified time period;

[0009] Based on the abnormal log data in the initial sample set, use an adversarial generation algorithm to generate supplementary abnormal samples according to the first ratio and add them to the initial sample set;

[0010] Perform upsampling and random sampling on the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set;

[0011] Train a target detection model based on the target sample set; the target detection model includes at least one of the following network architectures: a neural network or a random forest network;

[0012] Use the trained target detection model to perform abnormal detection on the log data generated by a user's access to a digital object to obtain a detection result; the detection result is normal or abnormal.

[0013] Optionally, obtaining the original log data set includes:

[0014] Obtain the normal log data and abnormal log data generated by accessing the target digital object within the specified time period as the first log data;

[0015] Obtain normal log data and abnormal log data from a public data set as the second log data;

[0016] Merge the first log data and the second log data to obtain the original log data set.

[0017] Optionally, preprocessing the original log data set to generate an initial sample set includes:

[0018] Delete invalid data and gap data;

[0019] Convert character data into unique values in an encoded format;

[0020] Add a determination label to each original log data set; the determination label is normal or abnormal;

[0021] Convert time information into a timestamp;

[0022] Adopt random sampling to extract part of the data in the original log data set to obtain intermediate data;

[0023] Perform normalization processing on the intermediate data to obtain the initial sample set.

[0024] Optionally, performing upsampling and random sampling on the initial sample set to obtain a target sample set includes:

[0025] Divide the initial sample set into an original training sample set and an original test sample set according to a preset ratio;

[0026] Use an oversampling algorithm to upsample the original training sample set to generate a first synthetic sample set;

[0027] Use an oversampling algorithm to upsample the original test sample set to generate a second synthetic sample set;

[0028] Randomly sample the original training sample set and the first synthetic sample set respectively, and use the obtained samples as the training set;

[0029] Randomly sample the original test sample set and the second synthetic sample set respectively, and use the obtained samples as the test set.

[0030] Optionally, training a target detection model based on the target sample set includes:

[0031] Train a first model using the target sample set; the first model is pre-constructed based on a neural network;

[0032] Train a second model using the target sample set; the second model is pre-constructed based on a random forest network;

[0033] Use the trained target detection model to perform anomaly detection on the log data generated by a user's access to a digital object to obtain a detection result, specifically including: using at least one of the first model and the second model to perform anomaly detection on the log data generated by a user's access to a digital object to obtain the detection result.

[0034] Optionally, the digital object anomaly log detection method further includes:

[0035] When using the first model and the second model to perform anomaly detection on the log data generated by a user's access to a digital object, and the detection results of the first model and the second model are different, determine that the log data is abnormal;

[0036] Add the log data to a first database and generate a first prompt message; the first prompt message is used to prompt that the log data needs secondary analysis.

[0037] According to a second aspect of the embodiments of the present application, there is provided a digital object anomaly log detection device for implementing the digital object anomaly log detection method provided in the first aspect of the embodiments of the present application. The device includes:

[0038] A preprocessing module, configured to obtain an original log data set, preprocess the original log data set, and generate an initial sample set; the original log data set includes: normal log data and abnormal log data generated by a user accessing a target digital object within a specified time period;

[0039] A generation module, configured to generate supplementary abnormal samples according to a first ratio by using an adversarial generation algorithm based on the abnormal log data in the initial sample set, and add them to the initial sample set; perform upsampling and random sampling on the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set;

[0040] A training module, configured to train a target detection model based on the target sample set; the target detection model includes at least one of the following network architectures: a neural network or a random forest network;

[0041] A detection module, configured to use the trained target detection model to perform abnormal detection on the log data generated by a user accessing a digital object, and obtain a detection result; the detection result is normal or abnormal.

[0042] Optionally, the generation module, configured to perform upsampling and random sampling on the initial sample set to obtain a target sample set, specifically includes:

[0043] Dividing the initial sample set into an original training sample and an original test sample according to a preset ratio;

[0044] Using an oversampling algorithm to perform upsampling on the original training sample to generate a first synthetic sample;

[0045] Using an oversampling algorithm to perform upsampling on the original test sample to generate a second synthetic sample;

[0046] Performing random sampling on the original training sample and the first synthetic sample respectively, and using the obtained samples as the training set;

[0047] Performing random sampling on the original test sample set and the second synthetic sample respectively, and using the obtained samples as the test set.

[0048] Optionally, the training module is configured to train a target detection model based on the target sample set, specifically includes: training a first model by using the target sample set; the first model is pre-constructed based on a neural network; training a second model by using the target sample set; the second model is pre-constructed based on a random forest network;

[0049] The detection module includes a pluggable first detection sub-module and a second detection sub-module; the first detection sub-module is configured to use the trained first model to perform anomaly detection on the log data generated by the user accessing the digital object, and obtain a first determination result; the first determination result is normal or abnormal; the second detection sub-module is configured to use the trained second model to perform anomaly detection on the log data generated by the user accessing the digital object, and obtain a second determination result; the second determination result is normal or abnormal; according to the first determination result and / or the second determination result, the detection result is obtained.

[0050] Optionally, the detection module is further configured to determine that the log data is abnormal when performing anomaly detection on the log data generated by the user accessing the digital object using the first model and the second model, and the detection results of the first model and the second model are different;

[0051] The device further includes:

[0052] A prompt module, configured to add the log data to a first database and generate a first prompt message; the first prompt message is used to prompt that the log data needs secondary analysis.

[0053] By using the digital object anomaly log detection method provided by this application, first, an original log data set is obtained, and the original log data therein is preprocessed to generate an initial sample set. The original log data set includes normal log data and abnormal log data. For the abnormal log data in the original log data set, an adversarial generation algorithm is used to generate supplementary abnormal samples, so as to increase the number of abnormal samples in the initial sample set. Further, the initial sample set is upsampled and randomly sampled to obtain a target sample set, and the target sample set is divided into a training set and a test set for training a target calibration model. In this application, the target detection model adopts a pluggable architecture, including at least one of a neural network or a random forest network. The trained target detection model is used to perform anomaly detection on the log data, so as to obtain the detection result.

[0054] This application uses an adversarial generation algorithm to generate supplementary abnormal samples, thereby expanding the number of abnormal log data in the training sample set, solving the problem of low generalization of the model obtained by training due to uneven positive and negative samples, and improving the accuracy of the model in identifying abnormal log data. Moreover, in this application, a pluggable target detection model is built based on a neural network and a random forest network, reducing the dependence of the model on training samples, and further improving the generalization ability and accuracy of the model. Description of the Drawings

[0055] To more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments of the present application. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0056] Figure 1 is a flowchart of a digital object exception log detection method proposed in an embodiment of the present application;

[0057] Figure 2 is a schematic flowchart of training an object detection model in an embodiment of the present application;

[0058] Figure 3 is a schematic diagram of a digital object exception log detection device proposed in an embodiment of the present application. Detailed implementation manners

[0059] The following will clearly and completely describe the technical solutions in the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some, rather than all, of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present application.

[0060] It should be understood that the "one embodiment" or "an embodiment" mentioned throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of the present application. Therefore, the appearances of "in one embodiment" or "in an embodiment" throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0061] In various embodiments of the present application, it should be understood that the sequence numbers of the following processes do not mean the order of execution in sequence. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present application.

[0062] Here, the exemplary embodiments will be described in detail, and the examples are shown in the drawings. When the following description involves the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementation manners described in the following exemplary embodiments do not represent all the implementation manners consistent with the present application. On the contrary, they are only examples of the devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0063] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other.

[0064] The present application will be described in detail below with reference to the accompanying drawings and in combination with embodiments.

[0065] Figure 1 It is a flowchart of a method for detecting digital object abnormal logs proposed in an embodiment of the present application. As Figure 1 shown, this method is applied to the digital networking, and includes:

[0066] S1: Obtain an original log data set, preprocess the original log data set to generate an initial sample set; the original log data set includes: normal log data and abnormal log data generated by a user accessing a target digital object within a specified time period;

[0067] S2: Based on the abnormal log data in the initial sample set, use an adversarial generation algorithm to generate supplementary abnormal samples according to a first ratio and add them to the initial sample set;

[0068] S3: Upsample and randomly sample the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set;

[0069] S4: Train a target detection model based on the target sample set; the target detection model includes at least one of the following network architectures: a neural network or a random forest network;

[0070] S5: Use the trained target detection model to perform abnormal detection on the log data generated by a user accessing a digital object to obtain a detection result; the detection result is normal or abnormal.

[0071] In the existing solution, a detection model built with a single LSTM network is used to detect user log data. However, due to the large difference between positive and negative samples of log data and the data dependence of the LSTM network, it is difficult for the detection model to capture abnormal features, and it tends to predict as the majority class (normal log) and ignore the minority class (abnormal log) during prediction, resulting in low accuracy of the detection result.

[0072] In this embodiment, on the one hand, a target sample set for training a target detection model is constructed. Aiming at the problem of imbalance in the ratio of positive and negative training samples in the existing solution, this solution uses an adversarial generation algorithm to generate supplementary abnormal samples during the construction of the target sample set, taking the negative samples in the target sample set, so as to alleviate the problem of the decline in the generalization ability of the model caused by the imbalance in the ratio of positive and negative samples.

[0073] In the process of constructing the target sample set, the obtained original log data set is first preprocessed, including supplementing or deleting blank data in the captured original log data, etc., to ensure that the log data in the obtained initial sample set is complete.

[0074] Then, the adversarial generation algorithm is used to generate supplementary abnormal samples according to the first ratio to expand the number of abnormal samples in the initial sample set. Generally, when the positive-negative sample ratio is close to 1:100 or larger, it is considered that the data set is significantly imbalanced. In this case, the model may tend to over-learn normal samples and fail to effectively capture the characteristics of abnormal samples. In this embodiment, in order to address the problem of positive-negative sample imbalance, a smaller positive-negative sample ratio is adopted (for example, negative sample:positive sample = 1:10 or 1:20) to ensure that the model can fully learn the characteristics of abnormal samples and improve the performance of the model. In practical applications, the specific positive-negative sample ratio adopted can be fine-tuned according to requirements, and this is not limited in this embodiment.

[0075] In this embodiment, the first ratio is set to 10%, that is, the GAN (Generative Adversarial Networks) algorithm is used to generate supplementary abnormal samples accounting for 10% of the number of abnormal log data in the initial sample set. For example, if there are 2000 log data in the initial sample set and 200 of them are abnormal log data, according to the first ratio of 10%, 20 supplementary abnormal samples (abnormal log data) are generated through the GAN algorithm. Figure 2 It is a schematic flowchart of training a target detection model in an embodiment of the present application. As Figure 2 shown, in one embodiment, for the initial sample set, DANS (Distributed Adversarial Networks) is used to generate supplementary abnormal samples to expand the number of negative samples in the initial sample set. Among them, DANS is a special adversarial generation network (GAN) structure.

[0076] Then, SMOTE oversampling and random sampling are performed on the expanded initial sample set to obtain the target sample set, which is used to iteratively train the target detection model.

[0077] In this embodiment, on the other hand, a target detection model for log data detection is constructed. Given that there is a strong data dependence when using a single LSTM network for log data anomaly detection in the existing solution, as Figure 2As shown in the figure, in this embodiment, a pluggable target detection model is constructed based on the LSTM network and the random forest network. Users can select at least one of the two networks as the target detection model to detect abnormal log data. Each algorithm will generate a corresponding detection result, and these results will be integrated into the final detection result for users to audit.

[0078] In this embodiment, a pluggable target detection model is built based on multiple network structures. Users can select multiple algorithms to detect user log data, so as to better capture abnormal features. At the same time, when constructing training samples, an adversarial generation algorithm is used to generate multiple abnormal data to expand abnormal samples, reduce the impact of the imbalance between positive and negative sample ratios on the model, improve the generalization ability of the model, and thus improve the accuracy and reliability of detection.

[0079] As an implementation manner of this application, obtaining the original log data set includes:

[0080] Obtaining the normal log data and abnormal log data generated by accessing the target digital object within the specified time period as the first log data;

[0081] Obtaining normal log data and abnormal log data from the public data set as the second log data;

[0082] Merging the first log data and the second log data to obtain the original log data set.

[0083] In one embodiment, the original log data is obtained by data scraping. According to the determined time period, the access log data of the target digital object is scraped to obtain the first log data. The target digital object can be set according to the actual application scenario. For example, the target digital object can be the accessible data publicly disclosed within an enterprise. In addition, in order to make the features of the log data more diverse, a part of the log data is also obtained from the public data set in this embodiment as the second log data. The public data set can be selected according to the actual application scenario, and this application does not limit it. In this embodiment, the CMU-CERT data set is selected as the source of the second log data.

[0084] After obtaining the first log data and the second log data, the first log data and the second log data are merged to obtain the original log data set. In this embodiment, diverse log data is obtained from different data sources to generate the original log data set for subsequent training of the target detection model, so as to improve the ability of the model to capture abnormal features and further improve the accuracy of the model in identifying abnormal log data.

[0085] As an implementation manner of this application, preprocessing the original log data set to generate an initial sample set includes:

[0086] Delete invalid data and missing data;

[0087] Convert character data into unique values in encoded format;

[0088] Add judgment labels to each original log data set; the judgment labels are normal or abnormal;

[0089] Convert time information into timestamps;

[0090] Randomly sample part of the data in the original log data set to obtain intermediate data;

[0091] Normalize the intermediate data to obtain the initial sample set.

[0092] In one embodiment, preprocess the original data set, specifically including the following steps:

[0093] (1) Import dependency packages: Before starting the preprocessing, import dependency packages in advance. A dependency package is a program package used to provide the software running environment required by this solution;

[0094] (2) Data cleaning: Identify and delete invalid data or missing data (data with missing values) in the original log data set;

[0095] (3) Data format conversion: Convert character data in the original data set into unique values in encoded format;

[0096] (4) Add labels: According to the classification rules of the data set, add an error column to the abnormal log data and assign values (labels). Among them, part of the labels of the abnormal log data come from the original marks of the log data, and part of the labels come from rule-based data marks;

[0097] (5) Convert timestamps: Convert the Datetime time in the log data into timestamps for convenient calculation;

[0098] (6) Random sampling: Randomly sample the original data set to obtain intermediate data;

[0099] (7) Data normalization: Convert the absolute values in the intermediate data into relative values, complete data normalization, and obtain the initial sample set.

[0100] As an implementation manner of this application, perform upsampling and random sampling on the initial sample set to obtain a target sample set, including:

[0101] Divide the initial sample set into original training samples and original test samples according to a preset ratio;

[0102] An oversampling algorithm is adopted to upsample the original training samples to generate first synthetic samples;

[0103] An oversampling algorithm is adopted to upsample the original test samples to generate second synthetic samples;

[0104] Random sampling is respectively performed on the original training samples and the first synthetic samples, and the obtained samples are used as the training set;

[0105] Random sampling is respectively performed on the original test sample set and the second synthetic samples, and the obtained samples are used as the test set.

[0106] In one embodiment, after generating supplementary abnormal samples through the GAN algorithm and expanding the initial sample set, the SMOTE (Synthetic Minority Oversampling) algorithm is adopted to perform upsampling and random sampling operations on the initial sample set in sequence to obtain a target sample set. Specifically, before upsampling, the initial sample set also needs to be divided. In this embodiment, the training set and the test set are divided from the initial sample set according to a ratio of 8:2, and SMOTE upsampling and random sampling operations are respectively performed on the training set and the test set.

[0107] SMOTE upsampling increases the number of minority class samples by synthesizing new minority class samples (i.e., abnormal log data) to achieve a more balanced positive and negative sample ratio. In SMOTE upsampling, for each minority class sample, its K nearest neighbor samples are selected, and then new synthetic samples are generated according to the differences between these nearest neighbor samples. The specific operation is to randomly take a position on the line connecting two adjacent minority class samples and perform interpolation according to the weight of this position to generate new synthetic samples. SMOTE upsampling can help the model better learn the characteristics of minority class samples and improve the prediction ability for minority classes. It alleviates the sample imbalance problem by expanding the minority class samples in the training set. Specifically, the basic steps of the SMOTE algorithm are as follows:

[0108] (1) Calculate the K nearest neighbors of each minority class sample (i.e., abnormal log data) through the sampling nearest neighbor algorithm;

[0109] (2) Randomly select N samples from the K nearest neighbors for random linear interpolation. In the algorithm, the k parameter is default set to 5, indicating the number of neighbors used to calculate the nearest neighbor; the random_state parameter is used to set the random seed to ensure the repeatability of the results. In this embodiment, the range of the Random_state parameter setting is [40 - 50];

[0110] (3) Construct new minority class samples;

[0111] (4) Combine the new samples with the original data to generate a new training set.

[0112] After SMOTE oversampling, the number of negative samples in the initial sample set is further expanded.

[0113] Next, further perform random sampling on the initial sample set to balance the ratio of positive and negative samples. After SMOTE oversampling, since the synthesized samples (abnormal log data) are added to the initial sample set, it may lead to a situation where there are too many samples of the minority class (negative samples). To avoid the model overfitting to the samples of the minority class, random sampling is required, that is, randomly select a part of the samples from the samples synthesized by oversampling in the initial dataset and the original samples in the initial dataset to generate the target sample set. It should be noted that the random sampling operation, like the SMOTE oversampling operation, is also performed separately for the training set and the test set.

[0114] In this embodiment, the number of abnormal log data in the initial sample set is further expanded through the SMOTE oversampling operation, thereby further alleviating the problem of poor model generalization caused by the imbalance between positive and negative samples. Further, through the random sampling operation, the ratio of positive and negative samples in the sample set is closer to balance, avoiding the model relying on the samples of the minority class due to too many samples of the minority class during the subsequent training process, thereby further improving the generalization ability of the model. By combining the two steps of SMOTE oversampling and random sampling, the imbalanced dataset can be effectively processed and the performance of the model on the samples of the minority class can be improved.

[0115] As an implementation manner of the present application, training a target detection model based on the target sample set includes:

[0116] Training a first model using the target sample set; the first model is pre-constructed based on a neural network;

[0117] Training a second model using the target sample set; the second model is pre-constructed based on a random forest network;

[0118] Using the trained target detection model to perform anomaly detection on the log data generated by a user accessing a digital object to obtain a detection result, specifically including: using at least one of the first model and the second model to perform anomaly detection on the log data generated by a user accessing a digital object to obtain the detection result.

[0119] In one embodiment, a target detection model is built using a neural network and / or a random forest network. In this embodiment, the target detection model has a pluggable architecture, which mainly includes two model structures. The first model is based on an LSTM network (neural network), and the second model is based on a random forest network. Specifically, the network structure of the first model is a recurrent neural network based on the LSTM architecture, including: an input layer, two hidden layers, and an output layer. When building the network of the first model, first create a sequential model model through the Sequential() function, and then use the model.add() method to add neural network layers layer by layer. In this embodiment, the LSTM network contains two stacked hidden layers, and each layer contains 100 nodes.

[0120] The network structure of the second model constructs a classifier based on a random forest network, where the random forest network uses 100 different decision trees (i.e., n_estimators = 100).

[0121] The target detection network adopts a pluggable architecture. By integrating the first model and the second model, the LSTM algorithm and the random forest algorithm are combined to give full play to the respective advantages of the two networks. The LSTM network controls the inflow and outflow of information by designing different gating structures, has better long-term memory and better gradient fluidity, so as to effectively process long-term dependencies in the sequence; the random forest is an ensemble learning method that performs classification and regression analysis by constructing multiple decision trees and integrating their results, and has strong robustness to noise and outliers, which helps to reduce the risk of overfitting.

[0122] In the case where the user selects both the first model and the second model to detect abnormal log data, the detection results output by the first model and the detection results output by the second model are integrated to obtain the final detection result.

[0123] In this embodiment, by making the first model and the second model pluggable to give full play to the respective advantages of the two models and improve the overall performance of the target detection model. This pluggable architecture can give full play to the advantages of different algorithms, and at the same time combine SMOTE oversampling and random sampling to solve the problem of unbalanced positive and negative ratios of sample data, which not only improves the flexibility of the target detection model, but also enables the model to perform excellently on different types of data sets.

[0124] In one embodiment, in addition to using an LSTM network, the first model can also adopt other types of neural network structures according to the actual application scenario. For example, an autoencoder can be used as the first model.

[0125] As an implementation manner of this application, the digital object abnormal log detection method further includes:

[0126] When performing anomaly detection on the log data generated by a user's access to a digital object using the first model and the second model, and when the detection result of the first model is different from the detection result of the second model, determine that the log data is abnormal;

[0127] Add the log data to a first database and generate a first prompt message; the first prompt message is used to prompt that the log data requires secondary analysis.

[0128] In one embodiment, when a user simultaneously selects the first model and the second model for anomaly log data detection, two detection results are obtained respectively. When integrating the two detection results, if the detection result of the first model is inconsistent with the detection result of the second model, the integrated output of the final detection result is abnormal. On this basis, due to the inconsistent output results of the two models, the user needs to further perform secondary analysis on the log data in the subsequent traceability or auditing stage. Therefore, in this embodiment, a first database is pre-constructed to store the abnormal log data that needs to be subjected to secondary analysis, facilitating the user to uniformly process the abnormal log data that needs to be subjected to secondary analysis regularly.

[0129] When the detection results output by the two models are inconsistent, add the log data to the first database and generate a first prompt message to be displayed on the user interface. The first prompt message can be presented in the form of a log, used to prompt the user that the current abnormal log data needs to be subjected to secondary analysis.

[0130] Based on the same inventive concept, an embodiment of the present application provides a digital object anomaly log detection device. Refer to Figure 3 , Figure 3 is a schematic diagram of a digital object anomaly log detection device 100 proposed in an embodiment of the present application. As Figure 3 shown, the device includes:

[0131] A preprocessing module 101, configured to obtain an original log data set, preprocess the original log data set, and generate an initial sample set; the original log data set includes: normal log data and abnormal log data generated by a user accessing a target digital object within a specified time period;

[0132] A generation module 102, configured to, based on the abnormal log data in the initial sample set, use an adversarial generation algorithm to generate supplementary abnormal samples according to a first ratio and add them to the initial sample set; perform upsampling and random sampling on the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set;

[0133] A training module 103, configured to train an object detection model based on the target sample set; the object detection model includes at least one of the following network architectures: a neural network or a random forest network;

[0134] A detection module 104, configured to use the trained object detection model to perform anomaly detection on log data generated by a user accessing a digital object, and obtain a detection result; the detection result is normal or abnormal.

[0135] As an implementation manner of this application, the preprocessing module 101 is configured to obtain an original log data set, specifically including:

[0136] Obtain normal log data and abnormal log data generated by accessing the target digital object within the specified time period as first log data;

[0137] Obtain normal log data and abnormal log data from a public data set as second log data;

[0138] Merge the first log data and the second log data to obtain the original log data set.

[0139] As an implementation manner of this application, the preprocessing module 101 is configured to preprocess the original log data set to generate an initial sample set, specifically including:

[0140] Delete invalid data and missing data;

[0141] Convert character data into unique values in an encoded format;

[0142] Add a determination label to each original log data set; the determination label is normal or abnormal;

[0143] Convert time information into a timestamp;

[0144] Randomly sample a part of the data in the original log data set to obtain intermediate data;

[0145] Normalize the intermediate data to obtain the initial sample set.

[0146] As an implementation manner of this application, the generation module 102 is configured to perform upsampling and random sampling on the initial sample set to obtain a target sample set, specifically including:

[0147] Divide the initial sample set into original training samples and original test samples according to a preset ratio;

[0148] Use an oversampling algorithm to perform upsampling on the original training samples to generate first synthetic samples;

[0149] An oversampling algorithm is adopted to upsample the original test samples to generate second synthetic samples;

[0150] The original training samples and the first synthetic samples are respectively randomly sampled, and the obtained samples are used as the training set;

[0151] The original test sample set and the second synthetic samples are respectively randomly sampled, and the obtained samples are used as the test set.

[0152] As an implementation manner of the present application, the training module 103 is configured to train a target detection model based on the target sample set, specifically including: training a first model using the target sample set; the first model is pre-constructed based on a neural network; training a second model using the target sample set; the second model is pre-constructed based on a random forest network;

[0153] The detection module 104 includes a pluggable first detection sub-module and a second detection sub-module; the first detection sub-module is configured to use the trained first model to perform anomaly detection on the log data generated by a user accessing a digital object to obtain a first determination result; the first determination result is normal or abnormal; the second detection sub-module is configured to use the trained second model to perform anomaly detection on the log data generated by a user accessing a digital object to obtain a second determination result; the second determination result is normal or abnormal; according to the first determination result and / or the second determination result, the detection result is obtained.

[0154] As an implementation manner of the present application, the detection module 104 is further configured to determine that the log data is abnormal when using the first model and the second model to perform anomaly detection on the log data generated by a user accessing a digital object, and the detection results of the first model and the second model are different;

[0155] The device further includes:

[0156] A prompt module, configured to add the log data to a first database and generate a first prompt message; the first prompt message is used to prompt that the log data needs secondary analysis.

[0157] Based on the same inventive concept, an embodiment of the present application provides a readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it implements the steps in the digital object anomaly log detection method as described in any one of the above embodiments of the present application.

[0158] Based on the same inventive concept, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes, it implements the steps in the digital object exception log detection method described in any of the above embodiments of the present application.

[0159] Regarding the device in the above embodiments, the specific manners in which each module performs operations have been described in detail in the embodiments related to the method, and will not be elaborated herein.

[0160] The above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0161] For the method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that the present application is not limited by the described action sequences, because according to the present application, some steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and components involved are not necessarily essential to the present application.

[0162] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a device, or a computer program product. Therefore, the embodiments of the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk memories, CD-ROMs, optical memories, etc.) containing computer-usable program codes.

[0163] The embodiments of the present application are described with reference to the flowcharts and / or block diagrams of the method, terminal device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing terminal devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal devices generate a device for implementing the functions specified in Figure 1 one process or multiple processes and / or blocks Figure 1 one block or multiple blocks.

[0164] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal device to work in a specific manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction device that implements the functions specified in one or more processes and / or blocks Figure 1 in one or more processes and / or blocks Figure 1 specified in one or more blocks or multiple blocks.

[0165] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal device, such that a series of operation steps are executed on the computer or other programmable terminal device to produce a computer-implemented process, and thus the instructions executed on the computer or other programmable terminal device provide steps for implementing the functions specified in one or more processes and / or blocks Figure 1 in one or more processes and / or blocks Figure 1 specified in one or more blocks or multiple blocks.

[0166] Although the preferred embodiments of the embodiments of the present application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0167] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or terminal device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0168] The above has introduced in detail the method and device for detecting digital object exception logs provided by the present application. Specific examples are used in this text to elaborate on the principle and implementation manner of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A method for detecting digital object exception logs, characterized in that Applied to the digital networking, including: Obtain an original log dataset, preprocess the original log dataset to generate an initial sample set; the original log dataset includes: normal log data and abnormal log data generated by a user accessing a target digital object within a specified time period; Based on the abnormal log data in the initial sample set, use an adversarial generation algorithm to generate supplementary abnormal samples according to a first ratio to expand the number of abnormal samples in the initial sample set and add them to the initial sample set; Perform upsampling and random sampling on the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set; the upsampling is used to increase the number of abnormal samples by synthesizing new abnormal samples to achieve a more balanced positive and negative sample ratio; the random sampling is used to randomly select a part of the samples from the newly synthesized abnormal samples by upsampling and the samples in the initial sample set respectively, so that the ratio of positive and negative samples in the target sample set is closer to balance; Train a target detection model based on the target sample set; the target detection model is a pluggable architecture, including at least one of the following network architectures: a first model constructed based on an LSTM network, a second model constructed based on a random forest network; Use the trained target detection model to perform abnormal detection on the log data generated by a user accessing a digital object to obtain a detection result, including: use at least one of the first model and the second model to perform abnormal detection on the log data generated by a user accessing a digital object to obtain the detection result; the detection result is normal or abnormal.

2. The digital object exception log detection method according to claim 1, wherein Obtain an original log dataset, including: Obtain the normal log data and abnormal log data generated by accessing the target digital object within the specified time period as first log data; Obtain normal log data and abnormal log data from a public dataset as second log data; Merge the first log data and the second log data to obtain the original log dataset.

3. The digital object exception log detection method according to claim 1, wherein Preprocess the original log dataset to generate an initial sample set, including: Delete invalid data and gap data; Convert character data into unique values in an encoded format; Add a determination label to each original log dataset; the determination label is normal or abnormal; Convert time information into a timestamp; Use random sampling to extract part of the data in the original log dataset to obtain intermediate data; Perform normalization processing on the intermediate data to obtain the initial sample set.

4. The digital object exception log detection method according to claim 1, wherein Perform upsampling and random sampling on the initial sample set to obtain a target sample set, including: Divide the initial sample set into an original training sample and an original test sample according to a preset ratio; Use an oversampling algorithm to perform upsampling on the original training sample to generate a first synthetic sample; Use an oversampling algorithm to perform upsampling on the original test sample to generate a second synthetic sample; Perform random sampling on the original training sample and the first synthetic sample respectively, and use the obtained samples as the training set; Randomly sample the second synthetic samples from the original test sample set, and use the obtained samples as the test set.

5. The digital object exception log detection method according to claim 1, characterized in that Training a target detection model based on the target sample set includes: Training a first model using the target sample set; Training a second model using the target sample set.

6. The digital object exception log detection method according to claim 5, characterized in that, It also includes: When using the first model and the second model to perform anomaly detection on the log data generated by a user accessing a digital object, and the detection results of the first model are different from those of the second model, determine that the log data is abnormal; Add the log data to the first database and generate a first prompt message; The first prompt message is used to prompt that the log data needs secondary analysis.

7. A digital object exception log detection device, characterized in that For implementing the method according to any one of claims 1-6, it includes: A preprocessing module, configured to obtain an original log data set, preprocess the original log data set, and generate an initial sample set; the original log data set includes: normal log data and abnormal log data generated by a user accessing a target digital object within a specified time period; A generation module, configured to, based on the abnormal log data in the initial sample set, use an adversarial generation algorithm to generate supplementary abnormal samples according to a first ratio to expand the number of abnormal samples in the initial sample set and add them to the initial sample set; perform upsampling and random sampling on the initial sample set to obtain a target sample set; the target sample set includes a training set and a test set; the upsampling is used to increase the number of abnormal samples by synthesizing new abnormal samples to achieve a more balanced positive and negative sample ratio; the random sampling is used to randomly select a part of the samples from the newly synthesized abnormal samples by upsampling and the samples in the initial sample set respectively, so that the ratio of positive and negative samples in the target sample set is closer to balance; A training module, configured to train a target detection model based on the target sample set; the target detection model is a pluggable architecture, including at least one of the following network architectures: a first model constructed based on an LSTM network, a second model constructed based on a random forest network; A detection module, including a pluggable first detection sub-module and a second detection sub-module; the first detection sub-module is configured to use the trained first model to perform anomaly detection on the log data generated by a user accessing a digital object to obtain a first determination result; the first determination result is normal or abnormal; the second detection sub-module is configured to use the trained second model to perform anomaly detection on the log data generated by a user accessing a digital object to obtain a second determination result; the second determination result is normal or abnormal; obtain the detection result according to the first determination result and / or the second determination result.

8. The digital object exception log detection device according to claim 7, characterized in that The generation module, configured to perform upsampling and random sampling on the initial sample set to obtain a target sample set, specifically includes: Divide the initial sample set into an original training sample and an original test sample according to a preset ratio; Use an oversampling algorithm to perform upsampling on the original training sample to generate a first synthetic sample; An oversampling algorithm is adopted to upsample the original test samples to generate second synthetic samples; Random sampling is respectively performed on the original training samples and the first synthetic samples, and the obtained samples are used as the training set; Random sampling is respectively performed on the original test sample set and the second synthetic samples, and the obtained samples are used as the test set.

9. The digital object exception log detection device according to claim 7, wherein The training module is configured to train an object detection model based on the target sample set, specifically including: Training a first model using the target sample set; Training a second model using the target sample set.

10. The digital object exception log detection device according to claim 9, characterized in that, The detection module is further configured to determine that the log data is abnormal when using the first model and the second model to perform anomaly detection on the log data generated by a user accessing a digital object, and the detection results of the first model and the second model are different; The device further includes: A prompt module, configured to add the log data to a first database and generate a first prompt message; the first prompt message is used to prompt that the log data needs secondary analysis.

Images