Data processing method and device, electronic equipment and computer readable storage medium
The online data processing method addresses inefficiencies in offline analysis by performing real-time data classification and model training, enhancing data processing efficiency and security in Internet data anomaly identification.
Patent Information
- Application Number
- CN202410360967.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-27
- Publication Date
- 2024-12-27
AI Technical Summary
Existing data processing methods for Internet data anomaly identification rely on offline analysis, which is inefficient and slow, leading to suboptimal data processing efficiency.
Implement an online data processing method that includes determining target data features, performing first data classification, obtaining sample data based on a trigger condition, training a target classification model, and conducting second data classification to achieve real-time automatic data processing.
Enables real-time automatic data processing by eliminating the need for offline analysis, thereby improving data processing efficiency and security through accurate and timely anomaly detection.
Smart Images
Figure CN119202872A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, electronic device and computer-readable storage medium. Background Art
[0002] With the development of computer technology and Internet technology, there is an increasing demand for the management and control of Internet data. For example, in the scenario of Internet data anomaly identification (or anomaly interception), data anomaly identification, classification, control, interception and other processing are required to improve security.
[0003] In the related art, for platform data in the scenario of Internet data anomaly identification, offline analysis is usually used to determine whether there is data anomaly and thus determine whether there is risk, and risk control (such as abnormal data identification, interception or processing) is achieved by setting a strategy based on the judgment threshold. The problem with the related art is that offline analysis needs to be performed at a certain period, and the data processing efficiency is slow. Summary of the invention
[0004] The embodiments of the present application provide a data processing method, device, electronic device and computer-readable storage medium. Without offline analysis, it can determine whether to perform model training based on the online first data classification, and perform a second data classification according to the target classification model obtained by the online training to process the platform data according to the second data classification result, thereby realizing real-time automatic data processing, which is conducive to improving data processing efficiency.
[0005] A first aspect of an embodiment of the present application provides a data processing method, the method comprising:
[0006] Determine a target data feature corresponding to a target classification task from data features of multiple data dimensions of platform data;
[0007] Performing a first data classification on the platform data according to the target data characteristics to obtain a first data classification result;
[0008] If the first data classification result meets the preset model training trigger condition, sample data corresponding to the target classification task is obtained, wherein a training sample in the sample data includes a training feature and a classification result label, and the training feature is determined according to a data feature of at least one data dimension of the platform data;
[0009] The target classification model corresponding to the target classification task is obtained by training the sample data;
[0010] Perform a second data classification on the platform data according to the target classification model to obtain a second data classification result, and process the platform data according to the second data classification result.
[0011] A second aspect of an embodiment of the present application provides a data processing device, the device comprising:
[0012] A target data feature determination module is used to determine the target data feature corresponding to the target classification task from the data features of multiple data dimensions of the platform data;
[0013] A first data classification module, used to perform a first data classification on the platform data according to the target data characteristics to obtain a first data classification result;
[0014] A sample acquisition module, configured to acquire sample data corresponding to the target classification task if the first data classification result meets a preset model training trigger condition, wherein a training sample in the sample data includes a training feature and a classification result label, and the training feature is determined according to a data feature of at least one data dimension of the platform data;
[0015] A model training module is used to train the target classification model corresponding to the target classification task according to the sample data;
[0016] The data processing module is used to perform a second data classification on the platform data according to the above target classification model, obtain a second data classification result, and process the above platform data according to the above second data classification result.
[0017] In some optional embodiments, the sample acquisition module includes:
[0018] An online sample acquisition unit, used to generate the sample data according to the first data classification result and the platform data corresponding to the first data classification result; and / or
[0019] The offline sample acquisition unit is used to acquire sample data corresponding to the above target classification task based on the offline data warehouse.
[0020] In some optional embodiments, the online sample acquisition unit is specifically used to:
[0021] Performing feature encoding conversion on data features of multiple data dimensions of the platform data to obtain feature tensors corresponding to the platform data;
[0022] The sample data is generated according to the feature tensor corresponding to the platform data and the first data classification result.
[0023] In some optional embodiments, the training features of the training samples include data features of the multiple data dimensions except for the target data features.
[0024] In some optional embodiments, the above model training module is specifically used for:
[0025] Obtain the classification model to be trained and the model hyperparameters corresponding to the above target classification task;
[0026] According to the above sample data and the above model hyperparameters, the above classification model is trained to obtain the above target classification model.
[0027] In some optional embodiments, the above device further includes:
[0028] A model comparison module is used to: determine the same category historical classification model corresponding to the target classification model from the trained historical classification models according to the target classification task;
[0029] According to the above target classification model and / or the above historical classification model of the same category, obtain model test sample data;
[0030] Performing a data classification test on the model test sample data according to the target classification model to obtain a first test result;
[0031] Performing data classification test on the above model test sample according to the above historical classification model of the same category to obtain a second test result;
[0032] According to the first test result and the second test result, a model with the most accurate data classification is determined from the target classification model and the historical classification models of the same category to replace the target classification model.
[0033] In some optional embodiments, the data processing module is specifically used for:
[0034] According to the target classification task and / or the target classification model, obtaining data control rules corresponding to the target classification task;
[0035] Deploy the target classification model, and perform a second data classification on the platform data according to the deployed target classification model to obtain a second data classification result;
[0036] The platform data is processed according to the data control rules and the second data classification result.
[0037] A third aspect of an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores multiple instructions; the processor loads instructions from the memory to execute the steps in the data processing method provided in the first aspect of the embodiment of the present application.
[0038] The fourth aspect of the embodiments of the present application provides a computer-readable storage medium, which stores multiple instructions. The instructions are suitable for loading by a processor to execute the steps in the data processing method provided in the first aspect of the embodiments of the present application.
[0039] By adopting the scheme of the embodiment of the present application, the target data features corresponding to the target classification task can be determined from the data features of multiple data dimensions of the platform data; the first data classification is performed on the above-mentioned platform data according to the above-mentioned target data features to obtain a first data classification result; if the above-mentioned first data classification result meets the preset model training trigger condition, the sample data corresponding to the above-mentioned target classification task is obtained, wherein a training sample in the above-mentioned sample data includes a training feature and a classification result label, and the above-mentioned training feature is determined according to the data features of at least one data dimension of the above-mentioned platform data; the target classification model corresponding to the above-mentioned target classification task is obtained according to the above-mentioned sample data training; the second data classification is performed on the platform data according to the above-mentioned target classification model to obtain a second data classification result, and the above-mentioned platform data is processed according to the above-mentioned second data classification result.
[0040] In this way, when processing data, there is no need to perform offline analysis. Instead, it can be determined whether to perform model training based on the online first data classification, and the second data classification can be performed according to the target classification model obtained through the online training to process the platform data according to the second data classification results, thereby realizing real-time automatic data processing, which is conducive to improving data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.
[0042] Figure 1 It is a flowchart of a data processing method provided in an embodiment of the present application;
[0043] Figure 2 It is a configuration diagram of a classification model provided in an embodiment of the present application;
[0044] Figure 3 This is a schematic diagram of an abnormal data interception effect based on a target classification model provided in an embodiment of the present application;
[0045] Figure 4 This is a schematic diagram of a data anomaly identification system architecture provided by an embodiment of the present application;
[0046] Figure 5It is a schematic diagram of a storage format of sample data provided in an embodiment of the present application;
[0047] Figure 6 This is a schematic diagram of a file format after data conversion provided by an embodiment of the present application;
[0048] Figure 7 is a structural block diagram of a data processing device provided in an embodiment of the present application;
[0049] Figure 8 It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.
[0051] The embodiments of the present application provide a data processing method, device, electronic device and computer-readable storage medium. Specifically, the data processing method of the embodiments of the present application can be performed by a computer device, wherein the computer device can be a terminal or a server. The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, a personal computer (PC, Personal Computer), a personal digital assistant (PDA, Personal Digital Assistant). The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms.
[0052] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.
[0053] The first aspect of the embodiment of the present application provides a data processing method. It should be noted that the embodiment of the present application takes data anomaly identification as an example for specific description, but in actual application, abnormal data interception, abnormal data control, etc. can also be implemented based on the data processing method provided in the embodiment of the present application, which is not specifically limited here. Figure 1 , Figure 1 : is a flow chart of a data processing method provided in an embodiment of the present application. The specific flow of the data processing method can be as follows:
[0054] 101. Determine target data features corresponding to the target classification task from data features of multiple data dimensions of platform data.
[0055] Among them, the above-mentioned platform data can be data in the data anomaly identification platform in the Internet data anomaly identification scenario, or it can be data in other platforms that require data monitoring and processing, and is not specifically limited here.
[0056] The above target classification task is a classification task that needs to be performed on the platform data. The target classification task can be set and adjusted according to actual needs and is not specifically limited here. For example, the above platform data can be the likes data or comment data for the content published on the platform, and the corresponding target classification task can be to classify the likes data or comment data and determine whether it is normal user interaction data or abnormal interaction data such as abnormal likes or comments.
[0057] Specifically, each piece of platform data has data features of multiple data dimensions. According to the target classification task, one or more target data features for classification can be determined from the data features of the multiple data dimensions.
[0058] 102. Perform a first data classification on the platform data according to the target data characteristics to obtain a first data classification result.
[0059] In an application scenario, the above-mentioned first data classification is a process of abnormal traffic perception for platform data, that is, firstly judging whether the platform data is abnormal data by abnormal traffic perception, so as to obtain the first data classification result.
[0060] 103. If the first data classification result satisfies a preset model training trigger condition, obtain sample data corresponding to the target classification task, wherein a training sample in the sample data includes training features and classification result labels, and the training features are determined based on data features of at least one data dimension of the platform data.
[0061] Among them, the above-mentioned model training trigger condition can be set and adjusted according to actual needs. For example, it can be set to that the number of abnormal data in the first data classification result exceeds a preset abnormal number threshold within a preset time length, but it is not a specific limitation.
[0062] If the first data classification result meets the preset model training trigger condition, the sample data corresponding to the target classification task is obtained for model training. In this way, the abnormality perception and model training process in the data processing process are connected, and it can be determined whether to trigger model training based on the real-time online abnormality perception results to achieve better data classification and data processing, improve the efficiency of data processing and the security of the entire data anomaly identification system.
[0063] In an application scenario, the training features of the training samples in the above sample data may include data features of all data dimensions of the platform data, so that the target classification model obtained through training can perform more accurate data classification.
[0064] In another application scenario, the training features of the training samples in the above-mentioned sample data may include one or more data features among the data features of all data dimensions of the platform data except the target data features. For example, in an embodiment of the present application, the training features of the above-mentioned training samples include data features among the data features of the above-mentioned multiple data dimensions except the above-mentioned target data features.
[0065] In this way, using features different from anomaly perception for model training is conducive to reducing the possibility of abnormal data providers discovering the standards for data anomaly judgment and thus evading them, further improving the security of the entire data anomaly identification system.
[0066] In the embodiment of the present application, sample data for model training may be obtained by online sample generation and / or offline sample acquisition. Specifically, the sample data corresponding to the target classification task may be obtained, including:
[0067] Generate the sample data according to the first data classification result and the platform data corresponding to the first data classification result; and / or
[0068] The sample data corresponding to the above target classification task is obtained based on the offline data warehouse.
[0069] The above-mentioned offline data warehouse is a pre-set database for storing offline data, which can be selected and adjusted according to actual needs and is not specifically limited here.
[0070] In an application scenario, the generating of the sample data according to the first data classification result and the platform data corresponding to the first data classification result includes:
[0071] Performing feature encoding conversion on data features of multiple data dimensions of the platform data to obtain feature tensors corresponding to the platform data;
[0072] The sample data is generated according to the feature tensor corresponding to the platform data and the first data classification result.
[0073] Specifically, for data features, feature encoding is required to convert them into feature tensors, so as to ensure that the corresponding data features can be identified and processed according to the corresponding data features during the model training process.
[0074] 104. A target classification model corresponding to the target classification task is obtained by training the sample data.
[0075] In an application scenario, corresponding classification models to be trained may be pre-set for different target classification tasks. In the embodiment of the present application, a classification model set for a target classification task is a binary classification model as an example for specific description, but it is not a specific limitation.
[0076] Specifically, the target classification model corresponding to the target classification task obtained by training the sample data includes:
[0077] Obtain the classification model to be trained and the model hyperparameters corresponding to the above target classification task;
[0078] According to the above sample data and the above model hyperparameters, the above classification model is trained to obtain the above target classification model.
[0079] The above-mentioned model hyperparameters refer to the parameters corresponding to the above-mentioned classification model that are preset and do not change with the training process. For example, they can be parameters in the machine learning algorithm used by the classification model that are not learned by the algorithm itself and need to be manually set. Specifically, for different target classification tasks, a binary classification model based on the same classification algorithm can be set, but for different specific scenarios or specific classification tasks, different model hyperparameters can be set to obtain a more accurate classification effect.
[0080] 105. Perform a second data classification on the platform data according to the target classification model to obtain a second data classification result, and process the platform data according to the second data classification result.
[0081] Specifically, after the training is completed and the target classification model is obtained, the second data classification can be performed according to the target classification model, so as to perform data processing based on the second data classification result. It should be noted that based on the target classification model, all platform data including platform data that have undergone the first data classification can be classified and processed, or only platform data that has not undergone the first data classification can be classified and processed, which is not specifically limited here.
[0082] In the embodiment of the present application, the target classification model obtained after training is also compared with the historical classification models of the same category that have been trained, and the model with the best classification effect is selected as the target classification model. Specifically, before performing the second data classification on the platform data according to the target classification model and obtaining the second data classification result, the method further includes:
[0083] According to the target classification task, determine the same category historical classification model corresponding to the target classification model from the trained historical classification models;
[0084] According to the above target classification model and / or the above historical classification model of the same category, obtain model test sample data;
[0085] Performing a data classification test on the model test sample data according to the target classification model to obtain a first test result;
[0086] Performing data classification test on the above model test sample according to the above historical classification model of the same category to obtain a second test result;
[0087] According to the first test result and the second test result, a model with the most accurate data classification is determined from the target classification model and the historical classification models of the same category to replace the target classification model.
[0088] Among them, the above-mentioned historical classification model of the same category is a model of the same category as the target classification model that has been historically trained, for example, a target classification model trained for the same or similar target classification task, or a target classification model obtained based on the same classification model to be trained and the same model hyper-parameter training, which is not specifically limited here.
[0089] Furthermore, performing a second data classification on the platform data according to the target classification model to obtain a second data classification result, and processing the platform data according to the second data classification result includes:
[0090] According to the target classification task and / or the target classification model, obtaining data control rules corresponding to the target classification task;
[0091] Deploy the target classification model, and perform a second data classification on the platform data according to the deployed target classification model to obtain a second data classification result;
[0092] The platform data is processed according to the data control rules and the second data classification result.
[0093] Among them, the above-mentioned data control rules are pre-set rules for processing different categories of data classified by the target classification task. For example, the classification results corresponding to the target classification task may include data belonging to the target classification (abnormal data) and data not belonging to the target classification (not abnormal data). The data control rules may be to intercept abnormal data, or to control the flow of abnormal data, report the source of abnormal data (to intercept or mark data from the same source), etc., which are not specifically limited here.
[0094] In the embodiment of the present application, the above data processing method is also described in detail in combination with a specific application scenario. Specifically, the above data processing method is applied to the automatic confrontation scenario of Internet data anomaly identification, for example, machine crawler identification and attack are performed based on the above data processing method. First, some identification capabilities are embedded on the end side (such as adding traffic signatures) to achieve anomaly perception (i.e., perform the first data classification). When the traffic enters the server, it is verified by the signature verification service, and the verification result is appended to the label of the soft verification signature, and flows into the data anomaly identification system with the request. In the data anomaly identification system, according to whether there is a soft verification signature, the traffic is split into black samples (soft verification signature failed + purification logic) and white samples (soft verification signature passed + purification logic). Through services that can support data processing and processing (such as near-line platforms), real-time traffic is processed, and finally training sample data can be generated in quasi-real time. The above process can realize a simple anomaly perception framework. If the magnitude of the soft verification signature within 5 minutes exceeds a certain threshold, it is considered that abnormal traffic has occurred and further data processing is required. In the platform, relevant sample data is obtained, and model training is performed in combination with the classification model algorithm and the configured feature processing logic. The resulting model can be used to identify machine traffic. The trained model is automatically uploaded to the online real-time data anomaly recognition service for grayscale pre-running. In the grayscale stage, the accuracy and recall rate of the model are compared to see if they meet the requirements for online launch, and a decision is made as to whether the model needs to be launched.
[0095] It should be noted that the training of the classification model in the embodiment of the present application can be achieved by processing and generating sample data based on data anomaly identification logs in various Internet anti-cheating scenarios.
[0096] The definition of a prediction interface of a model provided in an embodiment of the present application is as follows:
[0097]
[0098] Among them, modelId and trainingId in the passed parameters are used to locate which model needs to be used, and params stores the input parameters required for model prediction. The model output is the value parameter, which is a value between 0 and 1, indicating the probability that the predicted data is abnormal data.
[0099] In the scenario of machine crawler identification and attack, all machine traffic needs to be identified and automatically handled. In this case, the identification rules of abnormal traffic can be defined in the automatic confrontation platform. One rule is as follows:
[0100]
[0101]
[0102] It should be noted that when generating sample data, it is necessary to perform feature encoding for data features to convert features from the original data format into a digital or vector format that can be processed by the machine learning algorithm. The encoding method used when performing feature encoding can be set and adjusted according to actual needs, and is not specifically limited here. In the embodiment of the present application, for different data features, mapping or encoding methods can be selected according to actual needs for feature encoding. For example, when selecting the mapping method, numerical mapping is performed according to a preset numerical mapping rule, and when selecting the encoding method, numerical encoding is performed according to a preset encoding rule. Further, feature engineering can also be performed on the data according to actual needs, that is, the original data is processed and converted to extract features useful for target prediction, thereby improving the prediction performance and generalization ability of the model. For example, after feature engineering, a data tensor can be obtained, and specifically, input data, model parameters, intermediate calculation results and other data can all be in the form of data tensors.
[0103] Figure 2 is a schematic diagram of a configuration of a classification model provided in an embodiment of the present application, such as Figure 2 As shown, the configured classification model uses the XGBoost algorithm, and the specific hyperparameters can be referred to Figure 2 The setting is not specifically limited here. It should be noted that Figure 2 The specific values of each hyperparameter are coded in the text, but are not used as specific limitations.
[0104] In the embodiment of the present application, the sample data used in model training can be generated in real time based on the data after online abnormal perception. In the embodiment of the present application, a parameter control interface in the sample generation process can be provided to the user based on a visual interface, so as to facilitate the user to set the number and proportion of black and white samples according to actual needs. Based on the above parameter control interface, various control parameters in the sample generation process can be adjusted. For example, the ratio of the test set and the training set, the upper and lower limits of the number of samples, and the time interval to which the data used to generate the sample belong can be controlled, which is not specifically limited here.
[0105] Furthermore, a visualization interface for displaying the model training status is also provided in the embodiment of the present application. Based on the visualization interface, the training status of different models can be viewed in real time. For example, the model ID, training ID, training task creation time, task processing status (such as whether the training task is in progress or completed, or which specific training stage it is in), the number of associated training features, model factors, creators of training tasks, etc. of different models trained can be viewed. Among them, model factors refer to a special type of factors defined in the rule engine, which receive other factors as parameters, call model services and return model prediction results.
[0106] In the embodiment of the present application, a visual interface for configuring the control parameters of the target classification model can also be provided, for example, it can be used to control the training priority, training risk level, offline accuracy baseline, offline recall baseline, model strategy PK time and other parameters of the target classification model. Among them, the model strategy PK time is the time for comparing the target classification model with the historical classification model of the same category.
[0107] Figure 3 is a schematic diagram of an abnormal data interception effect based on a target classification model provided in an embodiment of the present application, such as Figure 3 As shown, in the embodiment of the present application, based on the deployed target classification model, data classification and interception of abnormal data can be better achieved.
[0108] Furthermore, in an embodiment of the present application, the above-mentioned data processing method can be applied to a specific Internet data anomaly identification platform to realize automatic identification and processing of abnormal data. Figure 4 This is a schematic diagram of the data anomaly identification system architecture provided by an embodiment of the present application. Among them, Rugal is an offline task scheduling platform, Feantispam is the front end of the anomaly identification configuration platform, Moss-service-web is a model management backend service, Moss-service-training is a model training cluster, Moss-service-serving is a model serving cluster, Sec-moss is a model management core database, Penancestare is an anomaly identification access gateway, Hammurabi-service-default is a real-time rule engine backend service, Hulk-service-default is a rule engine access layer service, Gurdian-management is a near-line engine configuration service, and MOSS is a data anomaly identification platform.
[0109] In the embodiment of the present application, the specific model training process includes multiple stages such as data preparation, data sampling, data conversion, model training and post-processing. Figure 4The data anomaly recognition system architecture shown can implement the above-mentioned model training process, thereby performing data processing based on the target classification model obtained through training, achieving seamless integration between the model training process and the Internet data anomaly recognition platform, and opening up the entire chain from anomaly perception, model training, model evaluation to online interception, thereby improving the efficiency and accuracy of intercepting abnormal data.
[0110] Specifically, the data preparation stage is the preparation stage of sample data. When the sample data corresponding to the model is in the processing process, the model training stays in the data preparation stage until the training sample is prepared and enters the next stage. In a specific application scenario, it is first necessary to define a standard sample structure. The sample structure needs to be aligned with the structure of the real-time log of data anomaly identification to achieve data alignment of the model traning and serving processes. In an application scenario, the defined standard sample structure includes descriptions such as field name, field Chinese name, field description, field security level, data type, and whether it is a primary key. In an embodiment of the present application, the sample structure can be defined as follows:
[0111] 1) The training is mainly based on binary classification models. The training samples need to include training-related X and Y values.
[0112] 2) X values: stored in a column called factors, with a data type of map<string,string> , a set of factor names corresponding to online traffic.
[0113] 3) Y value: stored in a column named label, the data type is string, 0 represents a white sample, and 1 represents a black sample. White samples represent samples with no abnormalities, and black samples represent samples with abnormalities.
[0114] In the offline scenario, by defining the location information where offline samples need to be stored (fixedly writing all offline samples to a specific schema), and agreeing on the structure of the sample library database table (for example, defining all data tables starting with moss_training_sample_ as training sample tables, so that the original database can be reused without creating a new database).
[0115] After the offline sample processing is completed, the data warehouse will generate a task completion message and send it through the message middleware. You can monitor all task completion messages in the data warehouse and consume the completion messages of the relevant sample library you are concerned about to learn whether the data has been prepared.
[0116] The platform starts an asynchronous thread operation to synchronously write the sample data in the remote data warehouse to the sample library of the data anomaly identification platform (MOSS), and write the data synchronization status to the sample library to complete the sample data synchronization process, thereby realizing the preparation of sample data. In actual use, you can choose to process samples from the offline logs of the data warehouse according to your needs, or you can directly obtain samples through the log stream of the near-line processing real-time rule engine, which is not specifically limited here.
[0117] After the data preparation stage is over, it means that the original data used for training is ready and the data sampling stage can be entered. This stage mainly samples various types of data required for the model training process from the original training data according to the relevant sampling conditions and parameters of the model configuration, and generates various types of sample data required for the subsequent process by random sampling. In an embodiment of the present application, a visual interface for controlling parameters in the data sampling process can also be provided to the user to better control the data sampling process. For example, the upper and lower limits of the sample, the ratio configuration of the training set and the test set, the configuration of the verification set, etc. can be realized based on the visual interface. After the data sampling is completed, the relevant files are generated, mainly including black samples for training, white samples for training, black samples for testing, white samples for testing, black samples for verification, and white samples for verification. Figure 5 is a schematic diagram of a storage format of sample data provided in an embodiment of the present application, such as Figure 5 As shown, the sample data is stored in the form of a CSV file. The file content is divided into two columns, corresponding to labels and factors respectively.
[0118] For the extracted sample data, data conversion is also required to convert the various types of data obtained in the data sampling stage into tensors with labeled data through feature engineering, so as to facilitate use in the subsequent model training stage. In an embodiment of the present application, for each feature engineering, the feature conversion method (such as encoding, mapping, etc.) associated with the feature engineering can be pre-defined. During model training, corresponding feature conversion is performed for the features that the model depends on. During feature conversion, the input is a key in the factors of the sample data, and the output is a vector (value type) after conversion.
[0119] Figure 6 is a schematic diagram of a file format after data conversion provided by an embodiment of the present application, such as Figure 6 As shown, in the embodiment of the present application, a tensor file is generated after data conversion and stored in the svm file format to improve the efficiency of data reading and use in subsequent processing.
[0120] Furthermore, the model is trained based on the processed data. When training the model, the algorithm to be used is specified, and the relevant training hyperparameters are set. On the tensor data set obtained during the data conversion process, multiple rounds of iterative training and fitting are performed to finally obtain a model that can be used to predict and classify new samples (traffic reacquired by the data anomaly recognition system). Specifically, the selected algorithm and relevant hyperparameters are applied to the tensor in the svm format obtained by data conversion, and finally a model that can be used for model serving is trained and serialized into a file.
[0121] In the embodiment of the present application, one or more model factors, model rules and model strategies can be pre-set according to actual needs, and after the model training is completed, the post-processing stage is entered to realize the deployment of the model. Specifically, the automatic confrontation platform will complete the signaling interaction of the control plane with the anomaly identification real-time rule engine, put the rules of the model class online, verify the relevant online threshold, and PK with the model effects of the same type, and finally realize the automatic online model and traffic interception. Based on the needs of the anomaly identification rule engine, various model factors, model rules and model strategies for online traffic interception are created and generated.
[0122] By adopting the scheme of the embodiment of the present application, the target data features corresponding to the target classification task can be determined from the data features of multiple data dimensions of the platform data; the first data classification is performed on the above-mentioned platform data according to the above-mentioned target data features to obtain a first data classification result; if the above-mentioned first data classification result meets the preset model training trigger condition, the sample data corresponding to the above-mentioned target classification task is obtained, wherein a training sample in the above-mentioned sample data includes a training feature and a classification result label, and the above-mentioned training feature is determined according to the data features of at least one data dimension of the above-mentioned platform data; the target classification model corresponding to the above-mentioned target classification task is obtained according to the above-mentioned sample data training; the second data classification is performed on the platform data according to the above-mentioned target classification model to obtain a second data classification result, and the above-mentioned platform data is processed according to the above-mentioned second data classification result.
[0123] In this way, when processing data, there is no need to perform offline analysis. Instead, it can be determined whether to perform model training based on the online first data classification, and the second data classification can be performed according to the target classification model obtained through the online training to process the platform data according to the second data classification results, thereby realizing real-time automatic data processing, which is conducive to improving data processing efficiency.
[0124] In view of the data processing method provided in the first aspect of the embodiment of the present application, the second aspect of the embodiment of the present application provides a data processing device, see Figure 7 , Figure 7 : is a structural block diagram of a data processing device provided in an embodiment of the present application, the device comprising:
[0125] A target data feature determination module 701 is used to determine a target data feature corresponding to a target classification task from data features of multiple data dimensions of platform data;
[0126] A first data classification module 702, configured to perform a first data classification on the platform data according to the target data characteristics to obtain a first data classification result;
[0127] A sample acquisition module 703 is used to acquire sample data corresponding to the target classification task if the first data classification result meets a preset model training trigger condition, wherein a training sample in the sample data includes a training feature and a classification result label, and the training feature is determined according to a data feature of at least one data dimension of the platform data;
[0128] A model training module 704 is used to train a target classification model corresponding to the target classification task according to the sample data;
[0129] The data processing module 705 is used to perform a second data classification on the platform data according to the target classification model, obtain a second data classification result, and process the platform data according to the second data classification result.
[0130] In some optional embodiments, the sample acquisition module 703 includes:
[0131] An online sample acquisition unit, used to generate the sample data according to the first data classification result and the platform data corresponding to the first data classification result; and / or
[0132] The offline sample acquisition unit is used to acquire sample data corresponding to the above target classification task based on the offline data warehouse.
[0133] In some optional embodiments, the online sample acquisition unit is specifically used to:
[0134] Performing feature encoding conversion on data features of multiple data dimensions of the platform data to obtain feature tensors corresponding to the platform data;
[0135] The sample data is generated according to the feature tensor corresponding to the platform data and the first data classification result.
[0136] In some optional embodiments, the training features of the training samples include data features of the multiple data dimensions except for the target data features.
[0137] In some optional embodiments, the model training module 704 is specifically used for:
[0138] Obtain the classification model to be trained and the model hyperparameters corresponding to the above target classification task;
[0139] According to the above sample data and the above model hyperparameters, the above classification model is trained to obtain the above target classification model.
[0140] In some optional embodiments, the above device further includes:
[0141] A model comparison module is used to: determine the same category historical classification model corresponding to the target classification model from the trained historical classification models according to the target classification task;
[0142] According to the above target classification model and / or the above historical classification model of the same category, obtain model test sample data;
[0143] Performing a data classification test on the model test sample data according to the target classification model to obtain a first test result;
[0144] Performing data classification test on the above model test sample according to the above historical classification model of the same category to obtain a second test result;
[0145] According to the first test result and the second test result, a model with the most accurate data classification is determined from the target classification model and the historical classification models of the same category to replace the target classification model.
[0146] In some optional embodiments, the data processing module 705 is specifically used for:
[0147] According to the target classification task and / or the target classification model, obtaining data control rules corresponding to the target classification task;
[0148] Deploy the target classification model, and perform a second data classification on the platform data according to the deployed target classification model to obtain a second data classification result;
[0149] The platform data is processed according to the data control rules and the second data classification result.
[0150] In this way, when processing data, there is no need to perform offline analysis. Instead, it can be determined whether to perform model training based on the online first data classification, and the second data classification can be performed according to the target classification model obtained through the online training to process the platform data according to the second data classification results, thereby realizing real-time automatic data processing, which is conducive to improving data processing efficiency.
[0151] It should be noted that the division of each module in the above device can be determined according to actual needs and is not specifically limited here.
[0152] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0153] Accordingly, the embodiment of the present application also provides an electronic device, which may be a terminal, and the terminal may be a smart phone, a tablet computer, a laptop computer, a touch screen, a game console, a personal computer (PC), a personal digital assistant (PDA), or other terminal devices. Figure 8 As shown, Figure 8 800 includes a processor 801 having one or more processing cores, a memory 802 having one or more computer-readable storage media, and a computer program stored in the memory 802 and executable on the processor. The processor 801 is electrically connected to the memory 802. Those skilled in the art will appreciate that the electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more than Figure 8 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.
[0154] The processor 801 is the control center of the electronic device 800, and uses various interfaces and lines to connect various parts of the entire electronic device 800. By running or loading software programs and / or modules stored in the memory 802, and calling data stored in the memory 802, the processor 801 executes various functions of the electronic device 800 and processes data, thereby monitoring the electronic device 800 as a whole. The processor 801 can be a central processing unit CPU, a graphics processing unit GPU, a network processor (NP, Network Processor), etc., and can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application.
[0155] In the embodiment of the present application, the processor 801 in the electronic device 800 will load instructions corresponding to the processes of one or more application programs into the memory 802 according to the following steps, and the processor 801 will run the application programs stored in the memory 802 to implement various functions, such as:
[0156] Determine a target data feature corresponding to a target classification task from data features of multiple data dimensions of platform data;
[0157] Performing a first data classification on the platform data according to the target data characteristics to obtain a first data classification result;
[0158] If the first data classification result meets the preset model training trigger condition, sample data corresponding to the target classification task is obtained, wherein a training sample in the sample data includes a training feature and a classification result label, and the training feature is determined according to a data feature of at least one data dimension of the platform data;
[0159] The target classification model corresponding to the target classification task is obtained by training the sample data;
[0160] Perform a second data classification on the platform data according to the target classification model to obtain a second data classification result, and process the platform data according to the second data classification result.
[0161] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0162] Optional, such as Figure 8 As shown, the electronic device 800 further includes: a touch screen 803, a radio frequency circuit 804, an audio circuit 805, an input unit 806, and a power supply 807. The processor 801 is electrically connected to the touch screen 803, the radio frequency circuit 804, the audio circuit 805, the input unit 806, and the power supply 807, respectively. Those skilled in the art can understand that Figure 8 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include Figure 8 More or fewer components may be shown, or certain components may be combined, or the components may be arranged differently.
[0163] The touch display screen 803 can be used to display a graphical user interface and receive operation instructions generated by the user acting on the graphical user interface. The touch display screen 803 may include a display panel and a touch panel. Among them, the display panel may be used to display information input by the user or information provided to the user and various graphical user interfaces of the electronic device, and these graphical user interfaces may be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel may be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light-emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel may be used to collect the user's touch operation on or near it (such as the user using any suitable object or attachment such as a finger, a stylus, etc. on the touch panel or near the touch panel), and generate corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts, a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch orientation, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 801, and can receive the command sent by the processor 801 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 801 to determine the type of touch event, and then the processor 801 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 803 to realize the input and output functions. However, in some embodiments, the touch panel and the display panel can be used as two independent components to realize the input and output functions. That is, the touch display screen 803 can also be used as a part of the input unit 806 to realize the input function.
[0164] The radio frequency circuit 804 may be used to send and receive radio frequency signals, so as to establish wireless communication with a network device or other electronic devices through wireless communication, and to send and receive signals with the network device or other electronic devices.
[0165] The audio circuit 805 can be used to provide an audio interface between the user and the electronic device through a speaker and a microphone. The audio circuit 805 can transmit the electrical signal converted from the received audio data to the speaker, which is converted into a sound signal for output; on the other hand, the microphone converts the collected sound signal into an electrical signal, which is received by the audio circuit 805 and converted into audio data, and then the audio data is output to the processor 801 for processing, and then sent to another electronic device through the radio frequency circuit 804, or the audio data is output to the memory 802 for further processing. The audio circuit 805 may also include an earplug jack to provide communication between an external headset and an electronic device.
[0166] The input unit 806 may be used to receive input numbers, character information or user feature information (such as fingerprint, iris, facial information, etc.), and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.
[0167] The power supply 807 is used to supply power to various components of the electronic device 800. Optionally, the power supply 807 can be logically connected to the processor 801 through a power management system, so that the power management system can manage charging, discharging, and power consumption. The power supply 807 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.
[0168] although Figure 8 Not shown, the electronic device 800 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.
[0169] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0170] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.
[0171] To this end, an embodiment of the present application provides a computer-readable storage medium, in which multiple computer programs are stored, and the computer program can be loaded by a processor to execute the steps in any data processing method provided in the embodiment of the present application. For example, the computer program can execute the following steps:
[0172] Determine a target data feature corresponding to a target classification task from data features of multiple data dimensions of platform data;
[0173] Performing a first data classification on the platform data according to the target data characteristics to obtain a first data classification result;
[0174] If the first data classification result meets the preset model training trigger condition, sample data corresponding to the target classification task is obtained, wherein a training sample in the sample data includes a training feature and a classification result label, and the training feature is determined according to a data feature of at least one data dimension of the platform data;
[0175] The target classification model corresponding to the target classification task is obtained by training the sample data;
[0176] Perform a second data classification on the platform data according to the target classification model to obtain a second data classification result, and process the platform data according to the second data classification result.
[0177] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.
[0178] The storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.
[0179] Since the computer program stored in the storage medium can execute the steps in any data processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any data processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.
[0180] According to one aspect of the present application, a computer program product or a computer program is also provided, the computer program product or the computer program includes a computer instruction, and the computer instruction is stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the method provided in various optional implementations in the above embodiments.
[0181] The data processing method, device, electronic device and computer-readable storage medium provided in the embodiments of the present application are introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.
Claims
1. A data processing method, characterized in that: include: Determine a target data feature corresponding to a target classification task from data features of multiple data dimensions of platform data; Performing a first data classification on the platform data according to the target data feature to obtain a first data classification result; If the first data classification result meets the preset model training trigger condition, sample data corresponding to the target classification task is obtained, wherein a training sample in the sample data includes a training feature and a classification result label, and the training feature is determined according to a data feature of at least one data dimension of the platform data; Obtaining a target classification model corresponding to the target classification task through training of the sample data; Perform a second data classification on the platform data according to the target classification model to obtain a second data classification result, and process the platform data according to the second data classification result.
2. The data processing method according to claim 1, characterized in that: The obtaining of sample data corresponding to the target classification task includes: Generate the sample data according to the first data classification result and the platform data corresponding to the first data classification result; and / or The sample data corresponding to the target classification task is obtained based on the offline data warehouse.
3. The data processing method according to claim 2, characterized in that: The generating the sample data according to the first data classification result and the platform data corresponding to the first data classification result includes: Performing feature encoding conversion on data features of multiple data dimensions of the platform data to obtain feature tensors corresponding to the platform data; The sample data is generated according to the feature tensor corresponding to the platform data and the first data classification result.
4. The data processing method according to claim 1, characterized in that: The training features of the training samples include data features of the multiple data dimensions except the target data features.
5. The data processing method according to claim 1, characterized in that: The step of obtaining a target classification model corresponding to the target classification task by training the sample data includes: Obtaining a classification model to be trained and model hyperparameters corresponding to the target classification task; The classification model is trained according to the sample data and the model hyperparameters to obtain the target classification model.
6. The data processing method according to claim 1, characterized in that: Before performing a second data classification on the platform data according to the target classification model to obtain a second data classification result, the method further includes: Determine, according to the target classification task, from the trained historical classification models, a historical classification model of the same category corresponding to the target classification model; Acquire model test sample data according to the target classification model and / or the historical classification model of the same category; Performing a data classification test on the model test sample data according to the target classification model to obtain a first test result; Performing a data classification test on the model test sample according to the historical classification model of the same category to obtain a second test result; According to the first test result and the second test result, a model with the most accurate data classification is determined from the target classification model and the historical classification models of the same category to replace the target classification model.
7. The data processing method according to any one of claims 1 to 6, characterized in that: The performing a second data classification on the platform data according to the target classification model to obtain a second data classification result, and processing the platform data according to the second data classification result, includes: According to the target classification task and / or the target classification model, obtaining a data control rule corresponding to the target classification task; Deploy the target classification model, and perform a second data classification on the platform data according to the deployed target classification model to obtain a second data classification result; The platform data is processed according to the data control rule and the second data classification result.
8. A data processing device, characterized in that: include: A target data feature determination module is used to determine the target data feature corresponding to the target classification task from the data features of multiple data dimensions of the platform data; A first data classification module, used to perform a first data classification on the platform data according to the target data feature to obtain a first data classification result; A sample acquisition module, configured to acquire sample data corresponding to the target classification task if the first data classification result meets a preset model training trigger condition, wherein a training sample in the sample data includes a training feature and a classification result label, and the training feature is determined according to a data feature of at least one data dimension of the platform data; A model training module, used for training and obtaining a target classification model corresponding to the target classification task according to the sample data; A data processing module is used to perform a second data classification on the platform data according to the target classification model, obtain a second data classification result, and process the platform data according to the second data classification result.
9. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the data processing method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the data processing method according to any one of claims 1 to 7.