Data processing method and device, equipment and computer readable storage medium

Abstract

Embodiments of the present application disclose a data processing method, device and equipment and a computer readable storage medium. The method comprises: obtaining positive samples of an anomaly recognition model, wherein the positive samples refer to original samples identified as normal by the anomaly recognition model; performing anomaly screening processing on the positive samples to obtain anomaly samples; filtering feature information of the anomaly samples based on tree structure information associated with the anomaly recognition model to obtain filtered initial samples; and iteratively training the anomaly recognition model based on the initial samples to obtain a trained anomaly recognition model. Through the method, the efficiency of model updating can be improved, thereby improving the accuracy of the model in identifying abnormal data and improving the accuracy of data processing.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically to a data processing method, apparatus, device, and computer-readable storage medium. Background Technology

[0002] Various multimedia content exists across different multimedia platforms, including images, text, audio, video, and live streaming. However, some multimedia platforms may contain unhealthy or abnormal content that negatively impacts user experience.

[0003] In related technologies, deep learning models are typically used to identify whether multimedia content is abnormal. However, abnormal multimedia content will continue to generate different anomalies, and the model will be unable to identify these new anomalies, resulting in inaccurate identification results of the deep learning model for abnormal data.

[0004] Therefore, improving the accuracy of models in identifying abnormal data is an urgent problem to be solved. Summary of the Invention

[0005] To address the aforementioned technical problems, embodiments of this application provide a data processing method, apparatus, device, and computer-readable storage medium.

[0006] The technical solution adopted in this application is as follows:

[0007] A data processing method, comprising:

[0008] Obtain positive samples from the anomaly detection model, whereby positive samples refer to original samples that are identified as normal by the anomaly detection model.

[0009] The positive samples are subjected to anomaly screening to obtain abnormal samples;

[0010] The feature information of the abnormal samples is filtered by the tree structure information associated with the anomaly recognition model to obtain the filtered initial samples.

[0011] The anomaly detection model is iteratively trained based on the initial samples to obtain the trained anomaly detection model.

[0012] A data processing device, comprising:

[0013] The acquisition unit is used to acquire positive samples of the anomaly recognition model, wherein the positive samples refer to the original samples that are identified as normal by the anomaly recognition model;

[0014] The processing unit is used to perform anomaly screening on the positive samples to obtain abnormal samples;

[0015] The processing unit is further configured to filter the feature information of the abnormal sample through the tree structure information associated with the anomaly recognition model to obtain the filtered initial sample.

[0016] The training unit is used to iteratively train the anomaly recognition model based on the initial samples to obtain the trained anomaly recognition model.

[0017] In one embodiment of this application, based on the foregoing scheme, the acquisition unit is further configured to acquire the tree structure information associated with the anomaly recognition model; the processing unit is further configured to calculate the feature importance of each feature dimension in the anomaly sample based on the tree structure information; and the filtering unit is further configured to filter the feature dimensions in the feature information of the anomaly sample whose feature importance is lower than the preset importance threshold to obtain the filtered initial sample.

[0018] In one embodiment of this application, based on the foregoing scheme, the processing unit is further configured to add the feature dimension to the tree structure information; calculate the gain value corresponding to at least two branch nodes of the feature dimension in the tree structure information; the acquisition unit is further configured to acquire the gain value corresponding to each other branch node in the tree structure information; the processing unit is further configured to calculate the feature importance corresponding to the feature dimension based on the gain value corresponding to each other branch node in the tree structure information and the gain value corresponding to at least two branch nodes of the feature dimension.

[0019] In one embodiment of this application, based on the foregoing scheme, the processing unit is further configured to, if the number of tree structure information is at least two, take the average feature importance of the feature dimension in the at least two tree structure information as the feature importance corresponding to the feature dimension.

[0020] In one embodiment of this application, based on the aforementioned scheme, the number of abnormal samples is at least two; the processing unit is further configured to divide the at least two abnormal samples into at least two sample subsets, each sample subset corresponding to a subset number; the acquisition unit is further configured to acquire a number of trainable models equal to the number of the at least two sample subsets, each trainable model corresponding to a model number, and each trainable model being identical to the anomaly recognition model; the processing unit is further configured to acquire, for each trainable model, other sample subsets besides those whose subset numbers match the model number of the trainable model; and the training unit is further configured to train the trainable model based on the acquired other sample subsets to obtain at least two trained models; and select the trained anomaly recognition model from the at least two trained models.

[0021] In one embodiment of this application, based on the foregoing scheme, the processing unit is further configured to calculate the prediction error for each trained model; and to determine the trained model with the smallest prediction error among at least two trained models as the trained anomaly recognition model.

[0022] In one embodiment of this application, based on the foregoing scheme, the steps are as follows: obtaining a target sample subset whose subset index matches the model index of the trained model; inputting the target sample subset into the trained model for testing; and calculating the prediction error of the trained model when testing the target sample subset.

[0023] In one embodiment of this application, based on the foregoing scheme, the processing unit is further configured to calculate the abnormal suspicion degree corresponding to the positive sample based on a preset suspicious analysis rule; filter out undetermined samples from the positive samples whose abnormal suspicion degree is greater than a preset suspicion degree threshold; obtain the review result of the undetermined samples; and determine the undetermined samples whose review result is characterized as abnormal as the abnormal samples.

[0024] A data processing device includes a processor and a memory, wherein the memory stores computer-readable instructions, and the computer-readable instructions are executed by the processor to implement the data processing method described above.

[0025] A computer-readable storage medium having stored computer-readable instructions thereon, which, when executed by a computer's processor, cause the computer to perform the data processing method described above.

[0026] A computer program product includes computer-readable instructions that, when executed by a processor, implement the data processing method described above.

[0027] In the above technical solutions:

[0028] On the one hand, even among the original samples identified as normal by the anomaly detection model, there may still be anomalous samples. Filtering out these anomalous samples from the positive samples can reflect the newly added anomaly types. Thus, iteratively training the anomaly detection model based on these newly added anomalous samples allows the model to promptly acquire the ability to identify these newly added anomalous samples that were not initially identified as anomalous. Compared to manually collecting anomalous samples, this method has a high degree of automation. By improving the efficiency of acquiring anomalous samples and enabling the anomaly detection model to iterate and upgrade more efficiently, the update efficiency of the anomaly detection model is improved, thereby enhancing the accuracy of the model in identifying anomalous data and also improving the accuracy of data processing.

[0029] On the other hand, filtering out features with low importance from the feature information reduces their interference with model training, allowing us to accurately identify the feature dimensions that have a greater impact on anomaly identification results. This not only reduces the complexity of training the model but also improves the generalization ability of the anomaly identification model, covering a wider range of anomaly identification scenarios and increasing the model's accuracy in identifying anomalous data.

[0030] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0031] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:

[0032] Figure 1 This is a schematic diagram of a data processing system architecture involved in this application;

[0033] Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment;

[0034] Figure 3 This is a flowchart illustrating a data processing method according to another exemplary embodiment;

[0035] Figure 4 This is a schematic diagram of a tree structure information involved in this application;

[0036] Figure 5 This is a flowchart illustrating a data processing method according to another exemplary embodiment;

[0037] Figure 6 This is a flowchart illustrating a data processing method according to another exemplary embodiment;

[0038] Figure 7 This is a schematic diagram of a data processing procedure involved in this application;

[0039] Figure 8 This is a block diagram illustrating a data processing apparatus according to an exemplary embodiment;

[0040] Figure 9 This is a schematic diagram of the structure of a computer system for a data processing device according to an exemplary embodiment. Detailed Implementation

[0041] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments identical to those described in this application. Rather, they are merely examples of apparatuses and methods identical to some aspects of this application as detailed in the appended claims.

[0042] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in at least one hardware module or integrated circuit, or in different network and / or processor devices and / or microcontroller devices.

[0043] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily need to be performed in the described order. For example, some operations / steps can be broken down, while others can be integrated or partially integrated; therefore, the actual execution order may change depending on the specific circumstances.

[0044] It should be noted that "multiple" as mentioned in this application refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0045] Before introducing the technical solutions of the embodiments of this application, let's first introduce the technical terms involved in the embodiments of this application.

[0046] Extreme Gradient Boosting (XGBoost) is a highly efficient gradient boosting tree algorithm that excels in solving regression, classification, and ranking problems. XGBoost improves the overall model's accuracy by integrating multiple decision trees and progressively correcting the prediction errors of previous trees. It employs the gradient boosting framework, continuously optimizing the model's predictive power by minimizing the loss function. XGBoost's advantages include its high computational efficiency, automatic handling of missing values, support for regularization to reduce overfitting, and the ability to accelerate the training process through parallel computation. XGBoost combines gradient boosting and regularization techniques, iteratively training a series of decision tree models to improve prediction accuracy and generalization ability.

[0047] XGBoost's tree structure, or decision tree, is built by progressively splitting the feature space. Each tree is generated by conditionally splitting the sample features, with the goal of maximizing the prediction accuracy of the sample at each node. During training, XGBoost uses gain to measure the effectiveness of each feature split; gain typically refers to the reduction in the value of the objective function (such as the loss function) before and after the split. Each decision tree consists of multiple nodes, which make conditional judgments based on the sample's feature values, dividing the sample set into different subsets with each split. By combining multiple decision trees, XGBoost can effectively reduce model bias and improve its generalization ability on complex datasets. The structural information of the decision tree, including the tree depth, the splitting conditions of each node, and the gain value, constitutes the core of the XGBoost model, directly affecting its performance and accuracy.

[0048] Feature information, also known as feature vectors, is a vector used to represent the characteristics of data. A feature vector is a multi-dimensional vector, where each dimension represents a feature or attribute of the data. These features can be numerical, categorical, or other forms of data. In machine learning and pattern recognition tasks, feature vectors are used to describe the key features of the data and are provided as input to algorithms for model training and prediction. By selecting and extracting appropriate features and representing them in vector form, machine learning algorithms can better understand and process data.

[0049] Automated Machine Learning (AutoML) refers to simplifying traditional machine learning tasks through automated processes, enabling non-experts and professional data scientists to build, train, and optimize machine learning models more efficiently. It makes machine learning easier to learn and apply by reducing complex tasks such as manual parameter tuning, feature engineering, and model selection.

[0050] In related technologies, deep learning models are typically used to identify whether multimedia content is abnormal. However, abnormal multimedia content will continue to generate new anomalies. If the model is not retrained, it will be unable to identify these new anomalies, resulting in inaccurate identification results of the deep learning model for abnormal data.

[0051] Based on this, embodiments of this application propose a system installation method, a system installation device, a system installation equipment, a computer-readable storage medium, and a computer program product. In these embodiments, on the one hand, even among the original samples identified as normal by the anomaly detection model, there may still be anomalous samples. Filtering out anomalous samples from these positive samples allows the model to reflect newly added anomaly types. Thus, iteratively training the anomaly detection model based on these newly added anomalous samples enables the model to promptly acquire the ability to identify these newly added anomalous samples that were not previously identified as anomalous. Compared to manually collecting anomalous samples, this method has a high degree of automation. By improving the efficiency of acquiring anomalous samples and enabling the anomaly detection model to iterate and upgrade more efficiently, the update efficiency of the anomaly detection model is improved, thereby improving the model's accuracy in identifying anomalous data and also improving the accuracy of data processing. On the other hand, filtering out feature dimensions with low feature importance reduces their interference with model training, accurately grasping the feature dimensions that have a significant impact on the anomaly detection results. This not only reduces the complexity of training the model but also improves the generalization ability of the anomaly detection model, covering a wider range of anomaly detection scenarios and improving the model's accuracy in identifying anomalous data.

[0052] Please see Figure 1 , Figure 1 This is a schematic diagram of a data processing system architecture involved in this application.

[0053] Figure 1 The data processing system architecture shown includes an application layer 110, a middleware layer 120, and a data layer 160.

[0054] The application layer 110 directly interacts with users or other external systems, and its core task is to meet various business needs of users. It encapsulates the underlying technical details and presents various functions in a user-friendly manner. The application layer 110 includes a data portal 111, data analysis 112, and a processing platform 113.

[0055] Data Portal 111 is a centralized data display and access platform that integrates data from multiple data sources, providing users with a unified data access portal. It can also display relevant data based on user permissions and has a data navigation function to help users quickly locate the data they need.

[0056] Data Analytics 112 is a module for Business Intelligence (BI) analysis. It uses data mining, visualization, statistical analysis and other methods to deeply analyze data, and creates reports and dashboards to intuitively display the analysis results, providing data support for enterprise decision-making and helping users to understand the business value and trends behind the data.

[0057] The 113 handling platform is a dedicated platform for handling abnormal situations. It mainly identifies abnormal data, abnormal user behavior, or problematic authors on the platform and takes corresponding handling measures according to pre-set procedures. At the same time, the entire handling process is recorded to ensure the normal operation of the platform and the quality and security of the data.

[0058] The middle platform layer 120 connects the application layer and the data layer, primarily responsible for core functions such as data processing, feature mining, and model updates. It provides data support and intelligent services to the application layer 110, while effectively managing and accessing various types of data from the data layer. The functional modules of the middle platform layer 120 include an automatic feature mining module 130, a model update module 140, a data engine 151, an intelligent annotation platform 152, a machine learning platform 153, and a feature platform 154.

[0059] The automatic feature mining module 130 mainly comprises three sub-modules: data acquisition 131, data processing 132, and feature mining 133. First, the data acquisition module 131 is responsible for collecting anomalous samples from various data sources (such as the various data storage areas in data layer 160). The data processing module 132 performs preprocessing operations such as cleaning, feature transformation, and standardization on the acquired data to remove noisy data and redundant information, ensuring that the data quality meets the requirements of model training. The feature mining module 133 automatically extracts important feature dimensions from the feature information of the processed anomalous samples and deletes unimportant feature dimensions, providing high-quality feature information for subsequent model training and analysis.

[0060] The model update module 140 is primarily used to continuously optimize and update the anomaly detection model to improve its adaptability to new anomaly samples and its predictive accuracy. The model update module 140 comprises the following sub-modules:

[0061] The model validation module 141 is mainly responsible for validating the performance of the trained model and evaluating its performance in practical applications, such as accuracy and recall.

[0062] The model deployment module 142 can adjust the parameters of the anomaly recognition model based on the trained model, deploy it to the server, and replace the old anomaly recognition model.

[0063] Data Engine 151 offers efficient data storage, retrieval, and computing capabilities, supporting rapid access and analysis of big data.

[0064] The intelligent annotation platform 152 is used for intelligent annotation of various samples, providing accurate labels for anomalous samples. By combining manual and automatic annotation methods, the intelligent annotation platform can accelerate the annotation process and ensure annotation quality, significantly improving annotation efficiency.

[0065] The machine learning platform 153 can support machine learning model training and evaluation for the middleware layer 120. This platform integrates various machine learning algorithms and tools to support model training, validation, and optimization.

[0066] The feature platform 154 provides feature management functions, which are responsible for performing feature transformation and feature importance calculation on the acquired samples.

[0067] Data layer 160 can store various types of sample data, including data storage areas 171, 172, and 173. These data storage areas can be flexibly selected according to different application scenarios, such as MySQL databases, highly fault-tolerant distributed file systems (Hadoop Distributed File System, HDFS), and search servers (Elasticsearch).

[0068] It should be noted that the anomaly identification model of this application can be deployed on a server. The server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0069] In the specific implementation of this application, user-related data is involved, such as the pinyin sequence, target text, and input process data entered by the user. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

[0070] Please see Figure 2 , Figure 2 This is a flowchart illustrating a data processing method according to an exemplary embodiment. The method can be applied to... Figure 2 The implementation environment is shown, and the method is executed by a computer. Of course, this method can also be applied to other implementation environments, and there are no restrictions on the subject executing this method.

[0071] The following section will use a computer as an example to illustrate this data processing method in detail. Figure 2 As shown, in an exemplary embodiment, the method includes at least the following steps:

[0072] S210. Obtain positive samples from the anomaly detection model. These positive samples refer to the original samples that are identified as normal by the anomaly detection model.

[0073] This anomaly detection model is deployed on a server and is in operation. It can perform anomaly detection on a large number of samples to be identified in the current network environment, thereby being able to identify normal original samples (which can be called positive samples) and abnormal original samples.

[0074] In this application, these original samples can be various types of multimedia content, such as audio, video, text, emoticons, live streaming audio and video, etc. Those skilled in the art can flexibly use various types of multimedia content as original samples, and the original sample can be at least one type of these multimedia content.

[0075] It should be noted that the anomaly detection model is a model deployed on a server to identify whether multimedia content contains anomalies. In this application, the computer may also be deployed on a server, and the anomaly detection model can be redeployed; therefore, the anomaly detection model can be dynamically updated and deployed. The anomaly detection model can be updated and deployed when necessary.

[0076] S220. Perform anomaly screening on positive samples to obtain abnormal samples.

[0077] In this positive sample, there are normal original samples that are accurately identified by the anomaly recognition model. Alternatively, if some new anomalies are added, and the anomaly recognition model has not encountered these anomalies before, it may mistakenly identify some abnormal samples as normal and treat them as positive samples.

[0078] Computers can combine user behavior or other relevant information to perform anomaly analysis on these positive samples using preset suspiciousness analysis rules, filtering out undetermined samples from the original samples whose anomaly suspicion level exceeds a preset suspicion threshold. These preset suspiciousness analysis rules can be flexibly set by those skilled in the art according to different application scenarios and can be modified to adapt to different anomaly identification scenarios. This makes the collection of anomaly samples more targeted.

[0079] In the process of conducting suspicion analysis, specific algorithms or machine learning models can be used for judgment. Furthermore, the preset suspicion threshold can be a threshold set by someone skilled in the art, and is not limited here. The computer can label each original sample with an anomalous suspicion level, thereby determining the anomalous suspicion level of each original sample. Next, the computer needs to send the pending samples to the review panel.

[0080] In addition, these pending samples can be stored in a specific data warehouse, from which computers can periodically retrieve these new pending data, or these new samples with questionable nature.

[0081] The review console can display all the suspicious pending samples sent by the computer to a human reviewer, who will then review each sample. The human reviewer may determine one pending sample to be abnormal or another to be normal. The review console can then output the corresponding review results for each pending sample sent by the computer and return them to the computer.

[0082] The computer can obtain the review results of pending samples, and each review result can show whether its corresponding pending sample is an anomalous sample. In this way, the computer can identify anomalous samples from positive samples. The number of anomalous samples is at least two, which can form an anomalous sample set.

[0083] In this embodiment, the abnormal samples in the abnormal sample set are those that the current anomaly identification model cannot recognize, but are themselves a type of abnormal sample. This type represents a newly added anomaly risk in the current network platform, which the current anomaly identification model cannot accurately identify, thus treating them as normal samples. However, through the combination of computer-based suspiciousness analysis and manual review at the review desk, as described in this application, these newly added abnormal samples can be identified.

[0084] After obtaining the anomaly sample set, the computer can store it in an audit result data table. The computer can then use a feature engineering platform to periodically retrieve anomaly samples from this table. Next, the computer can use automated machine learning to preprocess and partition the anomaly sample set. During data preprocessing and partitioning, the computer can combine specific algorithms, such as Scikit-learn and TensorFlow. For dataset partitioning, specific functions can be used, such as the `train_test_split` function from the Scikit-learn library.

[0085] Next, the computer can perform feature transformation on each outlier sample to obtain the feature information corresponding to each outlier sample, and then standardize the feature information. Specifically, the computer can use a standard scaler to standardize the Z-score, as shown in the formula:

[0086] z = (xu) / s

[0087] Where x is the eigenvalue, u is the mean of the eigenvalue, and s is the standard deviation of the eigenvalue.

[0088] Standardization is to standardize the feature values ​​of each feature dimension in the feature information of each abnormal sample to the range of 0 to 1, so as to facilitate subsequent processing.

[0089] For ease of explanation, the feature information of abnormal samples mentioned later refers to the feature information after standardization.

[0090] S230. The feature information of the abnormal samples is filtered by the tree structure information associated with the anomaly recognition model to obtain the filtered initial samples.

[0091] Each outlier sample's feature information includes feature values ​​from at least two feature dimensions. For example, audio data's feature information includes volume, voiceprint, pitch, timbre, and frequency features, among others. Different feature dimensions have varying degrees of influence on whether a sample is identified as an outlier. Feature dimensions with greater influence can be considered as having higher feature importance, while those with less influence can be considered as having lower feature importance. Thus, each feature dimension can have its own corresponding feature importance score.

[0092] In this application, the computer can obtain the tree structure information associated with the anomaly detection model currently deployed on the server. By adding each feature dimension of the anomaly sample to this tree structure information, the feature importance of each anomaly sample is calculated. Then, feature dimensions with feature importance lower than a preset importance threshold can be filtered out. Filtering out these feature dimensions means deleting these feature dimensions and their feature values. Thus, the filtered feature information of the anomaly sample has fewer feature dimensions than the unfiltered feature information, thereby reducing the size of the feature information. For ease of description, this application refers to the anomaly sample that has undergone feature dimension filtering as the filtered initial sample. It can be understood that the feature information in the initial sample has fewer dimensions than the feature information in the anomaly sample, and the feature information in the initial sample does not have redundant feature dimensions, making it more concise.

[0093] By filtering feature dimensions with lower importance, the computer selects the feature dimensions that have the greatest impact on the model's prediction from a large number of features, reducing the interference of irrelevant or redundant feature dimensions on the subsequent model training process. This significantly improves the efficiency and effectiveness of model training, thereby enhancing the model's generalization ability.

[0094] In practical applications, anomalous sample sets typically contain multiple feature dimensions, and different feature dimensions contribute differently to the model's prediction results. For example, in audio data, the influence of feature dimensions such as volume, voiceprint, and pitch varies. Some feature dimensions may play a significant role in determining whether a sample is anomalous, while others may have a smaller or even irrelevant impact on the final result. By calculating the feature importance of each feature dimension, we can identify which feature dimensions are most valuable for anomaly detection. Filtering out the feature values ​​of these feature dimensions can significantly reduce the model's input space, eliminating information that is ineffective or highly interfering with model training, thus allowing the model to focus more on the key feature dimensions.

[0095] S240. Iteratively train the anomaly recognition model based on the initial samples to obtain the trained anomaly recognition model.

[0096] By iteratively training the anomaly detection model based on filtered feature information from the initial samples, the computer can obtain a trained anomaly detection model. After training, the model parameters will change to reflect the parameters that enable it to recognize the initial samples. Thus, the trained anomaly detection model can then identify anomalies in the current samples.

[0097] Optionally, in this application, the anomaly detection model includes multiple weights and parameters, with each weight corresponding to a specific parameter. Those skilled in the art can write all weights into the anomaly detection model and determine the weight of each weight based on the anomalies in the current network environment. For weights corresponding to anomalies that have never occurred before, their weights can be set to 0. When such anomalies subsequently occur, these weights can be activated by modifying their parameters, thereby enabling the anomaly detection model to identify new anomalies.

[0098] Specifically, the computer iteratively trains the anomaly recognition model based on initial samples to obtain the trained anomaly recognition model. After training, the model can make more accurate predictions for specific anomaly samples (i.e., anomaly samples that the current anomaly recognition model fails to identify as anomalies).

[0099] It should be noted that the iterative training in this application refers to retraining a previously trained anomaly detection model, rather than training with a completely new initial model. In this way, each time an unidentifiable anomalous sample appears, the computer can further train the anomaly detection model, thereby enhancing its ability to identify various anomaly samples in the current network environment.

[0100] After obtaining the trained anomaly detection model, the computer can take down the anomaly detection model currently deployed on the server and redeploy the trained anomaly detection model, allowing the trained anomaly detection model to immediately enter the running state to identify various samples to be identified in the current network environment.

[0101] This method addresses the issue that, even among the original samples identified as normal by the anomaly detection model, there may still be anomalous samples. By filtering out these anomalous samples from the positive samples, the model can identify newly added anomalies. Iterative training of the anomaly detection model based on these newly added anomalous samples allows it to promptly acquire the ability to recognize these newly added anomalous samples that were not initially identified as anomalous. Compared to manually collecting anomaly samples, this method is highly automated. By improving the efficiency of anomaly sample acquisition and enabling more efficient iterative upgrades, the update efficiency of the anomaly detection model is improved, thereby enhancing the accuracy of the model in identifying anomalous data and improving the accuracy of data processing.

[0102] On the other hand, filtering out features with low importance from the feature information reduces their interference with model training, allowing us to accurately identify the feature dimensions that have a greater impact on anomaly identification results. This not only reduces the complexity of training the model but also improves the generalization ability of the anomaly identification model, covering a wider range of anomaly identification scenarios and increasing the model's accuracy in identifying anomalous data.

[0103] In one embodiment of this application, another data processing method is provided, which can be executed by a computer. For example... Figure 3 As shown, the data processing method may include S210 to S220, S310 to S330, and S240. That is, S310 to S330 are... Figure 2 The specific implementation method of S230 is shown.

[0104] The following describes S310 to S330:

[0105] S310. Obtain the tree structure information associated with the anomaly detection model.

[0106] The calculation of feature importance can employ XGBoost gradient boosting decision trees, which can serve as the tree structure information. In this application, the computer can obtain the tree structure information associated with the anomaly detection model currently deployed on the server.

[0107] Computers can obtain tree structure information related to anomaly detection through deployed anomaly detection models. This tree structure information is derived from the decision tree built by the existing anomaly detection model. All feature dimensions that the anomaly detection model can identify can be used to construct this tree structure information, thus reflecting the feature dimensions that the current anomaly detection model can cover.

[0108] The number of tree structure information can be at least two, and the feature dimensions of each tree structure information may be different.

[0109] S320. Calculate the feature importance of each feature dimension in the abnormal sample based on tree structure information.

[0110] The current anomaly detection model may not cover all feature dimensions, because some other types of feature dimensions may have been added. Therefore, the computer can add the feature dimensions corresponding to the current anomaly sample to the tree structure information, thereby calculating the feature importance of different feature dimensions.

[0111] Specifically, S320 may include S321 to S324. S321 to S324 are described below:

[0112] S321. Add the feature dimensions to the tree structure information.

[0113] The feature dimension can be any one of at least two feature dimensions from the feature information of the abnormal sample. Adding this feature dimension to the tree structure information allows the computer to generate at least two branch nodes for that feature dimension within the tree structure information. Specifically, adding the feature dimension to the tree structure information means further splitting the feature dimension at the leaf nodes of the tree structure information to obtain at least two branch nodes for that feature dimension.

[0114] S322. Calculate the gain value corresponding to at least two branch nodes of the feature dimension in the tree structure information.

[0115] Specifically, in this application, XGBoost can use LogLoss as the loss function, and each of the at least two branch nodes can calculate a gain value. Considering that decision trees are usually constructed in the form of binary trees, a feature information can typically have two branch nodes. Of course, depending on the type of decision tree, there can also be at least two branch nodes. For ease of explanation, this application's embodiment uses a feature dimension in the tree structure information corresponding to two branch nodes as an example. However, considering that branch nodes of different feature dimensions can be superimposed, the first feature dimension corresponds to 2 branch nodes, and each of these 2 branch nodes can further branch, resulting in 4 branch nodes, and these 4 branch nodes all belong to the second feature dimension. And so on, it can be seen that the nth feature dimension can correspond to 2... n Branch nodes.

[0116] For example, such as Figure 4 The diagram shows a tree structure provided in this application. Feature dimension 1 corresponds to branch node 1 (i.e., the left child node of feature dimension 1) and branch node 2 (i.e., the right child node of feature dimension 2). Feature dimension 2 further splits on the two branch nodes of feature dimension 1, resulting in branch nodes 3 to 6, and so on. Figure 4 The tree structure information contains J branch nodes, and these J branch nodes have already had the current feature dimension added to them.

[0117] Next, the computer adds the current feature dimension to the tree structure information, thus obtaining at least two branch nodes corresponding to that feature dimension. In this case, each branch node of that feature dimension can correspond to a gain value, calculated using the following formula:

[0118]

[0119] Among them, G i and H i G represents the first and second derivatives of the loss function. L H is the gradient of the left child node. L G is the sum of the second derivatives of the left child nodes. R H is the gradient of the right child node. R G is the sum of the second derivatives of the right child nodes; G is the gradient of the node before splitting; H is the sum of the second derivatives of the node before splitting. λ is the regularization parameter, controlling the model complexity, and γ is the minimum gain threshold for splitting.

[0120] S323. Obtain the gain value corresponding to each other branch node in the tree structure information.

[0121] In addition to at least two branch nodes having gain values ​​in the current feature dimension, all other branch nodes also have gain values. Therefore, the computer can obtain the gain value of each other branch node, which is also calculated using the formula described above.

[0122] S324. Based on the gain value corresponding to each other branch node in the tree structure information and the gain value corresponding to at least two branch nodes of the feature dimension, calculate the feature importance corresponding to the feature dimension.

[0123] Specifically, the computer can add the squared error of the gain value corresponding to each other branch node to the squared error of the gain values ​​corresponding to at least two branch nodes of the feature dimension, obtaining a sum of squared errors, which serves as the feature importance corresponding to that feature dimension. The specific calculation formula is as follows:

[0124]

[0125] Where j represents the index of the feature dimension, i.e., the feature importance corresponding to feature dimension j, and J is the current... It is the improvement of the squared error of branch node t, P(υ t =j) represents the sum of squared errors of at least two branch nodes corresponding to feature dimension j.

[0126] In one embodiment of this application, the number of tree structure information pieces can be at least two, and the feature dimensions of each tree structure information piece may be different. These dimensions reflect the model's ability to understand and judge features. If the number of tree structure information pieces is at least two, the average feature importance of the feature dimension in the at least two tree structure information pieces is taken as the feature importance corresponding to the feature dimension. The specific calculation formula is as follows:

[0127]

[0128] Where M is the amount of tree structure information, T m For the m-th tree structure information, It represents the feature importance corresponding to feature dimension j in the m-th tree structure.

[0129] After calculating the feature importance of the current feature dimension j, the computer can switch to another feature dimension until the feature importance of each feature dimension in the feature information of the abnormal sample has been calculated.

[0130] S330. Filter the feature dimensions whose feature importance is lower than the preset importance threshold to obtain the filtered feature information.

[0131] The preset importance threshold can be set by someone skilled in the art, and is not limited here. The computer can compare the feature importance of each feature dimension with the preset importance threshold, and filter the feature dimensions whose feature importance is lower than the preset importance threshold to obtain filtered feature information. The number of feature dimensions in the filtered feature information is less than the number of feature dimensions in the unfiltered feature information.

[0132] This method allows the computer to filter out features with low importance from the feature information, reducing their interference with model training and enabling it to accurately grasp the feature dimensions that have a greater impact on anomaly identification results. This not only reduces the complexity of training the model but also improves the generalization ability of the anomaly identification model, covering a wider range of anomaly identification scenarios and enhancing the model's accuracy in identifying anomalous data.

[0133] In one embodiment of this application, another data processing method is provided, which can be executed by a computer. For example... Figure 5 As shown, the data processing method may include S210 to S230 and S510 to S540. That is, S510 to S540 are... Figure 2 The specific implementation method of S240 is shown.

[0134] The following describes S510 to S530:

[0135] S510. Divide at least two abnormal samples into at least two sample subsets, and each sample subset has a corresponding subset number.

[0136] Among them, the at least two abnormal samples are all the abnormal samples in the current abnormal sample set.

[0137] In one embodiment of this application, when training an anomaly recognition model, the computer can use K-fold cross-validation to ensure that the model is trained sufficiently and effectively.

[0138] The computer can divide the abnormal sample set into at least two sample subsets, and the feature information of each sample subset is disjoint. Assume there are X abnormal samples in the abnormal sample set, and assume it is divided into k (k≥2) sample subsets, denoted as S1,…,S… k Thus, each sample subset contains X / k abnormal samples.

[0139] S520. Obtain the same number of training models as at least two sample subsets, each training model having a corresponding model number, and each training model being the same as the anomaly recognition model.

[0140] The computer can acquire the same number of trainable models as at least two sample subsets, i.e., acquire k trainable models, denoted as Q1,…,Q. k Furthermore, these models to be trained are the same as the anomaly detection models. For ease of distinction, the anomaly detection models can be referred to as the models to be trained during training.

[0141] S530. For each model to be trained, obtain other sample subsets besides the sample subset whose subset number matches the model number of the model to be trained, and train the model to be trained based on the obtained other sample subsets to obtain at least two trained models.

[0142] Specifically, the computer compares the i-th model Q to the training model. i To obtain S i All other sample subsets, that is, obtaining sample subsets S1 to S2. i-1 and S i+1 To S k Next, the sample subsets S1 to S2 will be... i-1 and S i+1 To S k Input into the model Q to be trained i Training is performed to obtain the trained Q. i .

[0143] By doing so, k trained models can be calculated.

[0144] S540. Select a trained anomaly detection model from at least two trained models.

[0145] In the at least two trained models, each model may have a different prediction error. This prediction error can be a generalization error, which refers to the error that exists when the trained model performs generalization recognition after it has acquired a certain level of generalization ability. The computer selects the anomaly recognition model based on the prediction error of each trained model.

[0146] Using this method, the computer can train each model based on initial samples, obtaining corresponding trained models. Finally, one trained model is selected as the final anomaly detection model. The selected model achieves the best detection performance, improving the accuracy of anomaly data identification.

[0147] In one embodiment of this application, another data processing method is provided, which can be executed by a computer. For example... Figure 6 As shown, the data processing method may include S210 to S230, S510 to S530, and S610 to S620. That is, S610 to S620 are...Figure 5 The specific implementation method of S540 is shown.

[0148] The following describes S610 to S620:

[0149] S610. Calculate the prediction error for each trained model.

[0150] Specifically, S610 may include S611 to S613, which are described below:

[0151] S611. Obtain the target sample subset whose subset index matches the model index of the trained model.

[0152] The computer needs to obtain a subset of target samples whose subset indices match the model indices of the trained model. In other words, the model Q to be trained... i The corresponding trained model obtains a sample subset S. i .

[0153] S612. Input a subset of the target samples into the trained model for testing.

[0154] The computer can select a subset of the target samples, namely S i As the model Q to be trained i The corresponding validation set of the trained model is input into it for testing.

[0155] S613. Calculate the prediction error of the trained model when testing the target sample subset.

[0156] The computer can first obtain the hypothesis function h corresponding to the trained model. i The prediction error is calculated. This prediction error can refer to the generalization error, or the error in the model Q to be trained. i The generalization error of the corresponding trained model can be expressed as: This allows us to calculate the prediction error for each trained model.

[0157] S620. The training model with the smallest prediction error among at least two trained models is determined as the trained anomaly recognition model.

[0158] Since each trained model has a corresponding prediction error, the computer can select the model with the smallest prediction error as the target model. A smaller prediction error indicates that the model not only has strong generalization ability but also higher accuracy in generalization identification. Therefore, selecting the model with the smallest prediction error will result in a more accurate identification of abnormal samples.

[0159] This method allows the computer to automatically iteratively train the training model (anomaly detection model) and select the model with the smallest prediction error from the trained models as the final anomaly detection model. This ensures that the trained model with the best detection performance is used as the final anomaly detection model. Subsequently, the computer can deploy this trained anomaly detection model to a server, replacing the original anomaly detection model. This allows for timely coverage of newly added anomalies in the current network environment, improving the accuracy of anomaly data identification.

[0160] In one embodiment of this application, such as Figure 7 The diagram shown is a schematic representation of a data processing flow provided in this application. Figure 7 In this process, the computer can identify suspicious samples (i.e., pending samples) from a large number of raw samples in the overall data based on preset suspicious analysis rules. These suspicious samples are then sent to a review panel where humans manually review each sample to determine if it is an anomaly. If it is an anomaly, it will be stored in an anomaly sample set. This ensures that at least two anomaly samples are stored in the anomaly sample set. Next, the anomaly sample set is fed into an automatic sample set partitioning module, a feature dimension filtering module, a model training module, and an automatic anomaly detection model deployment module. These modules have been automatically packaged into a single unit.

[0161] Automatic sample set partitioning involves dividing the sample set into training and testing sets. Feature dimension filtering involves filtering the feature information of each abnormal sample, removing feature dimensions with an importance lower than a preset importance threshold, resulting in filtered initial samples. Model training involves inputting the initial samples into the anomaly detection model for training. Automatic anomaly detection model deployment involves automatically replacing and redeploying the anomaly detection model based on the trained model with the smallest prediction error. This results in a new anomaly detection model.

[0162] Furthermore, as the abnormal sample set is updated, the anomaly detection model is continuously trained and updated, ensuring it remains constantly updated and does not need to be taken offline. This achieves automated collection of abnormal samples and iterative training of the anomaly detection model, enabling timely updates without relying on manual intervention.

[0163] Optionally, after obtaining the updated anomaly recognition model, the computer can also use anomaly samples in the test set to test the updated anomaly recognition model and obtain its evaluation results, which can be viewed by technical personnel.

[0164] Finally, the trained anomaly detection model can continue to work, performing anomaly detection on other samples in the overall data. When an anomaly sample is identified, the relevant information of the anomaly sample can be sent to the processing platform for appropriate processing, such as account suspension, deletion, or blocking, etc., without limitation.

[0165] Figure 8 This is a block diagram illustrating a data processing apparatus according to an embodiment of this application. Figure 8 As shown, this data processing device can be applied to a computer, and the device includes:

[0166] A data processing apparatus, comprising:

[0167] The acquisition unit 810 is used to acquire positive samples of the anomaly recognition model, wherein the positive samples refer to the original samples that are identified as normal by the anomaly recognition model;

[0168] Processing unit 820 is used to perform anomaly screening on the positive samples to obtain abnormal samples;

[0169] The processing unit 820 is further configured to filter the feature information of the abnormal sample through the tree structure information associated with the anomaly recognition model to obtain the filtered initial sample.

[0170] The training unit 830 is used to iteratively train the anomaly recognition model based on the initial samples to obtain the trained anomaly recognition model.

[0171] In one embodiment of this application, based on the foregoing scheme, the acquisition unit 810 is further configured to acquire the tree structure information associated with the anomaly recognition model; the processing unit 820 is further configured to calculate the feature importance of each feature dimension in the anomaly sample based on the tree structure information; and the filtering unit is further configured to filter the feature dimensions in the feature information of the anomaly sample whose feature importance is lower than the preset importance threshold to obtain the filtered initial sample.

[0172] In one embodiment of this application, based on the foregoing scheme, the processing unit 820 is further configured to add the feature dimension to the tree structure information; calculate the gain value corresponding to at least two branch nodes of the feature dimension in the tree structure information; the acquisition unit 810 is further configured to acquire the gain value corresponding to each other branch node in the tree structure information; the processing unit 820 is further configured to calculate the feature importance corresponding to the feature dimension based on the gain value corresponding to each other branch node in the tree structure information and the gain value corresponding to at least two branch nodes of the feature dimension.

[0173] In one embodiment of this application, based on the foregoing scheme, the processing unit 820 is further configured to, if the number of tree structure information is at least two, take the average feature importance of the feature dimension in the at least two tree structure information as the feature importance corresponding to the feature dimension.

[0174] In one embodiment of this application, based on the aforementioned scheme, the number of abnormal samples is at least two; the processing unit 820 is further configured to divide the at least two abnormal samples into at least two sample subsets, each sample subset corresponding to a subset number; the acquisition unit 810 is further configured to acquire a number of trainable models equal to the number of the at least two sample subsets, each trainable model corresponding to a model number, and each trainable model being the same as the abnormal identification model; the processing unit 820 is further configured to acquire, for each trainable model, other sample subsets besides those whose subset numbers match the model number of the trainable model; and the training unit 830 is further configured to train the trainable model based on the acquired other sample subsets to obtain at least two trained models; and select the trained abnormal identification model from the at least two trained models.

[0175] In one embodiment of this application, based on the aforementioned scheme, the processing unit 820 is further configured to calculate the prediction error for each trained model; and to determine the trained model with the smallest prediction error among at least two trained models as the trained anomaly recognition model.

[0176] In one embodiment of this application, based on the foregoing scheme, the steps are as follows: obtaining a target sample subset whose subset index matches the model index of the trained model; inputting the target sample subset into the trained model for testing; and calculating the prediction error of the trained model when testing the target sample subset.

[0177] In one embodiment of this application, based on the aforementioned scheme, the processing unit 820 is further configured to calculate the abnormal suspicion degree corresponding to the positive sample based on a preset suspicious analysis rule; filter out undetermined samples from the positive samples whose abnormal suspicion degree is greater than a preset suspicion degree threshold; obtain the review result of the undetermined samples; and determine the undetermined samples whose review result is characterized as abnormal as the abnormal samples.

[0178] It should be noted that the apparatus provided in the foregoing embodiments and the method provided in the foregoing embodiments belong to the same concept, and the specific way in which each module and unit performs operations has been described in detail in the method embodiments.

[0179] Embodiments of this application also provide a data processing device, including: at least one processor; and a memory for storing at least one program, which, when executed by the at least one processor, causes the electronic device to perform the aforementioned data processing method.

[0180] Figure 9 This is a schematic diagram of the structure of a computer system suitable for implementing the data processing apparatus of the present application.

[0181] It should be noted that, Figure 9 The computer system 900 of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.

[0182] like Figure 9 As shown, the computer system 900 includes a Central Processing Unit (CPU) 901, which can perform various appropriate actions and processes, such as executing the methods described in the above embodiments, based on programs stored in Read-Only Memory (ROM) 902 or programs loaded from storage portion 908 into Random Access Memory (RAM) 903. The RAM 903 also stores various programs and data required for system operation. The CPU 901, ROM 902, and RAM 903 are interconnected via a bus 904. An Input / Output (I / O) interface 905 is also connected to the bus 904.

[0183] The following components are connected to I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to I / O interface 905 as needed. Removable media 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 910 as needed so that computer programs read from them can be installed into storage section 908 as needed.

[0184] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 909, and / or installed from removable medium 911. When the computer program is executed by central processing unit (CPU) 901, it performs various functions defined in the system of this application.

[0185] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. For example, a computer-readable medium can be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable medium may include, but are not limited to: an electrical connection having at least one wire, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0186] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains at least one executable instruction for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0187] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.

[0188] Another aspect of this application provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the data processing method as described above. This computer-readable medium may be included in the electronic device described in the above embodiments, or it may exist independently and not assembled into the electronic device.

[0189] Another aspect of this application provides a computer program product or computer program including computer instructions stored in a computer-readable medium. A processor of a computer device reads the computer instructions from the computer-readable medium and executes the computer instructions, causing the computer device to perform the data processing methods provided in the various embodiments described above.

[0190] The above description is merely a preferred exemplary embodiment of this application and is not intended to limit the implementation of this application. Those skilled in the art can easily make corresponding modifications or alterations based on the main concept and spirit of this application. Therefore, the scope of protection of this application should be determined by the scope of protection claimed in the claims.

Claims

1. A data processing method, characterized in that, include: Obtain positive samples from the anomaly detection model, whereby positive samples refer to original samples that are identified as normal by the anomaly detection model. The positive samples are subjected to anomaly screening to obtain abnormal samples; The feature information of the abnormal samples is filtered by the tree structure information associated with the anomaly recognition model to obtain the filtered initial samples. The anomaly detection model is iteratively trained based on the initial samples to obtain the trained anomaly detection model.

2. The method according to claim 1, characterized in that, The step of filtering the feature information of the abnormal samples through the tree structure information associated with the anomaly recognition model to obtain the filtered initial samples includes: Obtain the tree structure information associated with the anomaly recognition model; Calculate the feature importance of each feature dimension in the abnormal sample based on the tree structure information; The feature dimensions whose feature importance is lower than the preset importance threshold in the feature information of the abnormal samples are filtered to obtain the filtered initial samples.

3. The method according to claim 2, characterized in that, The calculation of the feature importance of each feature dimension in the abnormal sample based on the tree structure information includes: Add the feature dimension to the tree structure information; Calculate the gain value corresponding to at least two branch nodes of the feature dimension in the tree structure information; Obtain the gain value corresponding to each other branch node in the tree structure information; Based on the gain value corresponding to each other branch node in the tree structure information and the gain value corresponding to at least two branch nodes of the feature dimension, the feature importance corresponding to the feature dimension is calculated.

4. The method according to claim 3, characterized in that, The step of calculating the feature importance corresponding to the feature dimension based on the gain value corresponding to each other branch node in the tree structure information and the gain values ​​corresponding to at least two branch nodes of the feature dimension includes: If the number of tree structure information is at least two, then the average feature importance of the feature dimension in the at least two tree structure information is taken as the feature importance corresponding to the feature dimension.

5. The method according to claim 1, characterized in that, The number of abnormal samples is at least two; the iterative training of the anomaly recognition model based on the initial samples to obtain the trained anomaly recognition model includes: Divide at least two abnormal samples into at least two sample subsets, and each sample subset has a corresponding subset number; Obtain the same number of trainable models as the at least two sample subsets, each trainable model having a corresponding model number, and each trainable model being the same as the anomaly detection model; For each model to be trained, obtain other sample subsets besides the sample subset whose subset index matches the model index of the model to be trained, and train the model to be trained based on the obtained other sample subsets to obtain at least two trained models. The trained anomaly detection model is selected from the at least two trained models.

6. The method according to claim 5, characterized in that, The step of selecting the trained anomaly detection model from the at least two trained models includes: Calculate the prediction error for each trained model; The training model with the smallest prediction error among at least two trained models is determined as the trained anomaly detection model.

7. The method according to claim 6, characterized in that, The calculation of prediction error for each trained model includes: Obtain the target sample subset whose subset index matches the model index of the trained model; The target sample subset is input into the trained model for testing; Calculate the prediction error of the trained model when testing the target sample subset.

8. The method according to claim 1, characterized in that, The positive samples are subjected to anomaly screening to obtain abnormal samples, including: The degree of suspicion of the positive sample is calculated based on the preset suspiciousness analysis rules; From the positive samples, select undetermined samples whose anomaly suspicion level is greater than a preset suspicion threshold; Obtain the review results for the pending samples; The pending samples whose audit results are characterized as abnormal are identified as the abnormal samples.

9. A data processing apparatus, characterized in that, include: The acquisition unit is used to acquire positive samples of the anomaly recognition model, wherein the positive samples refer to the original samples that are identified as normal by the anomaly recognition model; The processing unit is used to perform anomaly screening on the positive samples to obtain abnormal samples; The processing unit is further configured to filter the feature information of the abnormal sample through the tree structure information associated with the anomaly recognition model to obtain the filtered initial sample. The training unit is used to iteratively train the anomaly recognition model based on the initial samples to obtain the trained anomaly recognition model.

10. A data processing device, characterized in that, include: Memory, which stores computer-readable instructions; A processor reads computer-readable instructions stored in memory to perform the method of any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the computer's processor, cause the computer to perform the method of any one of claims 1 to 8.

12. A computer program product comprising computer instructions, characterized in that, When the computer instructions are executed by the processor, they implement the method as described in any one of claims 1 to 8.

Images