Data processing method and apparatus

Abstract

Embodiments of the present specification provide a data processing method and device, wherein the data processing method comprises: determining at least two data providing ends and data detection rules of the at least two data providing ends; performing clustering processing on the data detection rules of the at least two data providing ends to obtain similar data detection rules between the at least two data providing ends; constructing a data detection object based on the similar data detection rules, and utilizing the data detection object to detect to-be-detected data provided by the at least two data providing ends to determine target data in the to-be-detected data; thereby reducing the computing resources required to support the data detection object to perform data detection, and the maintenance cost of the data detection object, and avoiding the problems of waste of computing resources and human cost.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of computer technology, and in particular to a data processing method. Background Technology

[0002] With the development of internet technology, there is a need for data transmission between internet companies to better provide services to users. For example, in the field of internet finance, internet finance platforms will transmit data with banks so that users can manage their funds in the bank through the internet finance platform.

[0003] Therefore, in order to ensure data security and validity during data transmission between internet companies, internet companies need to perform data inspection on the received data. In existing technologies, the internet company will use decision trees to perform data inspection and build corresponding decision trees for each of the multiple internet companies that provide data. For example, an internet finance platform will build a corresponding decision tree for each bank for data inspection.

[0004] However, building multiple decision trees requires a significant amount of time and manpower. Furthermore, it also requires substantial computing resources to support data testing for these multiple decision trees, as well as significant manpower to maintain them, further wasting computing resources and manpower. Summary of the Invention

[0005] In view of the above, embodiments of this specification provide a data processing method. One or more embodiments of this specification also relate to a data processing apparatus, a computing device, a computer-readable storage medium, and a computer program, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a data processing method is provided, comprising:

[0007] Identify at least two data providers and the data detection rules for the at least two data providers;

[0008] Clustering is performed on the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers;

[0009] Based on the similar data detection rules, a data detection object is constructed, and the data detection object is used to detect the data to be detected provided by the at least two data providers to determine the target data in the data to be detected.

[0010] According to a second aspect of the embodiments of this specification, a data processing apparatus is provided, comprising:

[0011] The determination module is configured to determine at least two data providers and data detection rules for the at least two data providers;

[0012] The clustering module is configured to perform clustering processing on the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers.

[0013] The detection module is configured to construct a data detection object based on the similar data detection rules, and use the data detection object to detect the data to be detected provided by the at least two data providers, and determine the target data in the data to be detected.

[0014] According to a third aspect of the embodiments of this specification, a computing device is provided, comprising:

[0015] Memory and processor;

[0016] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method.

[0017] According to a fourth aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores computer-executable instructions, which, when executed by a processor, implement the steps of the data processing method.

[0018] According to a fifth aspect of the embodiments of this specification, a computer program is provided, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the data processing method.

[0019] The data processing method provided in this specification includes: determining at least two data providers and data detection rules for the at least two data providers; performing clustering processing on the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers; constructing a data detection object based on the similar data detection rules, and using the data detection object to detect the data to be detected provided by the at least two data providers to determine the target data in the data to be detected.

[0020] Specifically, this method clusters the data detection rules corresponding to at least two data providers and constructs a data detection object for each data provider based on the similar data detection rules obtained after clustering. This avoids the problem of constructing multiple similar decision trees for multiple data providers, which would require significant time and manpower, thus saving time and manpower costs. Furthermore, by using the data to be detected provided by at least two data providers within this data detection object, the target data in the data to be detected is identified. This reduces the computational resources required to support data detection for this data detection object, as well as the maintenance costs of the data detection object, avoiding the waste of computational resources and manpower costs. Attached Figure Description

[0021] Figure 1 This is a schematic diagram illustrating an application scenario of a data processing method provided in one embodiment of this specification;

[0022] Figure 2 This is a flowchart illustrating a data processing method provided in one embodiment of this specification;

[0023] Figure 3 This is a schematic diagram of a decision tree in a data processing method provided in one embodiment of this specification;

[0024] Figure 4 This is a flowchart illustrating the processing procedure of a data processing method provided in one embodiment of this specification.

[0025] Figure 5 This is a schematic diagram of a data processing structure provided in one embodiment of this specification;

[0026] Figure 6 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0027] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0028] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0029] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0030] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0031] Timeliness rules: Rules for checking whether data processing is delayed.

[0032] Consistency rules: Rules that check for abnormal relationships between data.

[0033] Accuracy rules: Rules for checking for anomalies in data content, density, thresholds, etc.

[0034] Integrity rules: Rules for checking for missing data.

[0035] Stability rules: Rules for checking whether the data fluctuates.

[0036] Uniqueness rule: A rule for checking whether data is duplicated.

[0037] Decision trees: a method for approximating discrete function values. It's a typical classification method that first processes the data, using inductive algorithms to generate readable rules and decision trees, and then uses these rules to analyze new data. Essentially, a decision tree is a process of classifying data through a series of rules.

[0038] K-means algorithm: A clustering algorithm that aggregates data by separating samples into n classes with the same variance.

[0039] KNN algorithm (K-Nearest Neighbor): A clustering algorithm that can be used for classification and regression.

[0040] With the development of internet technology, there is a need for data transmission between internet companies to better provide services to users. For example, in the field of internet finance, internet finance platforms will transmit data with banks so that users can manage their funds in the bank through the internet finance platform.

[0041] Therefore, in order to ensure data security and validity during data transmission between internet companies, internet companies need to conduct technical inspections on the data sent by other internet companies they connect to. In existing technologies, the internet company will use decision trees to perform data inspection and build corresponding decision trees for each of the other internet companies it connects to. For example, an internet finance platform will build a corresponding decision tree for each bank for data inspection.

[0042] As more channels (i.e., other internet companies, such as banks and other financial service institutions) connect, the number of detection rules for data inspection by this decision tree will also increase. This results in slow linear detection speed for all rules (if rules are added according to channels, the number of rules will be very large and difficult to maintain).

[0043] However, building multiple decision trees requires a significant amount of time and manpower. Furthermore, it also requires substantial computing resources to support data testing for these multiple decision trees, as well as significant manpower to maintain them, further wasting computing resources and manpower.

[0044] Furthermore, in practical applications, current conventional data verification methods cannot meet the accuracy requirements for detecting large-scale complex data, and manual detection is inefficient. For example, this specification provides a solution for quickly filtering outliers from large-scale data. This solution requires six steps to achieve data detection: data preprocessing, feature selection and transformation, variable initialization, iteration, outlier index calculation, and outlier filtering. Data preprocessing involves preprocessing the data to eliminate inconsistencies and normalize each data point. Specific operations include data cleaning, data integration, data transformation, and data reduction. The resulting feature matrix is ​​denoted as T, with a size of N*M, where N is the number of all samples and M is the number of original feature attributes.

[0045] Feature selection and transformation are as follows: Feature selection is to filter out attributes that contribute little or no to subsequent operations from all attributes. Feature transformation is to use the current attributes to obtain the attributes of the new feature space through transformation. The resulting feature matrix is ​​denoted as Ts, and its size is N*m, where N is the number of all samples and m is the number of attributes after filtering and transformation.

[0046] The initialization variables are: let Co and Cs be two zero vectors of length N, which are used to store the sum of outlier factors and the number of sample screenings in subsequent calculations, respectively.

[0047] The iteration is as follows: vectors Co and Cs are updated through the following iterations, and the iteration terminates after a certain number of iterations k:

[0048] (I) Randomly select a subset of samples with a fixed size of η;

[0049] (2) Add I to the corresponding element value in vector Cs;

[0050] (3) Filter the corresponding rows from matrix Ts and calculate the local isolation factor corresponding to the matrix;

[0051] (4) Add the local isolation factor obtained in step one to the corresponding values ​​of vector Co.

[0052] The outlier index is calculated as follows: the vector COI is obtained by calculating the outlier factor through vectors Co and Cs, and the calculation formula is: COI = Co / Cs.

[0053] Outlier data screening involves selecting the top I samples as outliers based on the numerical values ​​corresponding to the vector COIs, from largest to smallest.

[0054] However, this scheme has the following drawbacks:

[0055] 1. It cannot incorporate expert experience, has low interpretability, cannot guarantee that all abnormal data can be detected, and has low accuracy.

[0056] 2. For large batches of multi-feature data, compared with automatic detection methods, it is impossible to effectively prove that the detection efficiency is improved.

[0057] This specification addresses the issues of low efficiency or poor accuracy in detecting large-scale complex data using some data detection methods, as well as the significant time and manpower costs associated with constructing multiple decision trees. It provides a data processing method that improves the efficiency and accuracy of detecting large-scale complex data based on decision trees with similar decision rules. This approach involves identifying at least two data providers and their respective data detection rules; then, clustering these rules to obtain similar data detection rules between the two providers; finally, constructing a data detection object based on these similar rules, and using this object to detect the data provided by the at least two providers to identify the target data. This method achieves a significant leap in both efficiency and accuracy, even with numerous detection rules. Furthermore, it saves time and manpower, reduces the computational resources required to support the data detection object, and lowers the maintenance costs of the object, thus avoiding waste of computational resources and manpower.

[0058] Specifically, this specification provides a data processing method, and also relates to a data processing method, a computing device, a computer-readable storage medium, and a computer program, which will be described in detail in the following embodiments.

[0059] Figure 1 This diagram illustrates an application scenario of a data processing method according to an embodiment of this specification, wherein... Figure 1 In this context, financial institution 102 can be understood as the data provider, decision tree model 104 can be understood as the data detection object, and abnormal data can be understood as the target data. Based on this, the data processing method provided in this specification can receive the data to be detected sent by financial institution 102. The data to be detected can be financial data, including but not limited to bills, transaction records, credit limit increase information, credit information, etc.

[0060] After receiving the data to be detected, it can be input into the decision tree model 104, which then performs data detection to identify abnormal data. The decision tree model 104 is based on similarity detection rules corresponding to the at least two financial institutions 102. These similarity detection rules are obtained by clustering the data detection rules corresponding to the at least two financial institutions 102. In other words, the data processing method provided in this specification takes into account that some fields have the same detection rules across different channels. Therefore, it identifies channels with similar data detection rules and clusters the data detection rules used to verify the data provided by these channels to obtain similar data detection rules for these channels. The set of these similar data detection rules is then placed in a tree structure to construct a decision tree. For example, a channel could be a bank, and several channels such as Bank A, Bank B, and Bank C might share as many as a dozen rules (i.e., similar data detection rules). Therefore, clustering can be used to extract the similar data detection rules from these channels, build a decision tree, and then use this decision tree to detect the data provided by these channels.

[0061] Based on this, the data processing method provided in this specification considers that in scenarios involving data detection of financial data such as bank statements, bills, and credit limit increases, current conventional data verification methods cannot meet the accuracy requirements for detecting large-scale complex data, and manual detection is inefficient. Furthermore, it requires significant computing resources to support multiple decision trees for data detection, and incurs substantial manpower costs for maintaining these decision trees. By accumulating similar expert experience from multiple channels into the decision trees, and constructing a decision tree model, the efficiency and accuracy of detecting large-scale complex data are improved. This solves the problems of low detection efficiency for large-scale complex data, inability to cover all anomaly types, and low accuracy in existing solutions. Moreover, it reduces the computing resources required to support data detection using the decision trees, as well as the maintenance costs of the decision trees, avoiding the waste of computing resources and manpower.

[0062] Figure 2 A flowchart of a data processing method according to an embodiment of this specification is shown, which specifically includes the following steps.

[0063] Step 202: Determine at least two data providers and the data detection rules for the at least two data providers.

[0064] The data provider can be understood as the entity that provides the data to be tested. For example, the data provider could be other internet companies, banks, financial institutions, etc. The data testing rules can be understood as the rules used to verify and test the data to be tested. These rules include, but are not limited to, timeliness rules, consistency rules, accuracy rules, completeness rules, stability rules, uniqueness rules, rules for missing key fields, rules for abnormal relationships, rules for abnormal formats, rules for abnormal numerical thresholds, etc. This specification does not impose specific limitations on these rules.

[0065] The instruction manual requires that when validating large-scale, complex data, data detection rules are needed to determine whether the data meets the requirements. These data detection rules can be obtained through the accumulation of expert experience. In practical applications, the accumulation of expert experience can be executed through a neural network model. That is, the data detection rules for at least two data providers can be obtained through a rule-determining model. Specific steps include first determining the historical data to be detected for the at least two data providers, and the corresponding detection results. The historical data to be detected can be understood as the data provided by the at least two data providers in the past, after data detection. The data detection method for this historical data can be implemented using any method capable of data detection; this instruction manual does not impose specific restrictions. For example, the historical data to be detected can be processed manually, or it can be processed using an application, any data detection tool, or an artificial intelligence device.

[0066] After determining the historical data to be detected corresponding to at least two data providers, and the detection results corresponding to the historical data to be detected, the historical data to be detected and the detection results are input into the trained rule determination model. The rule determination model is then used to analyze the historical data to be detected and the detection results to obtain the data detection rules corresponding to at least two data providers.

[0067] For example, the data provider is Bank A. The historical data to be monitored could be the average monthly revenue of individual business owner A (100,000 yuan), the average monthly revenue of individual business owner B (150,000 yuan), and the average monthly revenue of individual business owner C (5 million yuan). Correspondingly, the monitoring results for this historical data can indicate whether the data is abnormal. The monitoring results for the above three historical data sets can be normal data, normal data, and abnormal data. The historical data and monitoring results are input into a rule-determining model. This model analyzes the monitoring results and the historical data to determine the data monitoring rule for Bank A. The rule is to classify the average monthly revenue of individual businesses exceeding 1 million yuan as abnormal data, and the average monthly revenue of individual businesses less than or equal to 1 million yuan as normal data.

[0068] It should be noted that the model determined by this rule can be any neural network model that implements the above rule. This specification does not impose any specific restrictions on it, such as convolutional neural networks.

[0069] Furthermore, in practical applications, the data processing method provided in this manual can classify abnormal data detected by data detection rules into multiple risk types, and classify the data detection rules into multiple rule types based on these risk types.

[0070] For example, by analyzing abnormal data, the risk type of the abnormal data can be classified into three categories: data with outdated parsing, data with incorrect data accuracy, and data with unstable magnitude. It should be noted that the above classification into three risk types is only an example to explain the data processing method provided in this manual. The number of risk types into which abnormal data is classified can be set according to the actual application scenario.

[0071] Based on the abnormal data of three risk types, the data detection rules are divided into three rule types: strong risk rules, high risk rules, and low risk rules. The specific data detection rules are shown in Table 1.

[0072] Table 1

[0073]

[0074]

[0075] As shown in Table 1 above, different data detection rules can have different detection methods under different rule types.

[0076] Step 204: Perform clustering processing on the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers.

[0077] Specifically, considering that some field content detection rules are the same across different channels, the data processing method provided in this specification clusters the data detection rules of at least two data providers to obtain similar data detection rules between them. It should be noted that any method capable of clustering data detection rules can be used during the clustering process, and this specification does not impose any specific limitations. In one embodiment provided in this specification, a clustering algorithm can be used to cluster the data processing rules, thereby improving the efficiency of obtaining similar data detection rules and avoiding the waste of manpower caused by manually clustering the data detection rules. Specifically, the clustering process of the data detection rules of the at least two data providers to obtain similar data detection rules between them includes:

[0078] A preset clustering algorithm is used to cluster the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers.

[0079] The preset clustering algorithm can be understood as a method that can perform clustering processing on the data detection rules. For example, the preset clustering algorithm can be the K-means algorithm, the KNN algorithm, etc.

[0080] Specifically, the data processing method provided in this specification can use a pre-defined clustering algorithm to cluster the data detection rules of at least two data providers, thereby obtaining similar data detection rules between at least two data providers. For example, data detection rules R1 and R3 corresponding to bank A, and data detection rules R1 and R2 corresponding to bank B, are input into the K-means algorithm. The K-means algorithm is then used to cluster these data detection rules, thereby determining that data detection rule R1 is a common data detection rule between bank A and bank B, that is, a data detection rule shared between bank A and bank B.

[0081] Furthermore, in one embodiment provided in this specification, the step of using a predictive clustering algorithm to cluster the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers includes:

[0082] The similarity between the data detection rules of the at least two data providers is calculated based on a preset clustering algorithm to determine the similarity between the data detection rules.

[0083] Based on the similarity, the data detection rules are clustered to obtain similar data detection rules between the at least two data providers.

[0084] Here, similarity can be understood as data that characterizes the degree of similarity between data detection rules. For example, similarity can be the similarity distance between data detection rules; similarity distance can be Minkowski distance, Euclidean distance, etc.

[0085] Specifically, in the process of clustering the data detection rules based on the preset clustering algorithm, the similarity of the data detection rules of at least two data providers can be calculated first to obtain the similarity between each data detection rule. Then, the preset clustering algorithm performs clustering processing on the data detection rules based on the similarity to obtain similar data detection rules between at least two data providers, thereby improving the efficiency of obtaining similar data detection rules and avoiding the waste of human resources caused by manually clustering the data detection rules.

[0086] Continuing with the previous example, data detection rules R1 and R3 corresponding to bank A, and data detection rules R1 and R2 corresponding to bank B, are input into the K-means algorithm. The K-means algorithm is used to determine the distance between these data detection rules. Based on the distance between the data detection rules, it is determined that data detection rule R1 corresponding to bank A and data detection rule R1 corresponding to bank B are the same data detection rule. The closer the distance between two data detection rules, the more similar they are.

[0087] Step 206: Construct a data detection object based on the similar data detection rules, and use the data detection object to detect the data to be detected provided by the at least two data providers, and determine the target data in the data to be detected.

[0088] Specifically, in the data processing method provided in this specification, after determining the similar data detection rule, the data detection object can be constructed based on the similar data detection rule. Then, the constructed data detection object is used to detect the data to be detected provided by at least two data providers, thereby determining the target data from the data to be detected.

[0089] In practical applications, the target data will differ depending on the scenario in which the data processing method is applied. For example, in an anomaly detection scenario, the target data will be anomaly data; in a normal data detection scenario, the target data will be normal data.

[0090] In one embodiment provided in this specification, the similar data detection rules are at least two;

[0091] Accordingly, constructing the data detection object based on the similar data detection rules includes:

[0092] At least two similar data detection rules are used as decision tree nodes, and a decision tree model is constructed based on the decision tree nodes.

[0093] Specifically, the data detection object can be a decision tree model, and at least two similar data detection rules are obtained by clustering the data detection rules from at least two data providers. Based on this, during the construction of the decision tree, these at least two similar data detection rules can be used as decision tree nodes, and the decision tree model can be constructed based on these decision tree nodes. See also Figure 3 , Figure 3 This is a schematic diagram of a decision tree in a data processing method provided in one embodiment of this specification; see also Figure 3 As can be seen, the decision tree includes multiple decision nodes, each consisting of a data detection rule. In practical applications, the data to be detected can be input into decision tree 1, and the data will be detected by multiple decision tree nodes in decision tree 1. The input data of subsequent decision tree nodes are the data to be detected that have passed the detection, which are output by the preceding decision tree nodes; while the data to be detected that fails the detection is classified separately, and a corresponding anomaly score is assigned to each type of data; finally, based on the anomaly score, it can be determined whether a data to be detected is normal data or abnormal data.

[0094] It should also be noted that, due to the potentially large number of similar data detection rules between at least two data providers, the decision tree constructed based on these rules will be quite large. Furthermore, the large number of nodes in this decision tree may lead to excessively long detection times, further reducing data detection efficiency. Therefore, multiple decision trees can be constructed based on similar data detection rules.

[0095] Alternatively, if the amount of data to be detected provided by at least two data providers is large, a single decision tree may not be able to quickly detect the large amount of data, resulting in low data detection efficiency. Therefore, multiple decision tree objects can be constructed based on similar data detection rules, where "multiple" can be understood as at least two.

[0096] It should be noted that the tree structures of the multiple decision trees constructed based on the similarity data detection rules can be the same or different. That is, the structures of the multiple data detection objects constructed based on the similarity data detection rules can be the same or different. For example, there are four similarity data detection rules: R1, R2, R3, and R4, between Bank A and Bank B. Considering the above issue, two decision tree models, T1 and T2, can be constructed based on these four similarity data detection rules. Both decision trees T1 and T2 contain decision tree nodes composed of rules R1, R2, R3, and R4. This ensures that the tree structure of each decision tree model is identical.

[0097] Alternatively, a decision tree T1 can be constructed based on rule R1 and rule R2, and another decision tree T2 can be constructed based on rule R3 and rule R4. The tree structures of decision tree T1 and decision tree T2 are different.

[0098] Furthermore, in the data processing method provided in this specification, to ensure that the constructed decision tree achieves better detection results, different similar data detection rules are analyzed, and the similar data detection rules with better detection results are used first to detect the data to be detected. This allows the decision tree to detect a large number of abnormal data from the data to be detected at the beginning of the data detection process, and the subsequent decision tree can detect a smaller number of data to be detected, thereby improving the efficiency of data detection and avoiding the problem of low data detection efficiency caused by all nodes in the decision tree needing to perform a large number of data detection operations. Based on this, the data processing method provided in this specification can use the most core information entropy decay scheme in the decision tree to find the matching order, thereby improving the efficiency of the decision tree. It constructs a decision tree that is more in line with practical applications at a lower cost. Compared with other complex decision tree algorithms, the decision tree constructed in this specification has high interpretability and can output the cause of anomalies. Specifically, the step of using at least two similar data detection rules as decision tree nodes and constructing a decision tree model based on the decision tree nodes includes:

[0099] Determine the data detection ratio of at least two similar data detection rules, wherein the data detection ratio is determined by the detection results of detecting historical data to be detected using the data detection rules;

[0100] The at least two similar data detection rules are sorted based on the data detection ratio to obtain a rule sequence;

[0101] At least two similar data detection rules are used as decision tree nodes, and a decision tree model is constructed based on the decision tree nodes and the rule sequence.

[0102] The data detection ratio can be understood as the coverage rate of the similar data detection rule over historical data to be detected. It should be noted that, compared to the general coverage calculation method (i.e., covered data / total data), the coverage calculation method provided in this embodiment is to check whether a piece of data to be detected can be detected and output as an anomaly by the current rule. If it can output an anomaly, the number of times it outputs an anomaly is incremented by 1. Finally, the coverage rate corresponding to the rule is determined by calculating the number of times the rule can output an anomaly / the number of anomaly data. This allows for a more accurate determination of the detection performance of each similar data detection rule on anomaly data. For example, for 100 pieces of data to be detected, similar data detection rule A outputs 50 anomaly data; and there are 70 anomaly data among the 100 pieces of data to be detected. Based on this, dividing 50 by 70 determines that the coverage rate of similar data detection rule A is approximately 0.7. Here, the covered data can be the anomaly data detected by the data detection rule.

[0103] The rule sequence can be understood as a sequence of rules sorted according to the size of the data detection ratio. For example, the coverage ratios of similar data detection rules [R1, R2, R3...R7] are [0.31, 0.27, 0.61, 0.35, 0.43, 0.29, 0.11], respectively. Sort the similar data detection rules based on the size of the coverage ratio to obtain the rule sequence [R3, R5, R4, R1, R6, R2, R7].

[0104] Specifically, in the process of constructing a decision tree, the data processing method provided in this manual first requires determining the data detection ratio of at least two similar data detection rules. It should be noted that the data detection ratio is determined by the detection results of the historical data to be detected using the data detection rules.

[0105] Then, based on the data detection ratio, the at least two similar data detection rules are sorted from largest to smallest to obtain the rule sequence of the at least two similar data detection rules;

[0106] Finally, these at least two data detection rules are used as decision tree nodes for data detection within the decision tree. Based on the position information of each similar data detection rule in the rule sequence, the priority of each decision tree node within the decision tree is determined. Finally, a decision tree is constructed based on this priority and the decision tree node. It should be noted that the earlier a similar data detection rule appears in the rule sequence, the higher the priority of its corresponding decision tree node within the decision tree. The similar data detection rule at the first position in the rule sequence has the highest priority and is the root node of the decision tree. See also... Figure 3 It can be seen that the rule for missing key field should be the first similar data detection rule in the rule sequence. Therefore, this rule has the highest priority when constructing the decision tree. The higher the priority of a decision tree node, the closer it is to the root node; the decision tree node with the highest priority is the root node of the decision tree.

[0107] Furthermore, in the embodiments provided in this specification, considering that in order to improve the detection performance of the decision tree, it is necessary to utilize the core information entropy decay scheme in the decision tree to find the matching order (i.e., the rule sequence mentioned above); and the key to determining this matching order lies in how to determine the coverage rate. Based on this, in order to more accurately determine the coverage rate of different rules for abnormal data, this specification can use an iterative query method to determine a better-performing coverage rate for each rule. Specifically, determining the data detection ratio of at least two similar data detection rules includes steps one and two:

[0108] Step 1: Use the at least two similar data detection rules as the first detection rule, and use the initial data detection ratio of the at least two similar data detection rules for the historical data to be detected as the first data detection ratio.

[0109] Specifically, the step of using the at least two similar data detection rules as the first detection rule, and using the initial data detection ratio of the at least two similar data detection rules for the historical data to be detected as the first data detection ratio, includes:

[0110] Determine at least two similar data detection rules and an initial data detection ratio for historical data to be detected, wherein the initial data detection ratio is determined by the detection results of the historical data to be detected using the data detection rules;

[0111] The at least two similar data detection rules are used as the first detection rule, and the initial data detection ratio is used as the first data detection ratio.

[0112] The historical data to be detected can be understood as data detected within a specific historical time interval, such as 30 days, 7 days, etc.

[0113] Following the previous example, the data processing method provided in this manual calculates the coverage rate C of each rule R for financial data within 30 days, using offline data on a daily basis and a 30-day time window. For example, the similar data detection rules are: [content anomaly, parameter error, correlation, size relationship, missing entire row, semantic incompleteness, data premature], which are formally defined as [r1, r2, r3...r7], where r1 represents content anomaly, r2 represents parameter error, and so on.

[0114] Then, the data coverage of each rule over the past 30 days is calculated offline → [c1, c2, c3....7], where c1 represents the coverage of r1. For example, the calculated coverage is [0.31, 0.27, 0.61, 0.35, 0.43, 0.29, 0.11].

[0115] Subsequently, the above 7 rules were determined as the first rule, and the above coverage rate was used as the first data detection ratio, so that it would be easier to determine a higher performance coverage rate based on the first rule and the first data as the similar data detection rule.

[0116] Step 2: Based on the first detection rule and the first data detection ratio, determine the corresponding data detection ratio for the at least two similar data detection rules.

[0117] Specifically, based on the first detection rule and the first data detection ratio, the corresponding data detection ratio is determined for at least two similar data detection rules through iterative query.

[0118] Specifically, determining the corresponding data detection ratio for the at least two similar data detection rules based on the first detection rule and the first data detection ratio includes:

[0119] S1: Use the historical data to be detected as the first data;

[0120] S2: Determine the first data detection ratio of the first detection rule for the first data;

[0121] S3: The first data detection ratio that meets the preset rule determination conditions is determined as the target data detection ratio, and the first detection rule corresponding to the target data detection ratio is taken as the target detection rule;

[0122] S4: If it is determined that there are other detection rules besides the target detection rule in the first detection rules, the detection pass data corresponding to the target detection rule is determined from the first data based on the target data detection ratio;

[0123] S5: Take the other detection rules as the first detection rule, take the detection pass data as the first data, and continue to execute step S2 until there are no other detection rules besides the target detection rule in the first detection rule;

[0124] S6: Based on the target data detection ratio of the target detection rule, determine the data detection ratio of the at least two similar data detection rules.

[0125] The target detection rule can be understood as a similar data detection rule to determine the final coverage.

[0126] The preset rule determination condition can be understood as the maximum coverage rate. Based on this, after determining the first data detection ratio of the first detection rule, the first data detection ratio that satisfies the preset rule determination condition can be the largest of the first data detection ratios.

[0127] Continuing with the previous example, the first data is financial data received within the last 30 days. The detection ratio for this first data is [0.31, 0.27, 0.61, 0.35, 0.43, 0.29, 0.11], and the first detection rules are [R1, R2, R3...R7]. Based on this, the final coverage rate corresponding to each rule is determined as follows:

[0128] S1: Determine the first detection rule [R1, R2, R3...R7], and the coverage of the first data [0.31, 0.27, 0.61, 0.35, 0.43, 0.29, 0.11].

[0129] S2: Determine the maximum coverage rate [0.61] from the coverage rates [0.31, 0.27, 0.61, 0.35, 0.43, 0.29, 0.11], and use the rule corresponding to the maximum coverage rate as the rule R3 to determine the final coverage rate; subsequently, R3 can be used as the root node of the decision tree, that is, the top node of the tree.

[0130] S3: Determine whether there are any other rules besides R3 in the first detection rule [R1, R2, R3...R7]. If so, based on the coverage rate of 0.61, determine the financial data that passed the detection of the R3 rule from the first data. That is, calculate the data that is not covered by the R3 rule, namely the data of 0.39(1-0.61).

[0131] S4: Use the six rules [R1, R2, R4, R5, R6, R7] as the first detection rules again, and use the data of 0.39 as the first data. Continue to execute step S1 until there are no other rules in the first detection rules besides the rule that determines the final coverage.

[0132] That is, calculate the coverage of the data with a value of 0.39 for all rules except R3, and get the coverage [c1, c2, c4, c5, c6, c7]. For example, if the new coverage is [0.43, 0.13, 0.45, 0.29, 0.09, 0.17], select the maximum coverage of 0.45, and use the corresponding rule R4 as the rule to determine the final coverage. This rule R4 can then be used as the second top node of the tree.

[0133] By analogy, all the top nodes of the tree are calculated, and R3, R4, R1, R7, R5, R2, and R6 are calculated as rules for determining the final coverage. Subsequently, R3, R4, R1, R7, R5, R2, and R6 can be used as the top nodes of the tree.

[0134] S5: Finally, we can determine the final coverage corresponding to the 7 rules R3, R4, R1, R7, R5, R2, and R6.

[0135] In practical applications, during the subsequent construction of the decision tree, R3, R4, R1, R7, R5, R2, and R6 can be used as the top nodes to construct the decision tree. (See [link to documentation]). Figure 3 The final decision tree can be constructed as follows: Figure 3 As shown.

[0136] In the embodiments provided in this specification, a first detection rule determined by at least two similar data detection rules, and a first data detection ratio determined by the initial data detection ratio of the at least two similar data detection rules for the historical data to be detected, are used to determine the corresponding data detection ratio for the at least two similar data detection rules. This determines a better coverage for each rule, further improving the detection performance of the constructed decision tree.

[0137] It should be noted that after constructing the decision tree model and incorporating expert experience (i.e., data detection rules) into the decision tree, it is necessary to further train the decision tree using training samples and sample labels. This automatically determines the priority of rules within the decision tree model, further improving its performance. Specifically, after using at least two similar data detection rules as decision tree nodes and constructing the decision tree model based on these nodes, the process further includes:

[0138] Determine the training samples and training labels for the decision tree model;

[0139] The training samples are processed using the decision tree model to obtain sample detection results;

[0140] Based on the sample detection results and the training labels, determine the sample data detection ratio of the at least two similar data detection rules for the training samples;

[0141] The decision tree model is adjusted based on the detection ratio of the sample data to obtain the adjusted decision tree model.

[0142] The training samples can be historically received financial data, and the training labels can be labels indicating whether the financial data is anomaly. It should be noted that these training samples are different from the historical data to be detected used when constructing the decision tree.

[0143] The detection result of this sample can be understood as the detection result that characterizes whether the training sample is the target data.

[0144] Specifically, the data processing method provided in this specification can determine the training samples and training labels for the decision tree model, input the training samples into the decision tree model, process the training samples using the decision tree model to obtain sample detection results, and then determine the sample data detection ratio of the at least two similar data detection rules for the training samples based on the sample detection results and the training labels. For example, the detection results of rule A for samples D1 and D2 indicate that samples D1 and D2 are both anomalous data, but based on the sample labels, only sample D1 is determined to be anomalous data; therefore, the coverage of rule A for the training samples is 0.5. It should be noted that the step of determining the sample data detection ratio of the at least two similar data detection rules for the training samples can be found in the corresponding or relevant content in the above description of determining the data detection ratio of at least two similar data detection rules, and this specification will not elaborate further on this. That is to say, the step of determining the sample data detection ratio of the at least two similar data detection rules for the training samples in the decision tree training stage is the same as the step of determining the data detection ratio of the at least two similar data detection rules in the decision tree construction stage.

[0145] Next, the decision tree model is adjusted based on the sample data detection ratio to obtain an adjusted decision tree model. This can be understood as sorting at least two similar data detection rules based on the sample data detection ratio to obtain a rule sequence, and then adjusting the priority of the decision tree nodes in the decision tree model based on the rule sequence, thereby obtaining the adjusted decision tree model. It should be noted that the steps of adjusting the decision tree model based on the sample data detection ratio to obtain the adjusted decision tree model can be found in the steps described above: sorting at least two similar data detection rules based on the data detection ratio to obtain a rule sequence; and constructing the decision tree model based on the decision tree nodes and the rule sequence. This specification will not elaborate further on these steps.

[0146] Furthermore, in one embodiment provided in this specification, the data detection object can be used to perform data detection on the data to be detected, thereby ensuring the security and integrity of the data. Specifically, using the data detection object to detect the data to be detected provided by at least two data providers and determine the target data in the data to be detected includes:

[0147] The data to be detected provided by the data provider is input into the data detection object to obtain the detection result of the data to be detected;

[0148] If the detection result meets the preset determination conditions, the data to be detected is determined as the target data.

[0149] The data to be detected can be understood as the constructed and trained decision tree model. The detection result can be understood as the anomaly score output by the decision tree model. The preset determination conditions can be set according to the actual application scenario; for example, the preset determination condition can be that the detection result is greater than a preset anomaly threshold.

[0150] Using the previous example, the data provider is a financial institution, and the data to be detected is financial data. Based on this, after receiving the financial data provided by the financial institution, the financial data can be input into a decision tree. The decision tree model can then perform data detection on the financial data and output the anomaly score for each piece of financial data.

[0151] If the anomaly score is greater than or equal to the preset anomaly threshold (40 points), the financial data can be identified as abnormal data. If the anomaly score is less than the preset anomaly threshold (40 points), the financial data can be identified as normal data. See details below. Figure 3 The process involves anomaly analysis to determine whether the data to be detected is abnormal. Simultaneously, based on... Figure 3 It can be seen that the preset determination condition can be a preset abnormal threshold or a preset abnormal interval.

[0152] Furthermore, in the embodiments provided in this specification, the data detection object includes at least two types of data detection units, and the at least two types of data detection units are arranged consecutively;

[0153] Accordingly, the step of inputting the data to be detected provided by the data provider into the data detection object to obtain the detection result of the data to be detected includes:

[0154] The data to be detected provided by the data provider is input into the data detection object;

[0155] The data to be detected is detected using the at least two data detection units in the data detection object to obtain the data detection type of the data to be detected. The input data of the subsequent data detection unit in the at least two data detection units is the detection pass data output by the corresponding preceding data detection unit in the at least two data detection units.

[0156] Based on the data detection type, a data score is determined for the data to be detected.

[0157] In this context, the data detection unit can be understood as a decision tree node in a decision tree; the preceding data detection unit can be understood as a data detection unit that has other connected data detection units following it; and the subsequent data detection unit can be understood as a data detection unit that has other connected data detection units preceding it. It should be noted that the preceding data detection unit corresponding to the subsequent data detection unit can be understood as the preceding data detection unit located before and connected to the subsequent data detection unit.

[0158] This data detection type can be understood as the risk type for abnormal data mentioned above. In practical applications, since the data detection rules are divided into corresponding rule types based on the risk type of abnormal data, the abnormal data detected based on a specific type of data detection rule will be identified as a corresponding risk class. For example, referring to Table 1 above, the rule "parsing time > 20 min" is classified as a strong rule based on the risk type "parsing outdated error data". Therefore, the abnormal data identified based on the rule "parsing time > 20 min" will be identified as abnormal data of the "parsing outdated error data" type.

[0159] In practical applications, the data processing method provided in this manual predetermines corresponding scores for different data detection types. Therefore, the data score determined for the data to be detected based on the data detection type can be understood as using the preset score corresponding to the data detection type as the data score corresponding to the data to be detected.

[0160] The data score can be understood as the anomaly score of the data to be detected.

[0161] Following the example above, see [link to example]. Figure 3 As can be seen, the data processing method provided in this specification can input the financial data provided by the bank (i.e., the data to be tested) into decision tree 1. Decision tree 1 first performs key field missing detection on the financial data through the highest priority root node (rule: missing key fields). Then, it inputs financial data that passes the detection and financial data that fails the detection. Among them, the financial data that passes the detection is input into the second decision tree node (rule: abnormal association relationship) for data detection; the financial data that fails the detection is assigned a corresponding data detection type, and the corresponding anomaly score is subsequently determined based on the data detection type. After the financial data has been detected by all nodes in decision tree 1, an anomaly score is determined for the financial data that finally passes the detection by all nodes, and a corresponding score is also determined for the financial data that fails the detection. Finally, the anomaly score of each financial data is output.

[0162] Subsequently, based on different practical needs, this anomaly score can be used to identify abnormal data from the financial data, or to identify normal data from the financial data.

[0163] The data processing method provided in this specification clusters the data detection rules corresponding to at least two data providers and constructs a data detection object for at least two data providers based on the similar data detection rules obtained after clustering. This avoids the problem of constructing multiple similar decision trees for multiple data providers, which would require significant time and manpower, thus saving time and manpower costs. Furthermore, by using the data to be detected provided by at least two data providers within this data detection object to determine the target data, the computational resources required to support data detection by this data detection object, as well as the maintenance costs of this data detection object, are reduced, avoiding the waste of computational resources and manpower costs.

[0164] The following is in conjunction with the appendix Figure 4 Taking the application of the data processing method provided in this specification in an abnormal data detection scenario as an example, the data processing method will be further explained. Figure 4 A flowchart illustrating the processing procedure of a data processing method according to an embodiment of this specification is shown, specifically including the following steps.

[0165] Step 402: Determine the data detection rules for multiple financial institutions.

[0166] The data detection rules can be rules obtained by accumulating expert experience.

[0167] Step 404: Cluster the data detection rules using a clustering algorithm to identify similar data detection rules.

[0168] Step 406: Determine the coverage of the similar data detection rule for the historical data to be detected.

[0169] Step 408: Sort the similar data detection rules from largest to smallest based on the coverage rate to obtain the sorting result of the similar data detection rules.

[0170] Step 410: Based on the ranking result, determine the priority of the similar data detection rule in the process of constructing the decision tree.

[0171] Step 412: Use the similar data detection rule as a decision tree node, and construct a decision tree based on the decision tree node and the priority.

[0172] Step 414: Train the constructed decision tree using training samples and sample labels, and automatically adjust the priority of decision tree nodes in the decision tree to obtain the trained decision tree.

[0173] Step 416: Obtain financial data from multiple financial institutions and input the financial data into the trained decision tree to obtain the anomaly scores corresponding to the financial data.

[0174] Step 418: Financial data with abnormal scores greater than a preset threshold are identified as abnormal data.

[0175] The data processing method provided in this manual, by leveraging expert experience and ensuring interpretability of the verification results, enables the detection of all known anomalies, thereby improving the efficiency of data detection. Furthermore, the decision tree constructed based on the accumulated expert experience enhances the efficiency and accuracy of data detection. With p rules, the decision tree reduces the time complexity from O(p) to O(logp), significantly improving detection efficiency for large-scale complex data.

[0176] Meanwhile, by constructing a decision tree using similar data detection rules across multiple financial institutions, the problem of needing to spend a significant amount of time and manpower to build multiple similar decision trees for multiple financial institutions is avoided, thus saving time and manpower costs. Furthermore, it reduces the computing resources required to support the data detection of this decision tree, as well as the maintenance costs of the decision tree, avoiding the waste of computing resources and manpower.

[0177] Corresponding to the above method embodiments, this specification also provides data processing apparatus embodiments. Figure 5 A schematic diagram of the structure of a data processing apparatus according to one embodiment of this specification is shown. Figure 5As shown, the device includes:

[0178] The determination module 502 is configured to determine at least two data providers and data detection rules for the at least two data providers;

[0179] Clustering module 504 is configured to perform clustering processing on the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers.

[0180] The detection module 506 is configured to construct a data detection object based on the similar data detection rules, and use the data detection object to detect the data to be detected provided by the at least two data providers, and determine the target data in the data to be detected.

[0181] Optionally, the clustering module 504 is further configured to:

[0182] A preset clustering algorithm is used to cluster the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers.

[0183] Optionally, the clustering module 504 is further configured to:

[0184] The similarity between the data detection rules of the at least two data providers is calculated based on a preset clustering algorithm to determine the similarity between the data detection rules.

[0185] Based on the similarity, the data detection rules are clustered to obtain similar data detection rules between the at least two data providers.

[0186] Optionally, the similar data detection rules are at least two;

[0187] Accordingly, the detection module 506 is further configured to:

[0188] At least two similar data detection rules are used as decision tree nodes, and a decision tree model is constructed based on the decision tree nodes.

[0189] Optionally, the detection module 506 is further configured to:

[0190] Determine the data detection ratio of at least two similar data detection rules, wherein the data detection ratio is determined by the detection results of detecting historical data to be detected using the data detection rules;

[0191] The at least two similar data detection rules are sorted based on the data detection ratio to obtain a rule sequence;

[0192] At least two similar data detection rules are used as decision tree nodes, and a decision tree model is constructed based on the decision tree nodes and the rule sequence.

[0193] Optionally, the detection module 506 is further configured to:

[0194] The at least two similar data detection rules are used as the first detection rule, and the initial data detection ratio of the at least two similar data detection rules for the historical data to be detected is used as the first data detection ratio.

[0195] Based on the first detection rule and the first data detection ratio, a corresponding data detection ratio is determined for the at least two similar data detection rules.

[0196] Optionally, the detection module 506 is further configured to:

[0197] Determine at least two similar data detection rules and an initial data detection ratio for historical data to be detected, wherein the initial data detection ratio is determined by the detection results of the historical data to be detected using the data detection rules;

[0198] The at least two similar data detection rules are used as the first detection rule, and the initial data detection ratio is used as the first data detection ratio.

[0199] Optionally, the detection module 506 is further configured to:

[0200] S1: Use the historical data to be detected as the first data;

[0201] S2: Determine the first data detection ratio of the first detection rule for the first data;

[0202] S3: The first data detection ratio that meets the preset rule determination conditions is determined as the target data detection ratio, and the first detection rule corresponding to the target data detection ratio is taken as the target detection rule;

[0203] S4: If it is determined that there are other detection rules besides the target detection rule in the first detection rules, the detection pass data corresponding to the target detection rule is determined from the first data based on the target data detection ratio;

[0204] S5: Take the other detection rules as the first detection rule, take the detection pass data as the first data, and continue to execute step S2 until there are no other detection rules besides the target detection rule in the first detection rule;

[0205] S6: Based on the target data detection ratio of the target detection rule, determine the data detection ratio of the at least two similar data detection rules.

[0206] Optionally, the data processing device further includes a training module, configured to:

[0207] Determine the training samples and training labels for the decision tree model;

[0208] The training samples are processed using the decision tree model to obtain sample detection results;

[0209] Based on the sample detection results and the training labels, determine the sample data detection ratio of the at least two similar data detection rules for the training samples;

[0210] The decision tree model is adjusted based on the detection ratio of the sample data to obtain the adjusted decision tree model.

[0211] Optionally, the detection module 506 is further configured to:

[0212] The data to be detected provided by the data provider is input into the data detection object to obtain the detection result of the data to be detected;

[0213] If the detection result meets the preset determination conditions, the data to be detected is determined as the target data.

[0214] Optionally, the data detection object includes at least two types of data detection units, and the at least two types of data detection units are arranged consecutively;

[0215] Accordingly, the detection module 506 is further configured to:

[0216] The data to be detected provided by the data provider is input into the data detection object;

[0217] The data to be detected is detected using the at least two data detection units in the data detection object to obtain the data detection type of the data to be detected. The input data of the subsequent data detection unit in the at least two data detection units is the detection pass data output by the corresponding preceding data detection unit in the at least two data detection units.

[0218] Based on the data detection type, a data score is determined for the data to be detected.

[0219] The data processing apparatus provided in this specification clusters the data detection rules corresponding to at least two data providers and constructs a data detection object for at least two data providers based on the similar data detection rules obtained after clustering. This avoids the problem of constructing multiple similar decision trees for multiple data providers, which would require significant time and manpower, thus saving time and manpower costs. Furthermore, by using the data to be detected provided by at least two data providers within this data detection object to determine the target data, the computational resources required to support the data detection object and the maintenance costs of the data detection object are reduced, avoiding the waste of computational resources and manpower costs.

[0220] The above is an illustrative scheme of a data processing apparatus according to this embodiment. It should be noted that the technical solution of this data processing apparatus and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the data processing apparatus, please refer to the description of the technical solution of the data processing method described above.

[0221] Figure 6 A structural block diagram of a computing device 600 according to one embodiment of this specification is shown. The components of the computing device 600 include, but are not limited to, a memory 610 and a processor 620. The processor 620 is connected to the memory 610 via a bus 630, and a database 650 is used to store data.

[0222] The computing device 600 also includes an access device 640, which enables the computing device 600 to communicate via one or more networks 660. Examples of these networks include a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), a Personal Area Network (PAN), or a combination of communication networks such as the Internet. The access device 640 may include one or more of any type of wired or wireless network interface (e.g., a Network Interface Card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) interface, a Wi-MAX interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0223] In one embodiment of this specification, the above-described components of the computing device 600 and Figure 6 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 6 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0224] The computing device 600 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or PCs. The computing device 600 can also be a mobile or stationary server.

[0225] The processor 620 is used to execute the following computer-executable instructions, which, when executed by the processor 620, implement the steps of the above-described data processing method.

[0226] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the data processing method described above.

[0227] An embodiment of this specification also provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the above-described data processing method.

[0228] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the data processing method described above belong to the same concept. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solution of the data processing method described above.

[0229] An embodiment of this specification also provides a computer program, wherein when the computer program is executed in a computer, it causes the computer to perform the steps of the above-described data processing method.

[0230] The above is an illustrative example of a computer program according to this embodiment. It should be noted that the technical solution of this computer program and the technical solution of the data processing method described above belong to the same concept. Details not described in detail in the technical solution of the computer program can be found in the description of the technical solution of the data processing method described above.

[0231] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0232] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drive, portable hard drive, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0233] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0234] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0235] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data processing method applied to anomaly data detection scenarios, the method comprising: Identify at least two data providers and data detection rules for the at least two data providers, wherein the data detection rules are described as text data; Clustering is performed on the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers; Based on the similar data detection rules, a unified data detection object is constructed for the at least two data providers, and the data detection object is used to detect the data to be detected provided by the at least two data providers to determine the target data in the data to be detected. The data detection object includes at least two consecutively arranged data detection units, and the input data of the subsequent data detection unit in the at least two data detection units is the detection passed data output by the preceding data detection unit.

2. The data processing method according to claim 1, wherein clustering the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers includes: A preset clustering algorithm is used to cluster the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers.

3. The data processing method according to claim 2, wherein the step of clustering the data detection rules of the at least two data providers using a preset clustering algorithm to obtain similar data detection rules between the at least two data providers includes: The similarity between the data detection rules of the at least two data providers is calculated based on a preset clustering algorithm to determine the similarity between the data detection rules. Based on the similarity, the data detection rules are clustered to obtain similar data detection rules between the at least two data providers.

4. The data processing method according to claim 1, wherein the similar data detection rules are at least two; Accordingly, constructing a unified data detection object for the at least two data providers based on the similar data detection rules includes: At least two similar data detection rules are used as decision tree nodes, and a decision tree model is constructed based on the decision tree nodes.

5. The data processing method according to claim 4, wherein the step of using at least two similar data detection rules as decision tree nodes and constructing a decision tree model based on the decision tree nodes includes: Determine the data detection ratio of at least two similar data detection rules, wherein the data detection ratio is determined by the detection results of detecting historical data to be detected using the data detection rules; The at least two similar data detection rules are sorted based on the data detection ratio to obtain a rule sequence; At least two similar data detection rules are used as decision tree nodes, and a decision tree model is constructed based on the decision tree nodes and the rule sequence.

6. The data processing method according to claim 5, wherein determining the data detection ratio of at least two similar data detection rules includes: The at least two similar data detection rules are used as the first detection rule, and the initial data detection ratio of the at least two similar data detection rules for the historical data to be detected is used as the first data detection ratio. Based on the first detection rule and the first data detection ratio, a corresponding data detection ratio is determined for the at least two similar data detection rules.

7. The data processing method according to claim 6, wherein the step of using the at least two similar data detection rules as the first detection rule, and using the initial data detection ratio of the at least two similar data detection rules for historical data to be detected as the first data detection ratio, comprises: Determine at least two similar data detection rules and an initial data detection ratio for historical data to be detected, wherein the initial data detection ratio is determined by the detection results of the historical data to be detected using the data detection rules; The at least two similar data detection rules are used as the first detection rule, and the initial data detection ratio is used as the first data detection ratio.

8. The data processing method according to claim 6, wherein determining the corresponding data detection ratio for the at least two similar data detection rules based on the first detection rule and the first data detection ratio includes: S1: Use the historical data to be detected as the first data; S2: Determine the first data detection ratio of the first detection rule for the first data; S3: The first data detection ratio that meets the preset rule determination conditions is determined as the target data detection ratio, and the first detection rule corresponding to the target data detection ratio is taken as the target detection rule; S4: If it is determined that there are other detection rules besides the target detection rule in the first detection rules, the detection pass data corresponding to the target detection rule is determined from the first data based on the target data detection ratio; S5: Take the other detection rules as the first detection rule, take the detection pass data as the first data, and continue to execute step S2 until there are no other detection rules besides the target detection rule in the first detection rule; S6: Based on the target data detection ratio of the target detection rule, determine the data detection ratio of the at least two similar data detection rules.

9. The data processing method according to claim 4, further comprising, after taking at least two similar data detection rules as decision tree nodes and constructing a decision tree model based on the decision tree nodes: Determine the training samples and training labels for the decision tree model; The training samples are processed using the decision tree model to obtain sample detection results; Based on the sample detection results and the training labels, determine the sample data detection ratio of the at least two similar data detection rules for the training samples; The decision tree model is adjusted based on the detection ratio of the sample data to obtain the adjusted decision tree model.

10. The data processing method according to claim 1, wherein the step of using the data detection object to detect the data to be detected provided by the at least two data providers, and determining the target data in the data to be detected, comprises: The data to be detected provided by the data provider is input into the data detection object to obtain the detection result of the data to be detected; If the detection result meets the preset determination conditions, the data to be detected is determined as the target data.

11. The data processing method according to claim 10, wherein the data detection object includes at least two types of data detection units, and the at least two types of data detection units are arranged consecutively; Accordingly, the step of inputting the data to be detected provided by the data provider into the data detection object to obtain the detection result of the data to be detected includes: The data to be detected provided by the data provider is input into the data detection object; The data to be detected is detected using at least two data detection units in the data detection object to obtain the data detection type of the data to be detected; Based on the data detection type, a data score is determined for the data to be detected.

12. A data processing apparatus, applied in an abnormal data detection scenario, the apparatus comprising: The determination module is configured to determine at least two data providers and data detection rules for the at least two data providers, wherein the data detection rules are described as text data; The clustering module is configured to perform clustering processing on the data detection rules of the at least two data providers to obtain similar data detection rules between the at least two data providers. The detection module is configured to construct a unified data detection object for the at least two data providers based on the similar data detection rules, and to use the data detection object to detect the data to be detected provided by the at least two data providers to determine the target data in the data to be detected. The data detection object includes at least two consecutively arranged data detection units, and the input data of the subsequent data detection unit in the at least two data detection units is the detection passed data output by the preceding data detection unit.

13. A computing device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions, which, when executed by the processor, implement the steps of the data processing method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing computer-executable instructions that, when executed by a processor, implement the steps of the data processing method according to any one of claims 1 to 11.

Images