Data processing method, device and electronic equipment

Abstract

The present application provides a data processing method, apparatus, and electronic device. The method comprises: determining, based on target data, data of multiple attribute characteristics included in the target data, where the target data belongs to a specified type of data that can be represented in a two-dimensional table format; mining associations between data of different attribute characteristics in the target data to obtain at least one set of association relationship rules, each set of association relationship rules including association rules that must be satisfied by data within at least two attribute characteristics; wherein the association relationship rules are used to determine abnormal data present in the specified type of data. The solution of the basic application can achieve the determination of abnormal data in data.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data processing method, device and electronic device. Background Art

[0002] With the advent of the big data era, daily life and work often involve a large amount of data that needs to be processed.

[0003] In many cases, there will be some abnormal data in the acquired data, and the abnormal data will affect the accuracy of data analysis. Therefore, how to determine the abnormal data in the data is a technical problem that technicians in this field need to solve. Summary of the Invention

[0004] The present application provides a data processing method, device and electronic device.

[0005] A data processing method includes:

[0006] Determining, based on target data, data of a plurality of attribute features included in the target data, wherein the target data belongs to a specified type of data that can be represented in a two-dimensional table form;

[0007] Mining association relationships between data with different attribute characteristics in the target data to obtain at least one set of association relationship rules, each set of association relationship rules including: association rules that must be satisfied by data within at least two attribute characteristics;

[0008] The association relationship rule is used to determine abnormal data in the specified type of data.

[0009] In a possible implementation, mining the association relationship between data with different attribute features in the target data to obtain at least one set of association relationship rules includes:

[0010] According to the set confidence level, the association relationship between different attribute features in the target data is mined using a data mining algorithm to obtain at least one set of association relationship rules.

[0011] In another possible implementation, the association relationship between different attribute features in the target data is mined according to a set confidence level and using a data mining algorithm to obtain at least one set of association relationship rules, including:

[0012] Performing frequent item set mining on each group of data in the target data using a data mining algorithm to obtain multiple frequent item sets that meet a set confidence level, where each group of data corresponds to a row of data in a two-dimensional table converted from the target data;

[0013] The data mining algorithm is used to analyze the association relationships between different attribute features in the multiple frequent item sets to obtain at least one set of association relationship rules.

[0014] In another possible implementation, mining the association relationship between data with different attribute features in the target data to obtain at least one set of association relationship rules includes:

[0015] For each attribute feature combination in the target data, the probability of data in the same group being equal under the attribute feature combination is calculated, wherein the target data includes at least one attribute feature combination, the attribute feature combination includes two attribute features in the target data, and each group of data corresponding to the target data is converted into a row of data in a two-dimensional table;

[0016] Determine at least one target attribute feature combination whose equality probability is higher than a probability threshold, and obtain an equality rule corresponding to the target attribute feature combination, wherein the equality rule of the target attribute feature combination represents that data belonging to the same group in two attributes within the attribute feature combination are equal.

[0017] In yet another possible implementation, the statistically calculating the probability of data in the same group being equal within each attribute feature combination in the target data includes:

[0018] Determining the correlation between any two attribute features in the target data based on the data under each attribute feature in the target data;

[0019] Determining at least one candidate attribute feature combination having a correlation higher than a correlation threshold, wherein the candidate attribute feature combination includes two attribute features having a correlation higher than the correlation threshold;

[0020] The probability of data being equal in the same group within each candidate attribute feature combination is respectively counted.

[0021] In yet another possible implementation, before determining the correlation between any two attribute features in the target data based on the data of each attribute feature in the target data, the method further includes:

[0022] Determine that the data in the target data is an attribute feature of non-numeric data, and convert the non-numeric data in the attribute feature of the non-numeric data into numeric data.

[0023] In yet another possible implementation, the method further includes:

[0024] According to the at least one set of association relationship rules, abnormal data existing in the data to be detected belonging to the specified type of data is determined, and the attribute characteristics included in the data to be detected are the same as those included in the target data.

[0025] A data processing device includes:

[0026] a data determining unit, configured to determine, based on target data, data of a plurality of attribute features included in the target data, wherein the target data belongs to a specified type of data that can be represented in a two-dimensional table form;

[0027] A data mining unit is used to mine the association relationship between data of different attribute characteristics in the target data to obtain at least one set of association relationship rules, each set of association relationship rules including: association rules that need to be satisfied by data within at least two attribute characteristics; wherein the association relationship rules are used to determine abnormal data existing in the specified type of data.

[0028] In a possible implementation, the data mining unit includes:

[0029] The algorithm mining unit is used to mine the association relationship between different attribute features in the target data according to the set confidence level and using the data mining algorithm to obtain at least one set of association relationship rules.

[0030] Among them, an electronic device includes: a processor and a memory;

[0031] Wherein, the processor is used to execute any one of the data processing methods described above;

[0032] The memory is used to store programs required by the processor to perform operations.

[0033] As can be seen from the above, this application mines the associations between data of different attribute characteristics in the target data and determines at least one set of association rules present in the target data. Because each set of association rules includes association rules that must be satisfied by data within at least two attribute characteristics, and because the association rules are obtained by mining target data belonging to a specified type of data, the association rules are universal for the specified type of data. Therefore, based on the association rules, abnormal data present in the specified type of data can be analyzed and detected. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without any creative work.

[0035] Figure 1 A schematic diagram of a data processing method provided in an embodiment of the present application is shown;

[0036] Figure 2 Another flow chart of a data processing method provided in an embodiment of the present application is shown;

[0037] Figure 3 Another flow chart of a data processing method provided in an embodiment of the present application is shown;

[0038] Figure 4 Another flow chart of a data processing method provided in an embodiment of the present application is shown;

[0039] Figure 5 Another flow chart of a data processing method provided in an embodiment of the present application is shown;

[0040] Figure 6 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown;

[0041] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0042] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0043] like Figure 1 , which shows a flow chart of an embodiment of a data processing method of the present application. The method of this embodiment may include:

[0044] S101, determining a plurality of attribute feature data included in the target data based on the target data.

[0045] The target data is a specified type of data that can be represented in a two-dimensional table format. That is, the target data can be in the form of a two-dimensional table, or can be data presented in other formats but can be converted into a two-dimensional table.

[0046] For example, the target data is a table of data; or it may be structured data or unstructured data that can be stored in a table format.

[0047] The designated type data is used to represent the data category to which the target data belongs. The designated type may specify the number and type of attribute features contained in the data. For example, the designated type data may be convertible into a two-dimensional table and include data with designated attribute features.

[0048] For example, the designated data type may be access record data generated by the same website in different actual time periods. Since the access record data generated at different times contain the same fields, the different access data records all belong to the same type of table data.

[0049] For another example, data belonging to a specified type of data may be student management data containing the same fields.

[0050] The attribute feature is a data representation of the target data in an attribute dimension. Specifically, the attribute feature in the target data corresponds to a field in the target data when the target data is represented by a two-dimensional table.

[0051] It can be understood that the target data may include multiple pieces of data, each piece of data corresponding to a row of records in the two-dimensional table converted from the target data.

[0052] S102: mining association relationships between data with different attribute features in the target data to obtain at least one set of association relationship rules.

[0053] Each set of association relationship rules includes: association rules that data within at least two attribute features must satisfy. It is understandable that the association rules corresponding to at least two attribute features are actually association rules determined based on the association relationship between the at least two attribute features.

[0054] It is understood that the association rule between the at least two attribute features can be a data relationship rule or a logical relationship rule satisfied between the at least two attribute features. It is determined that each set of association relationship rules is applicable to any set of data in the target data, i.e., the at least two attribute features in each set of data satisfy the association rule.

[0055] The target data may include multiple groups of data (also referred to as multiple pieces of data), and each group of data (each piece of data) corresponds to a row of data in the two-dimensional table converted from the target data.

[0056] For example, if the association rule is that the feature values ​​of at least two attribute features are the same, then for any set of data in the target data, the feature values ​​of at least two attribute features in the set of data should be the same.

[0057] For another example, the association rule may be that the sum of attribute feature A and attribute feature B is equal to attribute feature C. Accordingly, for any set of data, the sum of the values ​​of attribute feature A and attribute feature B in the set of data should be equal to the value of attribute feature C.

[0058] For another example, the association relationship rule may be: when attribute feature 1 is equal to S1 and attribute feature 2 is equal to S2, attribute feature 3 is equal to S3.

[0059] It is understandable that since the target data belongs to a specified type of data, mining the association relationships of the target data actually mines the association relationship rules that must be satisfied between different attribute features in the specified type of data. Based on this, the association relationship rules determined in this application can be used to identify abnormal data in the specified type of data.

[0060] Furthermore, the present application may also determine abnormal data present in the data to be detected that belongs to a specified type of data based on at least one set of association relationship rules, wherein the data to be detected has the same attribute characteristics as the target data. The data to be detected may be the target data or a piece of data other than the target data.

[0061] For example, based on the at least one set of association rules, it is possible to detect whether there is data in the target data that does not satisfy the association rules. Data in the target data that does not satisfy the association rules is considered abnormal data. For example, suppose the association rule mined is that the value of attribute feature A and the value of attribute feature B should be equal, but the values ​​of these two attribute features in a certain data item in the target data are not equal, then the data of these two attribute features in this data item is abnormal data.

[0062] For another example, abnormal data detection may be performed on a piece of data other than the target data and belonging to the same specified type of data based on the at least one set of association rules.

[0063] As can be seen from the above, this application mines the associations between data of different attribute characteristics in the target data and determines at least one set of association rules present in the target data. Because each set of association rules includes association rules that must be satisfied by data within at least two attribute characteristics, and because the association rules are obtained by mining target data belonging to a specified type of data, the association rules are universal for the specified type of data. Therefore, based on the association rules, abnormal data present in the specified type of data can be analyzed and detected.

[0064] It is understandable that there are many possible specific ways to mine the association relationship rules existing in the target data. The following describes several cases as examples.

[0065] In one possible scenario, the present application may mine the association relationships between different attribute features in the target data according to a set confidence level and utilize a data mining algorithm to obtain at least one set of association relationship rules.

[0066] The confidence level is a threshold value, which can be set as needed.

[0067] The data mining algorithm may be any association rule mining algorithm that can mine implicit relationships in data, such as the Apriori algorithm or the FPGrowth algorithm.

[0068] In response to this possible situation, a specific implementation method is introduced below.

[0069] like Figure 2 , which shows a flow chart of another embodiment of a data processing method of the present application. The method of this embodiment may include:

[0070] S201: Determine, based on target data, data of a plurality of attribute features included in the target data.

[0071] This step can be referred to the relevant introduction of the previous embodiment and will not be repeated here.

[0072] S202: Perform frequent item set mining on each group of data in the target data using a data mining algorithm to obtain a plurality of frequent item sets that meet a set confidence level.

[0073] As mentioned above, each set of data corresponds to a row of data in the two-dimensional table converted from the target data.

[0074] The definition of a frequent itemset is: there are a series of sets that have some common elements. The elements that appear frequently in the sets form a subset that meets a certain threshold condition (i.e., confidence level). In this application, a frequent itemset refers to a set of data items that appear in multiple groups of target data a number of times that meets the confidence level.

[0075] The confidence level reflects the probability of another data appearing after one data appears.

[0076] The specific implementation process of mining frequent itemsets may vary depending on the data mining algorithm used, and this application does not impose any restrictions on this.

[0077] It can be understood that before frequent item set mining, in order to reflect the attribute characteristics of each data item in the target data and avoid interference in frequent item set mining due to the same numerical values ​​under different attribute characteristics, this application will convert each data item in the target data into a data item with attribute characteristics. Specifically, the data item can be converted into the attribute characteristics + data item to which the data item belongs.

[0078] Each data item is a data value under an attribute feature in the target data.

[0079] For example, if the target data is a two-dimensional table, and the data under field T in a row of the two-dimensional table is aaa, then the data needs to be converted to field Taaa.

[0080] S203: Analyze the association relationships between different attribute features in the multiple frequent item sets using a data mining algorithm to obtain at least one set of association relationship rules.

[0081] Each set of association relationship rules includes: association rules that data within at least two attribute features must satisfy. The association relationship rules are used to determine abnormal data in the specified type of data.

[0082] Since the data mining algorithm can also be called an association mining algorithm, by analyzing the mined frequent item sets through the data mining algorithm, the association relationships between data items with different attribute characteristics in the frequent item sets can be determined, and the association relationships between different attribute characteristics can be finally obtained.

[0083] It can be understood that since data mining algorithms are relatively mature and can more accurately and comprehensively analyze the associations between different attribute features in a piece of data, this embodiment uses data mining algorithms to mine the association relationship rules existing in the target data, which can more comprehensively, accurately and efficiently mine the association relationship rules existing in the target data, thereby facilitating a more comprehensive and accurate analysis of abnormal data existing in the specified type of data based on the mined association relationship rules.

[0084] To facilitate understanding, the following uses a two-dimensional table as an example to illustrate the process of using a data mining algorithm to determine the association relationship rules between different fields in the two-dimensional table.

[0085] like Figure 3 , which shows a flow chart of an embodiment of a data processing method of the present application. The method of this embodiment may include:

[0086] S301: Obtain a two-dimensional table and determine the data of each row and each field of the two-dimensional table.

[0087] The two-dimensional table is a two-dimensional table of specified type of data, for example, the two-dimensional table may be a student family information record table.

[0088] S302 : For each data item in the two-dimensional table, convert the data item into data obtained by concatenating fields and data items to obtain a converted two-dimensional table.

[0089] Each data item in a two-dimensional table is a piece of data uniquely identified by its row and column. For example, the data in the first row and first column of a two-dimensional table is a data item, and the data in the first row and second column of a two-dimensional table is also a data item.

[0090] The data obtained by concatenating the fields and data items can reflect both the specific value of the data and the field to which the data belongs. This conversion can prevent data items in different fields from being the same and being considered as frequently occurring data.

[0091] For example, if the second row in the two-dimensional table has a data item of 100 in field S, it can be converted to field S100.

[0092] S303 , performing frequent item set mining on each row of data in the converted two-dimensional table using a data mining algorithm according to the set confidence level, and obtaining a plurality of frequent item sets that meet the confidence level.

[0093] It is understandable that since the data in the two-dimensional table has been converted, in the process of frequent item set mining for each row of data in the converted two-dimensional table, only data with the same fields and data items and frequently appearing can be identified as frequent items in the frequent item set.

[0094] S304: Analyze association relationships between different fields in the multiple frequent item sets using the data mining algorithm to obtain at least one set of field association rules.

[0095] Each group of fields may include at least two fields. Therefore, the association relationship rule of each group of fields is the association relationship rule satisfied by the data in at least two fields in the group of fields.

[0096] For example, the association relationship rule of a group of fields may be: when field S1 = B1 and field S3 = V1, field S6 = F1.

[0097] It is understandable that Figure 3 The target data is a two-dimensional table. It is understandable that if the target data is other data that can be converted into a two-dimensional table, then the target data can be converted into a two-dimensional table and then used. Figure 3 The method of the embodiment is used to mine association relationship rules.

[0098] In this embodiment, the mined association rules are also used to determine abnormal data in the two-dimensional table and other two-dimensional tables of the same specified data type. The other two-dimensional tables of the same type as the two-dimensional table all include the same fields.

[0099] For example, a two-dimensional table is a student performance table for Class 1, Grade 2. This table has fields such as student name, year and month of appearance, age, math score, Chinese score, and total score. Suppose, then, that the association rules mined from this student performance table include the following: If the date of birth is #year#month, then the age is 10 years old.

[0100] Combining the above two association rules, if a student's date of birth is #year#month, but his / her age is not equal to 10 years old, it means that there is an anomaly in the data of the student's age field in the student table.

[0101] Similarly, if the student report card for Class 2, Grade 2 has the same data type as the report card above, and also has fields such as student name, year and month of appearance, age, math score, Chinese score, and total score, then this association rule can be used to perform anomaly detection on the student report card for Class 2, Grade 2 to detect abnormal data where age does not match date of birth.

[0102] The following combination Figure 4 Another possible scenario of mining the association relationship between data with different attribute features in the target data in this application is introduced.

[0103] like Figure 4 , which shows another flow chart of a data processing method of the present application. The method of this embodiment may include:

[0104] S401: Determine, based on target data, data of multiple attribute features included in the target data.

[0105] The target data is of a specified type that can be represented in a two-dimensional table.

[0106] This step can be referred to the relevant introduction of the previous embodiment and will not be repeated here.

[0107] S402 , for each attribute feature combination in the target data, calculating the probability of data in the same group being equal under the attribute feature combination.

[0108] The target data includes at least one attribute feature combination, and the attribute feature combination includes two attribute features in the target data.

[0109] As mentioned above, each set of data (also referred to as each piece of data) corresponds to the target data and is converted into a row of data in a two-dimensional table.

[0110] It is understood that for any set of data, including data belonging to multiple different attribute characteristics of the target data, and for the same attribute characteristic combination, the data in the same group under the attribute characteristic combination is actually the data in the set of data belonging to each attribute characteristic corresponding to the attribute characteristic combination.

[0111] For example, assuming that the attribute feature combination includes attribute feature S1 and attribute feature S2, then for the data of each group, it can be determined that the data of attribute feature S1 and the data of attribute feature S2 in the group are the data belonging to the same group under the attribute feature combination.

[0112] In order to facilitate distinction, the probability that data in the same group are equal under the attribute feature combination is called equal probability.

[0113] The equal probability corresponding to an attribute feature combination reflects the probability of the occurrence of groups with the same data under this data feature combination in the target data, and this probability is the ratio of the groups with equal data of each attribute feature in this attribute feature combination to the total number of all groups in the target data.

[0114] Using the example of the attribute feature combination S1 and S2, we can first check whether the data for attribute feature S1 and attribute feature S2 within each group are identical. Finally, we can count the number of identical groups across all groups where the data for these two attributes are identical. Combining this number of identical groups with the total number of groups in the target data table can determine the probability that the data for these two attributes within the same group are identical, i.e., the probability of equality.

[0115] It is understandable that if the target data contains a large amount of data or contains many types of attribute features, then analyzing the equal probabilities corresponding to each combination of attribute features one by one will inevitably lead to excessive data processing.

[0116] In order to reduce the amount of data processing, the present application may also determine the correlation between any two attribute features in the target data based on the data under each attribute feature in the target data.

[0117] The correlation between two attribute features is the correlation calculated based on the data of the two attribute features.

[0118] Based on this, at least one candidate attribute feature combination with a correlation greater than a correlation threshold can be determined. The candidate attribute feature combination includes two attribute features with a correlation greater than the correlation threshold. Accordingly, the probability of equality of data in the same row within each candidate attribute feature combination can be calculated.

[0119] It can be understood that the main association relationship rule determined in this application is the equality rule between two attribute features. Since two attribute features with a correlation higher than the correlation threshold may be equal, this application only needs to analyze the equality probability corresponding to the attribute feature combination composed of two attribute features with a correlation higher than the correlation threshold.

[0120] The correlation threshold can be set as needed and is not limited thereto.

[0121] Optionally, considering that some data in the target data may be in non-numeric form, such as a string, to facilitate calculation of correlations between different attribute features, the present application may also determine that the data in the target data is an attribute feature of non-numeric data and convert the non-numeric data within the attribute feature of the non-numeric data into numeric data. This will be explained in detail later in conjunction with the case where the target data is a two-dimensional table and will not be repeated here.

[0122] S403: Determine at least one target attribute feature combination whose equality probability is higher than a probability threshold, and obtain an equality rule corresponding to the target attribute feature combination.

[0123] The probability threshold can be set as needed, for example, the probability threshold can be 90%, etc., and there is no limitation on this.

[0124] In order to facilitate distinction, the corresponding attribute features with equal probability higher than the probability threshold are combined into the target attribute feature combination.

[0125] The equality rule of the target attribute feature combination indicates that the data belonging to the same group in two attribute features within the attribute feature combination are equal.

[0126] It can be understood that for a combination of attribute features, if the corresponding probability of equality is the proportion of groups with the same attribute features within the combination, the higher the proportion, the higher the probability that the data in the same group under the two attribute features are equal. Based on this, when the corresponding probability of equality for the attribute feature combination exceeds the set probability threshold, it can be determined that the data in the same group under the two attribute features within the attribute feature combination are equal.

[0127] For example, assume the probability threshold is 90%, and the target data includes 100 data items. If the data in Field 1 and Field 2 are equal in 92 of the 100 data items, then the probability of equality for the field combination of Field 1 and Field 2 is 92%. This equality probability exceeds the probability threshold, and the data in Field 1 and Field 2 in any data item in the target data can be considered equal.

[0128] It is understood that after determining the equality rule for the target attribute feature combination, abnormal data in the data to be tested can be detected based on the equality rule for the target attribute feature combination. For example, if two data under two attribute features corresponding to the same group and target attribute combination in the data to be tested are not equal, it indicates that the data under these two attribute features in the group in the data to be tested is abnormal.

[0129] For ease of understanding Figure 4 In the embodiment, the process of mining the equality rules in the two-dimensional table is described below by taking the target data as a two-dimensional table as an example.

[0130] like Figure 5 , which shows a flow chart of an embodiment of a data processing method of the present application. The method of this embodiment may include:

[0131] S501, obtain a two-dimensional table.

[0132] The two-dimensional table includes multiple rows and columns of data, wherein each column in the two-dimensional table corresponds to a field.

[0133] S502: Determine a field in the two-dimensional table that contains non-numeric data, and convert the non-numeric data in the field into numeric data.

[0134] The non-numeric data may be a string of data or other numerical values ​​that are not represented by numbers.

[0135] In this application, the purpose of converting non-numeric data into numeric data is only to facilitate the subsequent calculation of the correlation between two fields.

[0136] There are many ways to convert non-numeric data in a field into numeric data, as long as it is ensured that the data in the field can still be distinguished.

[0137] For example, for a field containing non-numeric data, a unique identification value representing the value in the field can be assigned to each value according to the type of the value contained in the field.

[0138] To facilitate understanding, the following uses an example of a two-dimensional table to illustrate the process of converting each data under the field of non-numeric data in the two-dimensional table.

[0139] Table 1 is a two-dimensional table.

[0140] Table 1

[0141]

[0142]

[0143] The first column in Table 1 is an identification number for each record. For example, Table 1 contains 17 rows of data, that is, 17 records, and these 17 records are marked as record 1 to record 17 in sequence.

[0144] Meanwhile, Table 1 includes 8 fields, such as SystemA.Column1 and SystemA.Column2, etc. The data in the column where each field is located is the data under the field.

[0145] As can be seen from Table 1, the data under each field in Table 1 are in character format, not numeric data. Therefore, data conversion needs to be performed on the data under each field separately.

[0146] On this basis, for each field, different types of data under this field are numbered in sequence so that each type of data under this field corresponds to a code.

[0147] Correspondingly, according to the relationship between each type of data and the code under the field, the data in each row under the field is converted into the code corresponding to the data.

[0148] The two-dimensional table converted from Table 1 can be shown in Table 2.

[0149] Table 2

[0150]

[0151] Comparing Tables 1 and 2, we can see that for each field, a value in Table 2 represents the original data in that field. For example, in the field "SystemA.Column1," which contains only two types of data, B1 and B2, each B1 in that field can be converted to the number 0, while each B2 can be converted to the number 1. The same applies to other fields and will not be repeated here.

[0152] Of course, Table 1 and Table 2 are merely examples of two-dimensional table conversion. Other methods of converting non-numeric data into numeric data are also applicable to this embodiment and are not limited thereto.

[0153] S503: Determine the correlation between any two fields in the two-dimensional table based on the data of each attribute feature in the two-dimensional table.

[0154] For example, the data under each field can form a matrix. For any two fields, the correlation between the matrices corresponding to the two fields can be calculated to obtain the correlation between the two fields.

[0155] S504: Determine at least one candidate field combination whose relevance is higher than a relevance threshold.

[0156] Each candidate field combination includes two fields whose correlation is higher than a correlation threshold.

[0157] S505 , respectively counting the probability of data in the same row being equal within each candidate field combination.

[0158] For example, taking Table 2 as an example, we can count which rows in SystemA.Column1 and SystemA.Column2 have equal data, obtain the number of identical rows in these two fields, and then calculate the ratio of identical data to the total number of rows to obtain the probability of equality.

[0159] S506: Determine at least one target field combination having an equal probability higher than a probability threshold from the at least one candidate field combination, and obtain an equal rule corresponding to the target field combination.

[0160] The equality rule of the target field combination indicates that the data in the two fields within the target field combination and belonging to the same row are equal.

[0161] For example, after the above analysis in Table 2, it can be found that the data in the same row of any two fields among SystemA.Column1, SystemB.Column2, and SystemC.Column1 are the same.

[0162] After obtaining the equality rule, anomaly detection can be performed on the two-dimensional table to be detected. If the data in a row under two fields in the target field combination in the two-dimensional table to be detected are not equal, it means that there is an anomaly in the data in the row under these two fields.

[0163] The two-dimensional table to be detected may be the two-dimensional table from which the equal rule is mined, or a two-dimensional table of the same type as the two-dimensional table.

[0164] For example, assuming that the equality rules corresponding to multiple target field combinations can obtain SystemA.Column1=SystemB.Column2=SystemC.Column1, then combining this rule to check Table 1, it can be seen that in Table 1, the data "Value9" in the 10th row of Table 1 and belonging to the SystemB.Column2 field is not equal to the data of the SystemA.Column1 and SystemC.Column1 fields in the 10th row, then the data "Value9" in the 10th row and belonging to the SystemB.Column2 field is abnormal data.

[0165] Corresponding to a data processing method of the present application, the present application also provides a data processing device.

[0166] like Figure 6 FIG. 1 shows a schematic diagram of a structure of a data processing device, which may include:

[0167] The data determining unit 601 is configured to determine, based on target data, data of a plurality of attribute features included in the target data, wherein the target data belongs to a specified type of data that can be represented in a two-dimensional table format;

[0168] The data mining unit 602 is used to mine the association relationship between data of different attribute characteristics in the target data to obtain at least one set of association relationship rules, each set of association relationship rules including: association rules that must be satisfied by data within at least two attribute characteristics; wherein the association relationship rules are used to determine abnormal data existing in the specified type of data.

[0169] In a possible implementation, the data mining unit includes:

[0170] The algorithm mining unit is used to mine the association relationship between different attribute features in the target data according to the set confidence level and using the data mining algorithm to obtain at least one set of association relationship rules.

[0171] In another possible implementation, the algorithm mining unit includes:

[0172] An itemset mining unit is used to perform frequent itemset mining on each group of data in the target data using a data mining algorithm to obtain multiple frequent itemsets that meet a set confidence level, where each group of data corresponds to a row of data in the two-dimensional table converted from the target data;

[0173] The rule analysis unit is used to analyze the association relationship between different attribute features in the multiple frequent item sets using the data mining algorithm to obtain at least one set of association relationship rules.

[0174] In another possible implementation of the present application, the data mining unit includes:

[0175] a probability determination unit, configured to calculate, for each attribute feature combination in target data, a statistical probability that data in the same group under the attribute feature combination are equal, wherein the target data includes at least one attribute feature combination, the attribute feature combination includes two attribute features in the target data, and each group of data corresponding to the target data is converted into a row of data in a two-dimensional table;

[0176] A rule determination unit is used to determine at least one target attribute feature combination whose equality probability is higher than a probability threshold, and obtain an equality rule corresponding to the target attribute feature combination. The equality rule of the target attribute feature combination represents that the data belonging to the same group in two attributes within the attribute feature combination are equal.

[0177] In a possible implementation, the probability determination unit includes:

[0178] a correlation calculation unit, configured to determine, for each attribute feature combination in the target data, a correlation between any two attribute features in the target data based on data associated with each attribute feature in the target data;

[0179] a combination determining unit, configured to determine at least one candidate attribute feature combination having a correlation higher than a correlation threshold, wherein the candidate attribute feature combination includes two attribute features having a correlation higher than the correlation threshold;

[0180] The probability statistics unit is used to respectively count the equal probabilities of the data in the same group within each candidate attribute feature combination being equal.

[0181] In an optional manner, the device further includes:

[0182] The data conversion unit is configured to, before the correlation calculation unit determines the correlation between the two attribute features in the target data, further include:

[0183] Determine that the data in the target data is an attribute feature of non-numeric data, and convert the non-numeric data in the attribute feature of the non-numeric data into numeric data.

[0184] In any one of the above embodiments of the device of the present application, the device further comprises:

[0185] The anomaly detection unit is used to determine abnormal data existing in the data to be detected belonging to the specified type of data based on the at least one group of association relationship rules, and the attribute characteristics contained in the data to be detected are the same as those of the target data.

[0186] On the other hand, the present application also provides an electronic device, such as Figure 7 , which shows a schematic diagram of the composition structure of the electronic device, the electronic device can be any type of electronic device, and the electronic device at least includes a memory 701 and a processor 702;

[0187] The processor 701 is configured to execute the data processing method in any one of the above embodiments.

[0188] The memory 702 is used to store programs required by the processor to perform operations.

[0189] It is understandable that the electronic device may further include a display unit 703 and an input unit 704 .

[0190] Of course, the electronic device may also have Figure 7 There is no limitation to more or fewer components.

[0191] On the other hand, the present application also provides a computer-readable storage medium, which stores at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the data processing method described in any one of the above embodiments.

[0192] The present application also proposes a computer program, which includes computer instructions stored in a computer-readable storage medium. When the computer program is run on an electronic device, it is used to execute the data processing method in any of the above embodiments.

[0193] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other. At the same time, the features described in the various embodiments in this specification can be replaced or combined with each other, so that professionals in this field can implement or use this application. For device embodiments, since they are basically similar to method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0194] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0195] The above description of the disclosed embodiments will enable one skilled in the art to implement or use the present application. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application is not limited to the embodiments shown herein, but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

[0196] The above is only a preferred embodiment of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A data processing method, comprising: Determining, based on target data, data of a plurality of attribute features included in the target data, wherein the target data belongs to a specified type of data that can be represented in a two-dimensional table form; mining association relationships between data of different attribute characteristics in the target data to obtain at least one set of association relationship rules, each set of association relationship rules including association rules that must be satisfied by data within at least two attribute characteristics, and at least one set of association relationship rules being determined based on the association relationship between the at least two attribute characteristics; wherein the association relationship rules are used to determine abnormal data present in the data of the specified type; The mining of association relationships between data with different attribute features in the target data to obtain at least one set of association relationship rules includes: For each attribute feature combination in the target data, the probability of data in the same group being equal under the attribute feature combination is calculated, wherein the target data includes at least one attribute feature combination, the attribute feature combination includes two attribute features in the target data, and each group of data corresponding to the target data is converted into a row of data in a two-dimensional table; Determine at least one target attribute feature combination whose equality probability is higher than the probability threshold, and obtain the equality rule corresponding to the target attribute feature combination, the equality rule of the target attribute feature combination representing that the data belonging to the same group in the two attributes within the attribute feature combination are equal; if two data belonging to the same group in the data to be detected and under the two attribute features corresponding to the target attribute feature combination are not equal, determine that the data under the two attribute features in the group in the data to be detected are abnormal.

2. The method according to claim 1, wherein mining the association relationship between data with different attribute characteristics in the target data to obtain at least one set of association relationship rules further comprises: According to the set confidence level, the association relationship between different attribute features in the target data is mined using a data mining algorithm to obtain at least one set of association relationship rules.

3. The method according to claim 2, wherein the association relationship between different attribute features in the target data is mined using a data mining algorithm according to a set confidence level to obtain at least one set of association relationship rules, including: Performing frequent item set mining on each group of data in the target data using a data mining algorithm to obtain multiple frequent item sets that meet a set confidence level, where each group of data corresponds to a row of data in a two-dimensional table converted from the target data; The data mining algorithm is used to analyze the association relationships between different attribute features in the multiple frequent item sets to obtain at least one set of association relationship rules.

4. The method according to claim 1, wherein the statistically calculating the probability of data in the same group being equal within each attribute feature combination in the target data comprises: Determining the correlation between any two attribute features in the target data based on the data under each attribute feature in the target data; Determining at least one candidate attribute feature combination having a correlation higher than a correlation threshold, wherein the candidate attribute feature combination includes two attribute features having a correlation higher than the correlation threshold; The probability of data being equal in the same group within each candidate attribute feature combination is respectively counted.

5. The method according to claim 4, before determining the correlation between any two attribute features in the target data based on the data of each attribute feature in the target data, further comprising: Determine that the data in the target data is an attribute feature of non-numeric data, and convert the non-numeric data in the attribute feature of the non-numeric data into numeric data.

6. The method according to claim 1, further comprising: According to the at least one set of association relationship rules, abnormal data existing in the data to be detected belonging to the specified type of data is determined, and the attribute characteristics included in the data to be detected are the same as those included in the target data.

7. A data processing device comprising: a data determining unit, configured to determine, based on target data, data of a plurality of attribute features included in the target data, wherein the target data belongs to a specified type of data that can be represented in a two-dimensional table form; a data mining unit, configured to mine associations between data of different attribute characteristics in the target data to obtain at least one set of association rules, each set of association rules including association rules that must be satisfied by data within at least two attribute characteristics, the at least one set of association rules being determined based on the association between the at least two attribute characteristics; wherein the association rules are used to determine abnormal data present in the data of the specified type; The data mining unit is specifically used for: For each attribute feature combination in the target data, the probability of data in the same group being equal under the attribute feature combination is calculated, wherein the target data includes at least one attribute feature combination, the attribute feature combination includes two attribute features in the target data, and each group of data corresponding to the target data is converted into a row of data in a two-dimensional table; Determine at least one target attribute feature combination whose equality probability is higher than the probability threshold, and obtain the equality rule corresponding to the target attribute feature combination, the equality rule of the target attribute feature combination representing that the data belonging to the same group in the two attributes within the attribute feature combination are equal; if two data belonging to the same group in the data to be detected and under the two attribute features corresponding to the target attribute feature combination are not equal, determine that the data under the two attribute features in the group in the data to be detected are abnormal.

8. The apparatus according to claim 7, wherein the data mining unit further comprises: The algorithm mining unit is used to mine the association relationship between different attribute features in the target data according to the set confidence level and using the data mining algorithm to obtain at least one set of association relationship rules.

9. An electronic device comprising: processor and memory; Wherein, the processor is used to execute the data processing method according to any one of claims 1 to 6 above; The memory is used to store programs required by the processor to perform operations.

Images