A data processing method, a data processing device, equipment and a storage medium
By using feature selection and association mining techniques, frequent itemsets are filtered from the dataset, solving the problem of low accuracy in manual case selection and achieving more efficient data processing and accurate case selection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHINA MOBILEHANGZHOUINFORMATION TECH CO LTD
- Filing Date
- 2021-11-04
- Publication Date
- 2026-05-15
AI Technical Summary
In existing technologies, case selection relies on manual identification, resulting in low accuracy and significant influence from subjective human factors.
By employing feature selection and association mining, frequent itemsets are located from the reference dataset, candidate attribute subsets are filtered using mutual information conditions, and data is filtered based on classification labels to reduce the generation of invalid rule itemsets.
This improved the accuracy of case selection, reduced data dimensionality, minimized the impact of subjective human factors, and enhanced the reliability of the selection results.
Smart Images
Figure CN116069993B_ABST
Abstract
Description
Technical Field
[0001] This application relates to, but is not limited to, the field of computers, and particularly to a data processing method, a data processing apparatus, an electronic device, and a computer-readable storage medium. Background Technology
[0002] Experience comes from practice, and wisdom originates from the grassroots. In our daily work, we often need to discover, select, and promote outstanding cases and best practices in grassroots work to create a demonstrative effect that leads to broader adoption. However, the selection of cases in related technologies relies on manual verification, and the results are significantly influenced by subjective human factors, resulting in low accuracy. Summary of the Invention
[0003] In view of this, this application provides a data processing method, a data processing device, an electronic device, and a computer-readable storage medium, which solves the problem in the related technology that the screening of cases relies on manual identification, the screening results are greatly affected by human subjective factors, and the accuracy is low.
[0004] To achieve the above objectives, the technical solution of this application is implemented as follows:
[0005] A data processing method, the method comprising:
[0006] Attribute sets are constructed by referencing the attributes of data from each category in the dataset;
[0007] A subset of candidate attributes is selected from the attribute set; wherein, the subset of candidate attributes is a subset that satisfies the mutual information condition between the attribute set and the value range set composed of the values of the data of each category;
[0008] Based on the classification identifiers of each category, the association rules between each attribute subset in the candidate attribute subset are analyzed to locate the frequent itemsets with the target association rule; wherein, the support of each itemset in the frequent itemset with the target association rule is greater than the support threshold, and the confidence of each itemset is greater than the confidence threshold, and each itemset includes at least two attribute subsets.
[0009] A data processing apparatus, the data processing apparatus comprising:
[0010] The processing unit is used to construct an attribute set based on the attributes of the data of each category in the reference dataset;
[0011] The processing unit is further configured to filter out a subset of candidate attributes from the attribute set; wherein the subset of candidate attributes is a subset that satisfies the mutual information condition between the attribute set and the value range set composed of the values of the data of each category;
[0012] The processing unit is further configured to analyze the association rules between the attribute subsets in the candidate attribute subset based on the classification identifiers of each category, and locate the frequent itemsets with the target association rule; wherein, the support of each itemset in the frequent itemset with the target association rule is greater than the support threshold, and the confidence of each itemset is greater than the confidence threshold, and each itemset includes at least two attribute subsets.
[0013] An electronic device, the electronic device comprising:
[0014] Memory, used to store executable instructions;
[0015] The processor, when executing executable instructions stored in the memory, implements the steps of the data processing method described above.
[0016] A computer-readable storage medium storing one or more applications that can be executed by one or more processors to implement the steps of the data processing method described above.
[0017] The data processing method, data processing apparatus, electronic device, and computer-readable storage medium provided in this application construct an attribute set based on the attributes of data of each category in a reference dataset; select candidate attribute subsets from the attribute set; wherein, the candidate attribute subset is a subset that satisfies the mutual information condition between the attribute set and the value range set composed of the values of data of each category; based on the classification identifier of each category, analyze the association rules between the attribute subsets in the candidate attribute subset, and locate frequent itemsets with target association rules; wherein, the support of each itemset in the frequent itemset with target association rules is greater than the support threshold, and the confidence of each itemset is greater than the confidence threshold, and each itemset includes at least two attribute subsets. It should be noted that in related technologies, case selection is based on specific selection rules. This application adopts feature selection and association mining to locate specific rules, i.e., frequent itemsets, from the reference dataset, solving the problem that the case selection work in related technologies relies on manual identification, and the selection results are greatly affected by human subjective factors, resulting in low accuracy. Meanwhile, this application uses the mutual information between the attribute subset and the value range set as the filtering condition when selecting candidate attribute subsets. This achieves better dimensionality reduction without compromising the classification filtering capability of the resulting candidate attribute subsets. Furthermore, when selecting frequent itemsets from the candidate attribute subsets, this application uses the classification labels of each category to first filter the reference dataset, removing data subsets irrelevant to the attribute subsets. This reduces the amount of data that the electronic device needs to scan, avoids counting data sets irrelevant to the classification labels caused by directly scanning the reference dataset, reduces the generation of invalid rule itemsets, and thus significantly reduces data dimensionality. Attached Figure Description
[0018] Figure 1 A flowchart illustrating a data processing method provided in this application;
[0019] Figure 2 A flowchart illustrating yet another data processing method provided in this application;
[0020] Figure 3 A flowchart illustrating the process for determining a subset of candidate attributes provided in this application;
[0021] Figure 4 A schematic diagram of the structure of a data processing device provided in this application;
[0022] Figure 5 A schematic diagram of the structure of an electronic device provided for the implementation of this application. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0024] In the following description, references to "some embodiments" refer to a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same or different subsets of all possible embodiments and may be combined with each other without conflict. Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art to which the embodiments of this application pertain. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit the application.
[0025] See Figure 1 , Figure 1 This is a schematic diagram of an implementation flow of the data processing method provided in this application embodiment. This data processing method can be applied to electronic devices; the data processing method includes the following steps:
[0026] Step 101: Construct an attribute set based on the attributes of the data in each category of the reference dataset.
[0027] In this embodiment, the electronic device acquires a reference dataset containing multiple categories of data; acquires the attributes of each category of data, and constructs an attribute set using the attributes of all categories of data. The attribute set contains at least two attribute subsets. For example, the reference dataset D contains data of multiple categories, and the attributes of each category of data in the reference dataset construct an attribute set I = {i1, i2, i3, ..., i...}.m} represents the set of all features in the reference dataset D.
[0028] In this application embodiment, the electronic device may include mobile terminal devices such as mobile phones, tablets, laptops, personal digital assistants (PDAs), wearable devices, and in-vehicle devices, as well as fixed terminal devices such as desktop computers.
[0029] Step 102: Filter out a subset of candidate attributes from the attribute set.
[0030] Among them, the candidate attribute subset is a subset that satisfies the mutual information condition between the attribute set and the value range set composed of the values of each category of data.
[0031] In this embodiment of the application, the electronic device selects candidate attribute subsets from multiple attribute subsets of the attribute set that satisfy the mutual information condition between the value range set composed of the values of data of each category.
[0032] In some embodiments, the mutual information condition between the candidate attribute subset and the value range set can be that the mutual information between the candidate attribute subset and the value range set is greater than the mutual information threshold; or that the mutual information between each attribute subset in the candidate attribute subset and the value range set is greater than the mutual information threshold.
[0033] Step 103: Based on the classification identifiers of each category, analyze the association rules between each attribute subset in the candidate attribute subset, and locate the frequent itemsets with the target association rules.
[0034] Among them, the support of each item set in the frequent itemset with the target association rule is greater than the support threshold, and the confidence of each item set is greater than the confidence threshold.
[0035] Each item set in a frequent itemset includes at least two subsets of attributes.
[0036] In this embodiment of the application, the support of each itemset in the frequent itemsets of the reference dataset is the frequency of occurrence of data records containing itemset T in the reference dataset, denoted as:
[0037]
[0038] Where D represents the reference dataset; T represents the itemset; and X represents the subset of data.
[0039] In this embodiment, each itemset includes at least two attribute subsets. The confidence score of an itemset T containing attribute subsets X and Y is the frequency of transactions in the reference dataset D that contain attribute subset X and transactions that contain attribute subset Y. It should be noted that the attribute subset X of itemset T... The prerequisite for the validity of attribute subset Y (association rule) is that attribute subset X occurs, denoted as:
[0040]
[0041] Where D represents the reference dataset; X and Y represent subsets of attributes of itemset T.
[0042] In this application embodiment, the confidence threshold and support threshold can be preset by the electronic device or set by the user based on actual data. This application does not impose any limitations on this.
[0043] This application first divides the reference dataset into multiple data subsets based on the classification attribute values. Each data subset has a unified classification filtering label. Then, it performs separate classification association rule mining on each data subset, that is, the attribute set I = {i1, i2, i3, ..., i...} in the reference dataset D. m Each of the following corresponds to a subset of data and has a unique identifier TID.
[0044] In this embodiment of the application, the attribute set I = {i1,i2,i3,…,i...} m In the set I, each attribute subset has at least one corresponding data subset. It should be noted that the attribute set I = {i1, i2, i3, ..., i...} m A subset of} can be called an itemset. In the mining process of each individual data subset, the itemset {I} is... i} Support number and itemset {I} in the reference dataset i The reference dataset contains the classification identifier C. i itemset {I i C i The number of supported itemsets is the same. Therefore, when selecting frequent itemsets from the candidate attribute subset, this application uses the classification labels of each category to first filter the reference dataset, filtering out data subsets that are irrelevant to the attribute subset. This reduces the amount of data that the electronic device needs to scan, thereby avoiding the counting operation of datasets irrelevant to the classification labels caused by directly scanning the reference dataset, reducing the generation of invalid rule itemsets, and significantly reducing the data dimensionality.
[0045] In this embodiment of the application, the electronic device determines the data subset corresponding to each attribute subset in the candidate attribute subset, analyzes the association rules between each attribute subset under the data subset, and locates the frequent itemsets with the target association rule from the candidate attribute subset.
[0046] The data processing method provided in this application includes: constructing an attribute set based on the attributes of data of each category in a reference dataset; filtering candidate attribute subsets from the attribute set; wherein, the candidate attribute subset is a subset that satisfies the mutual information condition between the attribute set and the value range set composed of the values of data of each category; analyzing the association rules between the attribute subsets in the candidate attribute subset based on the classification identifier of each category, and locating frequent itemsets with target association rules; wherein, the support of each itemset in the frequent itemset with target association rules is greater than the support threshold, and the confidence of each itemset is greater than the confidence threshold, and each itemset includes at least two attribute subsets. It should be noted that in related technologies, case screening is based on specific screening rules. This application uses feature selection and association mining to locate specific rules, i.e., frequent itemsets, from the reference dataset, solving the problem that case screening in related technologies relies on manual identification, and the screening results are greatly affected by subjective human factors, resulting in low accuracy. Meanwhile, this application uses the mutual information between the attribute subset and the value range set as the filtering condition when selecting candidate attribute subsets. This achieves better dimensionality reduction without compromising the classification filtering capability of the resulting candidate attribute subsets. Furthermore, when selecting frequent itemsets from the candidate attribute subsets, this application uses the classification labels of each category to filter the reference dataset first, removing data subsets irrelevant to the attribute subsets. This reduces the amount of data that the electronic device needs to scan, avoiding the counting operations on datasets irrelevant to the classification labels caused by directly scanning the reference dataset. This reduces the generation of invalid rule itemsets and significantly lowers the data dimensionality.
[0047] See Figure 2 , Figure 2 This is a schematic diagram of an implementation flow of the data processing method provided in an embodiment of this application. This control method can be applied to electronic devices; the data processing method includes the following steps:
[0048] Step 201: Construct an attribute set based on the attributes of the data in each category of the reference dataset.
[0049] Step 202: Obtain the first mutual information between the attribute set and the value range set, as well as the weight coefficient corresponding to each attribute subset in the attribute set.
[0050] In this embodiment of the application, obtaining the weight coefficient corresponding to each subset of attributes in the attribute set in step 202 can be achieved through the following steps:
[0051] Step 1: Establish the attribute subset weight set w = (w1, w2, ..., w t ,…,w T ).
[0052] In this context, all weight values in the attribute subset weight set w are initialized to 0; T is used to represent the number of attribute subsets contained in the attribute set.
[0053] Step 2: Randomly select a data sample x from the reference dataset. i According to formula (3), select k items related to x. i The nearest neighbor samples of the same type form NH, and k samples are related to x. i The nearest neighbor samples of the outlier form NMHz.
[0054]
[0055] Where A represents the attribute set; t represents the attribute features in the attribute set, i.e., the attribute subset; NH represents a set containing k attributes related to x. i The sample set of nearest neighbors of the same class; NMHz represents the set of k nearest neighbors of x. i The sample set of nearest neighbor samples of different classes; z is used to represent the sample class, and z ≠ class(xi); class(xi) is used to represent sample x. i The category. This application uses sample x i and sample x j The sum of the differences of all attributes between them is used as the selection criterion to select the nearest neighbor sample.
[0056] In this embodiment of the application, it is assumed that the reference dataset is D, which contains a total of |y| categories, for sample x i If it belongs to the k-th class (k∈{1, 2, |y|}), this application first searches for x in the samples of the k-th class. i Find the nearest neighbor of x, and then find x in the samples of each class other than the k-th class. i The nearest neighbor sample.
[0057] It should be noted that for discrete attribute features t, sample x i and sample x j The difference between them is defined as:
[0058]
[0059] For a continuous attribute t, sample x i and sample x j The difference between them is defined as:
[0060]
[0061] Where, max t min represents the maximum value of attribute feature t; t This represents the minimum value of attribute feature t.
[0062] Third step: For each attribute feature t in the attribute set, calculate sample x based on formula (4) and formula (5). i The difference between the feature and all its nearest neighbor samples.
[0063] Step 4: Calculate and update the feature weights according to formula (6).
[0064]
[0065] Where n represents the number of iterations; Z represents the value range of each category's data attribute in the reference dataset; p(z) represents the value of sample x. j The proportion of the class to which it belongs to the overall reference dataset; p(class(x) i )) represents sample x i The proportion of the category to which it belongs to the overall reference dataset.
[0066] It should be noted that if sample x i The difference between attribute feature t in a sample and attribute feature t in its nearest neighbor sample of the same class, diff(t,x) i ,x NH Less than sample x i The difference between attribute feature t in the class and attribute feature t in the nearest neighbor sample of the different class If the feature has strong classification and screening capabilities, its weight should be increased; conversely, if the feature has weak capabilities, its weight should be reduced.
[0067] Fifth step: Iterate through steps two through four n times to determine the weight coefficient for each subset of attributes in the attribute set.
[0068] In this embodiment of the application, diff(t,x) is used. i ,x NH )and The weight coefficient of feature t is adjusted by the difference between the two values. After randomly selecting n samples, the weight coefficient corresponding to each subset of attributes in the attribute set can be obtained.
[0069] In this embodiment of the application, the electronic device calculates the first mutual information between the attribute set X and the value range set Y composed of the values of each category of data according to formula (7).
[0070]
[0071] Wherein, P(X=o) x ∧Y=o y ) represents the joint probability density function of attribute set X and range set Y; x O represents the attribute in attribute set X; xThis represents a subset of attributes in attribute set X; y O represents the values in the range set Y; y P(X=o) represents a subset of the range in the range set Y; x P(Y = o) represents the marginal probability distribution function of attribute set X; y ) represents the marginal probability distribution function of the range set Y.
[0072] Step 203: Based on the weight coefficients and the first mutual information, select a subset of candidate attributes from the attribute set.
[0073] In the embodiments of this application, the process of selecting a subset of candidate attributes from the attribute set based on the weight coefficients and the first mutual information in step 203 can be achieved through the following steps:
[0074] Step A1: Determine the largest weight coefficient from the weight coefficients corresponding to each attribute in the attribute set, and determine the first attribute subset corresponding to the largest weight coefficient.
[0075] In some embodiments, after obtaining the weight coefficients corresponding to each subset of attributes in the attribute set, the weight coefficient set is sorted to obtain a sorted weight coefficient set w = (w1, w2, ..., w...). t ,…,w T It should be noted that the sorting rule is based on the size of the weight coefficients, decreasing from largest to smallest. The electronic device retrieves w1, the largest weight coefficient, from the sorted set of weight coefficients and determines the first attribute subset corresponding to w1.
[0076] Step A2: Obtain the second mutual information between the first attribute subset and the value range set.
[0077] In this embodiment of the application, the electronic device calculates the second mutual information between the value range set and the first attribute subset corresponding to the weight coefficient w1.
[0078] Step A3: Determine the candidate attribute subset based on the first mutual information and the second mutual information.
[0079] In the embodiments of this application, the process of determining the candidate attribute subset based on the first mutual information and the second mutual information in step A3 can be achieved through the following steps:
[0080] Step B1: If the first mutual information is less than the second mutual information, determine the first attribute subset as the candidate feature subset.
[0081] In this embodiment of the application, if the second mutual information between the first attribute subset corresponding to the weight coefficient w1 and the value range set is greater than the first mutual information, that is, the mutual information between the first attribute subset and the value range set is greater than the mutual information between the original feature set and the value range set, retaining the first attribute subset can ensure that the classification and screening ability will not decrease, then the first attribute subset is a candidate feature subset.
[0082] Step B2: If the first mutual information is greater than the second mutual information, obtain at least one subset of the second attributes that satisfies the target condition from the attribute set.
[0083] The second attribute subset is different from the first attribute subset.
[0084] In this embodiment, at least one subset of second attributes and one subset of first attributes can form a union. The objective condition is that the third mutual information between the value range set and the union is greater than the first mutual information.
[0085] In this embodiment, if the second mutual information between the first attribute subset corresponding to weight coefficient w1 and the value range set is less than the first mutual information, the electronic device obtains w2 (the second largest weight coefficient) from the sorted set of weight coefficients and determines the second attribute subset corresponding to w2. The first union of the first attribute subset corresponding to weight coefficient w1 and the second attribute subset corresponding to weight coefficient w2 is obtained. The electronic device calculates the mutual information between the first union and the value range set. If the mutual information between the first union and the value range set is greater than the first mutual information, then the first union is a candidate attribute subset. If the mutual information between the first union and the value range set is still less than the first mutual information, the electronic device continues to obtain weight coefficients, such as w3, from the sorted set of weight coefficients and determines the second attribute subset corresponding to the weight coefficient. Further, second attribute subsets are added to the first union to obtain a new union until the mutual information between the new union and the value range set is greater than the first mutual information. Only then will the electronic device stop obtaining new weight coefficients from the sorted set of weight coefficients.
[0086] In this embodiment of the application, obtaining at least one second attribute subset that satisfies the target condition from the attribute set means that the mutual information between the union of the at least one second attribute subset obtained from the attribute set and the first attribute subset and the value range set is greater than the first mutual information. In this case, the at least one second attribute subset is an attribute subset that satisfies the target condition.
[0087] Step B3: Determine the union of the first attribute subset and at least one second attribute subset as the candidate attribute subset.
[0088] Figure 3 This is a flowchart illustrating a process for determining a subset of candidate attributes provided in this application. Figure 3 As shown, determining a subset of candidate attributes can be achieved through the following steps:
[0089] Step 301: Obtain the attribute set and the weight coefficient corresponding to each attribute sub-attribute in the attribute set.
[0090] Step 302: Calculate the first mutual information between the attribute set and the value range set.
[0091] In this embodiment of the application, the electronic device calculates the first mutual information of the value range set composed of the attribute set and the values of data of each category.
[0092] Step 303: Add the attribute subset with the largest weight coefficient to the feature subset.
[0093] Step 304: Calculate the second mutual information between the feature subset and the value range set.
[0094] Step 305: Compare whether the first mutual information is greater than the second mutual information. If the first mutual information is greater than the second mutual information, proceed to step 306; if the first mutual information is less than the second mutual information, proceed to step 307.
[0095] Step 306: Determine the feature subset as the candidate attribute subset.
[0096] In this embodiment of the application, the feature subset with the largest weight coefficient is determined as the candidate attribute subset.
[0097] Step 307: Delete the subset of attributes with the largest weight coefficient in the attribute set to obtain the first condition attribute subset, and then execute step 308.
[0098] Step 308: Select the attribute subset with the largest feature weight from the first conditional attribute subset, add the attribute subset to the feature subset, and then jump to step 304.
[0099] Step 204: Obtain the category identifier corresponding to each attribute subset in the candidate attribute subset, and determine the data corresponding to the category identifier from the data of each category.
[0100] This application first divides the reference dataset into multiple data subsets based on the classification attribute values. Each data subset has a unified classification filtering label. Then, it performs separate classification association rule mining on each data subset, that is, the attribute set I = {i1, i2, i3, ..., i...} in the reference dataset D. m Each attribute corresponds to a subset of data and has a unique identifier (TID). The electronic device obtains the category identifier corresponding to each attribute subset in the candidate attribute subset and determines the data subset corresponding to the category identifier from the data of each category.
[0101] Step 205: Calculate the number of times the j-th data in the i-th attribute subset appears in the dataset with the classification label in the i-th attribute subset.
[0102] Where, i and j are positive integers.
[0103] In the embodiment of the present application, for each attribute subset in the candidate attribute subsets, calculate the number of occurrences of the j-th data in the i-th attribute subset in the candidate attribute subsets in the dataset with the classification identifier. That is to say, when calculating the number of occurrences of data in each attribute subset in the present application, first perform a screening on the data through the classification identifier, and then calculate in the data subset corresponding to the attribute subset. Compared with calculating in the reference dataset when calculating the number of occurrences of data in each attribute subset in the related art, the present application reduces the amount of data that the electronic device needs to scan, thereby avoiding the counting operation on the dataset irrelevant to the classification identifier caused by directly scanning the reference dataset, reducing the problem of the generation of invalid rule item sets, and further significantly reducing the data dimension.
[0104] Step 206: Delete the attribute subsets in the candidate attribute subsets whose number of occurrences is less than the number threshold to obtain the reference attribute subsets.
[0105] It should be noted that if all supersets of a frequent item set are non-frequent item sets, this frequent item set is a maximum frequent item set; for example, if each element in a set S2 is in a set S1, and the set S1 may contain elements not in S2, then the set S1 is a superset of S2.
[0106] In the embodiment of the present application, according to the theory that the superset of the maximum frequent item set is a non-frequent item set and the subset of the maximum frequent item set is a frequent item set, determine the maximum frequent item set in the candidate attribute subsets in advance, and prune them to avoid their self-join operations, further reducing the number of candidate item sets and improving the mining efficiency of classification association rules.
[0107] In the embodiment of the present application, the electronic device calculates the number of occurrences of each attribute subset in the candidate attribute subset L k-1 and represents it by L k-1 (p); and records the attribute subsets whose number of occurrences is less than the number threshold, that is, k - 1, denoted as P = {p||L k-1 (p)| < k - 1}, and delete the frequent item sets in L k-1 that contain any element in P to obtain the reference attribute subset denoted as L k-1’ .
[0108] Step 207: Perform a self-join operation on the reference attribute subsets to obtain multiple candidate frequent item sets.
[0109] In the embodiment of the present application, for the reference attribute subset L k-1If any two attribute subsets, such as itemset1 and itemset2, have the same first k-2 items, then itemset1 and itemset2 are said to be connectable, meaning they can be self-joined. The resulting itemset from joining attribute subsets itemset1 and itemset2 is {itemset1[1], itemset1[2], ..., itemset1[k-1], itemset2[k-1]}. Performing a self-join operation on any two attribute subsets in the reference attribute subset yields multiple candidate frequent itemsets.
[0110] For example, a subset of reference attributes is a 2-itemset, which is self-joined to obtain at least one 3-itemset; for at least one 3-itemset, a self-join is performed to obtain at least one 4-itemset, until all frequent itemsets corresponding to the subset of reference attributes are found.
[0111] Step 208: Calculate the support and confidence of each candidate frequent itemset in the multiple candidate frequent itemsets.
[0112] Step 209: Based on the support and confidence of each candidate frequent itemset, select frequent itemsets from multiple candidate frequent itemsets.
[0113] In this embodiment of the application, the electronic device determines the frequent itemsets with the target association rule based on the support and confidence of each candidate frequent itemset. That is, it selects the itemsets with support greater than the support threshold and confidence greater than the confidence threshold in each candidate frequent itemset as frequent itemsets with the target association rule.
[0114] Step 210: Use frequent itemsets to filter the reference dataset and obtain the filtering results.
[0115] In this embodiment of the application, for cases where the number of rules obtained from the screening of frequent itemsets with target association rules is huge, i.e., the number of rules obtained from the classification association rules is huge, this application also performs rule pruning on the frequent itemsets with target association rules.
[0116] In this embodiment of the application, the electronic device sorts each item set in the frequent itemset sequentially according to the confidence, support and rule itemset dimensions to obtain the sorted frequent itemset.
[0117] It should be noted that each item set in the frequent itemset is first sorted by confidence from largest to smallest. If the confidence is the same, it is then sorted by support from largest to smallest. If the confidence and support are the same, it is then sorted by rule item set dimension from smallest to largest.
[0118] Step 211: Based on the screening results, determine the classification accuracy of each item set in the frequent itemset.
[0119] Step 212: Select the target frequent itemset from the frequent itemsets.
[0120] Among them, the classification accuracy of the frequent itemset of the target is greater than the accuracy threshold, and the classification accuracy of the frequent subset of the target obtained by multiple screenings of the reference dataset is fixed.
[0121] In this embodiment, the electronic device selects the highest priority first itemset from the sorted frequent itemsets for examination. The classification accuracy is recorded as the ratio of correctly classified samples of the first itemset to the number of samples covered by the rule in the reference dataset. Samples associated with the first itemset are then deleted from the reference dataset, resulting in a first dataset, which is then labeled. Next, the second itemset with the second highest priority is selected from the sorted frequent itemsets for examination. The classification accuracy is recorded as the ratio of correctly classified samples of the second itemset to the number of samples covered by the rule in the first dataset. Samples corresponding to the second itemset are then deleted from the first dataset, and the second itemset is labeled. This process continues, selecting itemsets from the sorted frequent itemsets and calculating the classification accuracy of each itemset, until no samples remain in the reference dataset or all itemsets have been traversed.
[0122] In this embodiment of the application, if there is an itemset that has not been traversed, the itemset is removed; if there is a sample that cannot be classified by any itemset, the sample is assigned to the category with the most samples in the database, i.e., the default category.
[0123] In this embodiment, the electronic device sorts the classification accuracy of each item set in the frequent itemset, removes itemsets with a classification accuracy of 0 or a non-zero correct classification ratio but ranked last, and obtains the target frequent itemset. The overall classification accuracy of the target frequent itemset on the reference dataset is calculated. Finally, the electronic device filters the reference dataset based on the target frequent itemset until the classification accuracy of the target frequent itemset on the database no longer improves, thus obtaining the final target frequent itemset.
[0124] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.
[0125] Based on the foregoing embodiments, embodiments of this application provide a data processing apparatus that can be applied to... Figures 1 to 2 In the data processing apparatus provided in the corresponding embodiment, refer to Figure 4 As shown, the data processing device 4 may include:
[0126] Processing unit 401 is used to construct an attribute set based on the attributes of data of each category in the reference dataset;
[0127] Processing unit 401 is used to filter out candidate attribute subsets from the attribute set; wherein, the candidate attribute subset is a subset that satisfies the mutual information condition between the attribute set and the value range set composed of the values of each category of data;
[0128] The processing unit 401 is used to analyze the association rules between each attribute subset in the candidate attribute subset based on the classification identifier of each category, and locate the frequent itemsets with the target association rule; wherein, the support of each itemset in the frequent itemset with the target association rule is greater than the support threshold, and the confidence of each itemset is greater than the confidence threshold, and each itemset includes at least two attribute subsets.
[0129] In other embodiments of this application, the data processing apparatus 4 further includes:
[0130] The acquisition unit 402 is used to acquire the classification identifier corresponding to each attribute subset in the candidate attribute subset, and to determine the data corresponding to the classification identifier from the data of each category;
[0131] Processing unit 401 is used to calculate the number of times the j-th data in the i-th attribute subset appears in the dataset with the classification label of the i-th attribute subset; where i and j are positive integers;
[0132] Processing unit 401 is used to delete the attribute subsets in the candidate attribute subsets whose occurrence frequency is less than the frequency threshold, and obtain the reference attribute subset;
[0133] Processing unit 401 is used to perform a self-join operation on the reference attribute subset to obtain multiple candidate frequent itemsets;
[0134] Processing unit 401 is used to calculate the support and confidence of each candidate frequent itemset among multiple candidate frequent itemsets.
[0135] Processing unit 401 is used to select frequent itemsets from multiple candidate frequent itemsets based on the support and confidence of each candidate frequent itemset.
[0136] In other embodiments of this application, the data processing apparatus 4 further includes:
[0137] The acquisition unit 402 is used to acquire the first mutual information between the attribute set and the value range set, as well as the weight coefficient corresponding to each attribute subset in the attribute set;
[0138] Processing unit 401 is used to filter out a subset of candidate attributes from the attribute set based on the weight coefficients and the first mutual information.
[0139] In other embodiments of this application, the data processing apparatus 4 further includes:
[0140] Processing unit 401 is used to determine the largest weight coefficient from the weight coefficients corresponding to each subset of attributes in the attribute set, and to determine the first subset of attributes corresponding to the largest weight coefficient;
[0141] Acquisition unit 402 is used to acquire the second mutual information between the first attribute subset and the value range set;
[0142] Processing unit 401 is used to determine a subset of candidate attributes based on the first mutual information and the second mutual information.
[0143] In other embodiments of this application, the data processing apparatus 4 further includes:
[0144] Processing unit 401 is used to determine the first attribute subset as a candidate feature subset if the first mutual information is less than the second mutual information.
[0145] Processing unit 401 is configured to, if the first mutual information is greater than the second mutual information, obtain at least one second attribute subset from the attribute set that satisfies the target condition; wherein the second attribute subset is different from the first attribute subset.
[0146] Processing unit 401 is used to determine the union of a first attribute subset and at least one second attribute subset as a candidate attribute subset.
[0147] In other embodiments of this application, the objective condition is that the third mutual information between the union and the range set is greater than the first mutual information.
[0148] In other embodiments of this application, the data processing apparatus 4 further includes:
[0149] Processing unit 401 is used to filter the reference dataset using frequent itemsets to obtain the filtering results;
[0150] Processing unit 401 is used to determine the classification accuracy of each item set in the frequent itemset based on the screening results;
[0151] Processing unit 401 is used to filter out target frequent itemsets from frequent itemsets; wherein the classification accuracy of the target frequent itemsets is greater than the accuracy threshold, and the classification accuracy of the target frequent subset obtained by filtering the reference dataset multiple times is fixed.
[0152] It should be noted that the specific implementation process of the steps executed by the processor in this embodiment can be referred to Figures 1 to 2 The implementation process of the data processing method provided in the corresponding embodiments will not be described in detail here.
[0153] This application provides an electronic device 5, which can be applied to... Figures 1 to 2 In a corresponding embodiment of the data processing method, referring to Figure 5 As shown, the electronic device 5 includes: a processor 501, a memory 502, and a communication bus 503, wherein:
[0154] The communication bus 503 is used to realize the communication connection between the processor 501 and the memory 502.
[0155] The processor 501 is used to execute the data processing program stored in the memory 502 to perform the following steps:
[0156] Attribute sets are constructed by referencing the attributes of data from each category in the dataset;
[0157] Candidate attribute subsets are selected from the attribute set; where the candidate attribute subset is a subset that satisfies the mutual information condition between the attribute set and the value range set composed of the values of each category of data.
[0158] Based on the classification labels of each category, the association rules between each attribute subset in the candidate attribute subset are analyzed to locate the frequent itemsets with the target association rule; wherein, the support of each itemet in the frequent itemset with the target association rule is greater than the support threshold, and the confidence of each itemet is greater than the confidence threshold, and each itemet includes at least two attribute subsets.
[0159] In other embodiments of this application, processor 501 is used to execute a data processing program stored in memory 502 to perform the following steps:
[0160] Obtain the category identifier corresponding to each attribute subset in the candidate attribute subset, and determine the data corresponding to the category identifier from the data of each category;
[0161] Calculate the number of times the j-th data point in the i-th attribute subset appears in the dataset with the classification label of the i-th attribute subset; where i and j are positive integers;
[0162] Remove the subsets of attributes that appear less than a threshold number of times from the candidate attribute subset to obtain the reference attribute subset;
[0163] Perform a self-join operation on the reference attribute subset to obtain multiple candidate frequent itemsets;
[0164] Calculate the support and confidence of each candidate frequent itemset from multiple candidate frequent itemsets.
[0165] Frequent itemsets are selected from multiple candidate frequent itemsets based on the support and confidence of each candidate frequent itemset.
[0166] In other embodiments of this application, processor 501 is used to execute a data processing program stored in memory 502 to perform the following steps:
[0167] Obtain the first mutual information between the attribute set and the value range set, as well as the weight coefficient corresponding to each attribute subset in the attribute set;
[0168] Based on the weighting coefficients and the first mutual information, a subset of candidate attributes is selected from the attribute set.
[0169] In other embodiments of this application, processor 501 is used to execute a data processing program stored in memory 502 to perform the following steps:
[0170] Determine the largest weight coefficient from the weight coefficients corresponding to each subset of attributes in the attribute set, and determine the first subset of attributes corresponding to the largest weight coefficient;
[0171] Obtain the second mutual information between the first attribute subset and the value range set;
[0172] Based on the first mutual information and the second mutual information, a subset of candidate attributes is determined.
[0173] In other embodiments of this application, processor 501 is used to execute a data processing program stored in memory 502 to perform the following steps:
[0174] If the first mutual information is less than the second mutual information, the first attribute subset is determined as the candidate feature subset.
[0175] If the first mutual information is greater than the second mutual information, at least one subset of second attributes that satisfies the target condition is obtained from the attribute set; wherein the subset of second attributes is different from the subset of first attributes.
[0176] The union of the first attribute subset and at least one second attribute subset is determined as the candidate attribute subset.
[0177] In other embodiments of this application, the objective condition is that the third mutual information between the union and the range set is greater than the first mutual information.
[0178] In other embodiments of this application, processor 501 is used to execute a data processing program stored in memory 502 to perform the following steps:
[0179] The reference dataset is filtered using frequent itemsets to obtain the filtering results;
[0180] Based on the screening results, determine the classification accuracy of each item set in the frequent itemset;
[0181] Select a target frequent itemset from the frequent itemsets; wherein the classification accuracy of the target frequent itemset is greater than the accuracy threshold, and the classification accuracy of the target frequent subset obtained by multiple selections from the reference dataset is fixed.
[0182] It should be noted that the specific implementation process of the steps executed by the processor in this embodiment can be referred to Figures 1 to 2 The implementation process of the data processing method provided in the corresponding embodiments will not be described in detail here.
[0183] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium that stores one or more applications, which can be executed by one or more processors to achieve, as Figures 1 to 2 The corresponding implementation provides a data processing method.
[0184] It should be noted that the aforementioned computer-readable storage media can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), magnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM), etc.; or it can be various electronic devices that include one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.
[0185] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.
[0186] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.
[0187] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0188] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer application products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer application instructions. These computer application instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0189] These computer application instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0190] These computer application instructions can also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0191] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A data processing method, characterized in that, The method includes: Attribute sets are constructed by referencing the attributes of data from each category in the dataset; A subset of candidate attributes is selected from the attribute set; wherein, the subset of candidate attributes is a subset that satisfies the mutual information condition between the attribute set and the value range set composed of the values of the data of each category; Obtain the classification identifier corresponding to each attribute subset in the candidate attribute subset, and determine the data corresponding to the classification identifier from the data of each category; Calculate the number of times the j-th data in the i-th attribute subset of the candidate attribute subset appears in the dataset with the classification label of the i-th attribute subset; where i and j are positive integers; Delete the subset of attributes in the candidate attribute subset whose occurrence frequency is less than the frequency threshold to obtain the reference attribute subset; Perform a self-join operation on the reference attribute subset to obtain multiple candidate frequent itemsets; Calculate the support and confidence of each candidate frequent itemset from the plurality of candidate frequent itemsets; Based on the support and confidence of each candidate frequent itemset, frequent itemsets with target association rules are selected from the multiple candidate frequent itemsets; wherein, the support of each itemset in the frequent itemsets with target association rules is greater than the support threshold, and the confidence of each itemset is greater than the confidence threshold, and each itemset includes at least two attribute subsets.
2. The method according to claim 1, characterized in that, The step of filtering a subset of candidate attributes from the attribute set includes: Obtain the first mutual information between the attribute set and the value range set, and the weight coefficient corresponding to each attribute subset in the attribute set; Based on the weight coefficients and the first mutual information, a subset of candidate attributes is selected from the attribute set.
3. The method according to claim 2, characterized in that, The step of selecting a subset of candidate attributes from the attribute set based on the weight coefficients and the first mutual information includes: From the weight coefficients corresponding to each subset of attributes in the attribute set, determine the largest weight coefficient, and determine the first subset of attributes corresponding to the largest weight coefficient; Obtain the second mutual information between the first attribute subset and the value range set; The candidate attribute subset is determined based on the first mutual information and the second mutual information.
4. The method according to claim 3, characterized in that, The step of determining the candidate attribute subset based on the first mutual information and the second mutual information includes: If the first mutual information is less than the second mutual information, the first attribute subset is determined to be the candidate attribute subset; If the first mutual information is greater than the second mutual information, at least one second attribute subset that satisfies the target condition is obtained from the attribute set; wherein the second attribute subset is different from the first attribute subset; The union of the first attribute subset and the at least one second attribute subset is determined as the candidate attribute subset.
5. The method according to claim 4, characterized in that, The target condition is that the third mutual information between the union and the value range set is greater than the first mutual information.
6. The method according to any one of claims 1 to 5, characterized in that, After locating frequent itemsets with target association rules, the method includes: The reference dataset is filtered using the frequent itemsets to obtain the filtering results; Based on the filtering results, the classification accuracy of each item set in the frequent itemset is determined; Select a target frequent itemset from the frequent itemset; wherein the classification accuracy of the target frequent itemset is greater than the accuracy threshold, and the classification accuracy of the target frequent itemset obtained by multiple selections from the reference dataset is fixed.
7. A data processing apparatus, characterized in that, The data processing device includes: The processing unit is used to construct an attribute set based on the attributes of the data of each category in the reference dataset; The processing unit is further configured to filter out a subset of candidate attributes from the attribute set; wherein the subset of candidate attributes is a subset that satisfies the mutual information condition between the attribute set and the value range set composed of the values of the data of each category; The acquisition unit is used to acquire the classification identifier corresponding to each attribute subset in the candidate attribute subset, and to determine the data corresponding to the classification identifier from the data of each category; The processing unit is further configured to calculate the number of times the j-th data in the i-th attribute subset of the candidate attribute subset appears in the dataset with the classification identifier of the i-th attribute subset; where i and j are positive integers; The processing unit is further configured to delete the subset of attributes in the candidate attribute subset whose occurrence frequency is less than the frequency threshold, and obtain a reference attribute subset; The processing unit is also used to perform a self-join operation on the reference attribute subset to obtain multiple candidate frequent itemsets; The processing unit is also used to calculate the support and confidence of each candidate frequent itemset in the plurality of candidate frequent itemsets. The processing unit is further configured to filter out frequent itemsets with target association rules from the plurality of candidate frequent itemsets based on the support and confidence of each candidate frequent itemset; wherein, the support of each itemset in the frequent itemset with target association rules is greater than a support threshold, and the confidence of each itemset is greater than a confidence threshold, and each itemset includes at least two attribute subsets.
8. An electronic device, characterized in that, The electronic device includes: Memory, used to store executable instructions; A processor, when executing executable instructions stored in the memory, implements the data processing method according to any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the data processing method as described in any one of claims 1 to 6.