A data processing method and device based on relational data

Abstract

The present application provides a data processing method and apparatus based on relational data, which is used to repair target data with missing data segments through data relations and verify the validity of the repaired target data. The method comprises: obtaining target data and performing data screening based on the semantics of the target data to determine sampled data; generating template predicates based on the sampled data and constructing a target template based on the template predicates; performing data screening on the target data based on the constant predicates to construct a predicate set; performing association rule mining based on the predicate set to generate a candidate rule set; determining valid rules within the candidate rule set based on the target data, and determining valid data based on the valid rules. This method allows for the discovery of rules with constants in large-scale relational data without enumerating all possible constants, thereby greatly improving the execution efficiency of rule discovery.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data processing method and device based on relational data. Background Art

[0002] Discovering rules in large-scale relational data is a time-consuming and labor-intensive process. When constants are allowed in rules, the cost of rule discovery increases exponentially.

[0003] For example, consider the following simple Conditional Functional Dependency (CFD):

[0004] Address = "Shenzhen, Guangdong Province" -> Postal Code = "518000"

[0005] This CFD describes a scenario where, if an address attribute is in Shenzhen, Guangdong Province, then its corresponding postal code attribute must be 518000. This rule can be widely used for error checking and correction in relational data. Specifically, when the data stored in the relational database violates this rule (i.e., an address attribute is in Shenzhen, Guangdong Province, but its postal code attribute is not 518000), it can be determined that there is an error in the data, and further data correction can be performed. In this rule, "Shenzhen, Guangdong Province" and "518000" are both constants, and the address and postal code are the attribute names of the data.

[0006] When discovering rules with constants in large-scale data, one must not only consider the permutations and combinations of different data attributes, but also the constants that each attribute might match. This enumeration process is very expensive. For example, the following CFDs:

[0007] Address = "Guangzhou City, Guangdong Province" -> Postal Code = "510000"

[0008] Address = "Dongguan City, Guangdong Province" -> Postal Code = "523000"

[0009] Address = "Foshan City, Guangdong Province" -> Postal Code = "528010"

[0010] The scenarios described by these CFDs are similar, differing only in the use of different constants. While the attributes of the rules (i.e., address and zip code) are the same, the matching constants are different. Enumerating all possible matching constants in the data would undoubtedly significantly reduce the efficiency of rule discovery. Rule discovery in a typical relational database could take days or even weeks.

[0011] The limited expressive power of CFD rules limits their applicability in real-world scenarios. To support constant predicates, CFD rule mining requires enumerating all possible combinations of attributes and constants, which is time-consuming and labor-intensive. Summary of the Invention

[0012] In view of the above problems, the present application is proposed to provide a data processing method based on relational data and a device thereof that overcomes the above problems or at least partially solves the above problems, including:

[0013] A data processing method based on relational data, the method being used to repair target data with missing data segments through data relations and verify the validity of the repaired target data, comprising:

[0014] Acquire target data, and perform data screening based on the meaning of the target data to determine sampled data, wherein the sampled data is a constant predicate and includes at least one;

[0015] generating a template predicate according to the sampled data, and constructing a target template according to the template predicate;

[0016] Performing data screening on the target data according to the constant predicate to construct a predicate set;

[0017] Performing association rule mining based on the total set of predicates to generate a total set of candidate rules;

[0018] Valid rules in the total set of candidate rules are determined based on the target data, and valid data are determined based on the valid rules.

[0019] Furthermore, the step of obtaining target data and performing data screening to determine sampled data based on the meaning of the target data, wherein the sampled data is a constant predicate, includes at least one step, including:

[0020] Obtaining data attributes in the target data;

[0021] Determining the word sense type of the target data according to the data attribute, wherein the word sense type includes a constant and a non-constant;

[0022] The target data having the word meaning type corresponding to the constant is screened and determined as the sampled data.

[0023] Furthermore, the step of generating a template predicate based on the sampled data and constructing a target template based on the template predicate includes:

[0024] generating a template predicate according to the sampled data;

[0025] When the template predicate has a valid value in the target data, the template predicate is determined to be a valid predicate;

[0026] A target template is constructed according to the valid predicate.

[0027] Furthermore, the step of constructing a target template based on the valid predicate includes:

[0028] Combining the valid predicates to generate permutations and combinations;

[0029] Screening the permutations and combinations to determine a template predicate combination;

[0030] The target template is constructed according to the template predicate combination and the effective predicate.

[0031] Furthermore, the step of filtering the target data according to the constant predicate to construct a predicate set includes:

[0032] Performing data screening on the target data according to the constant predicate to determine non-constant predicate data, wherein the non-constant predicate data is a non-constant predicate set;

[0033] Supplementing the non-constant predicate data with constant values ​​according to the target template to generate constant predicate data, wherein the constant predicate data is a constant predicate set;

[0034] The total predicate set is constructed according to the non-constant predicate set and the constant predicate set.

[0035] Furthermore, the step of performing association rule mining based on the total set of predicates to generate a candidate rule set includes:

[0036] Performing a depth-first search based on the total set of predicates to generate a first candidate rule set; or,

[0037] Performing a breadth-first search based on the total set of predicates to generate a second candidate rule set;

[0038] The total set of candidate rules is generated according to the first candidate rule set or the second candidate rule set.

[0039] Furthermore, the step of determining valid rules in the total set of candidate rules based on the target data, and determining valid data based on the valid rules, includes:

[0040] Obtain each sub-candidate rule in the total set of candidate rules;

[0041] Verifying each of the candidate sub-rules according to the target data to determine the validity of each of the candidate sub-rules; wherein, when there is a sub-target data in the target data corresponding to the current candidate sub-rule, the current candidate sub-rule is determined to be a valid rule;

[0042] The sub-target data corresponding to the valid rule is obtained, and the sub-target data is marked as the valid data.

[0043] The present application also discloses a data processing device based on relational data, which is used to repair target data with missing data segments through data relations and verify the validity of the repaired target data, including:

[0044] an acquisition module, configured to acquire target data and perform data screening to determine sampled data based on the meaning of the target data, wherein the sampled data is a constant predicate and includes at least one;

[0045] A first construction module is configured to generate a template predicate based on the sampled data, and to construct a target template based on the template predicate;

[0046] A second construction module is used to perform data screening on the target data according to the constant predicate to construct a predicate set;

[0047] A generation module, configured to mine association rules based on the predicate set to generate a candidate rule set;

[0048] A determination module is used to determine valid rules in the total set of candidate rules based on the target data, and to determine valid data based on the valid rules.

[0049] The present application also discloses a device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the steps of the data processing method based on relational data as described above are implemented.

[0050] The present application also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method based on relational data as described above are implemented.

[0051] This application has the following advantages:

[0052] In an embodiment of the present application, target data is obtained and sampled data is determined based on the meaning of the target data, wherein the sampled data is a constant predicate and includes at least one; a template predicate is generated based on the sampled data, and a target template is constructed based on the template predicate; the target data is filtered based on the constant predicate to construct a predicate set; association rule mining is performed based on the predicate set to generate a candidate rule set; valid rules within the candidate rule set are determined based on the target data, and valid data is determined based on the valid rules. By proposing a data processing method to repair constants, when discovering rules with constants in large-scale relational data, it is not necessary to enumerate all possible constants and valid rules with constants can be discovered, thereby greatly improving the execution efficiency of rule discovery. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0054] Figure 1 This is a flowchart of a data processing method based on relational data provided by an embodiment of the present application;

[0055] Figure 2 This is a flowchart of a data processing method based on relational data provided by an embodiment of the present application;

[0056] Figure 3 This is a flowchart of a data processing method based on relational data provided by an embodiment of the present application;

[0057] Figure 4 This is a flowchart of a data processing method based on relational data provided by an embodiment of the present application;

[0058] Figure 5 This is a flowchart of a data processing method based on relational data provided by an embodiment of the present application;

[0059] Figure 6 This is a flowchart of a data processing method based on relational data provided by an embodiment of the present application;

[0060] Figure 7 This is a flowchart of a data processing method based on relational data provided by an embodiment of the present application;

[0061] Figure 8This is a structural block diagram of a data processing device based on relational data provided by an embodiment of the present application;

[0062] Figure 9 It is a structural diagram of a computer device provided by one embodiment of the present invention. DETAILED DESCRIPTION

[0063] To make the objectives, features, and advantages of this application more readily apparent, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It is apparent that the embodiments described are only a portion of the embodiments of this application, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments in this application without inventive effort are also within the scope of protection of this application.

[0064] It should be noted that, unlike CFD rules, the rules used in this invention are entity enhancing rules (REE). The basic component of REE is the predicate p, which is defined as follows:

[0065] p:=R(t)|tA◎c|tA◎sB|M(tA,sB)

[0066] Among them, ◎ is an operator, which can be equal to or not equal to.

[0067] R(t) means that t is a tuple variable in the relational table R.

[0068] tA represents the A attribute of variable t; M is a machine learning model. If tA and sB are related, the machine learning model will return true (correct), otherwise it returns false (wrong).

[0069] tA◎c carries a constant and is called a constant predicate.

[0070] tA◎sB does not carry a constant and is called a variable predicate.

[0071] M(tA,sB) is called the machine learning predicate.

[0072] Based on predicates, REE is defined as: X->e; where (1) X is the combination of multiple predicates, which is called the condition of this REE; (2) e is a predicate, which is called the result of this REE.

[0073] A specific REE example is as follows:

[0074] Express(t) ∧ Express(s) ∧ t.recipient = s.recipient ∧ t.address = "Shenzhen, Guangdong Province" -> s.zip = "510000"

[0075] The scenario described by this REE is that if the recipients of courier t and courier s are the same person, and the address of t is "Shenzhen City, Guangdong Province", then the postal code of s must be "510000".

[0076] REE rules can be discovered in relational data through depth-first or breadth-first search.

[0077] The most relevant to the REE rule is the CFD rule mentioned above; the CFD rule supports constant predicates and variable predicates with only one tuple variable, and can be seen as a special case of the REE rule.

[0078] The CFD-based rule mining algorithm also uses breadth-first or depth-first search methods to mine rules.

[0079] The overall technical solution of the present invention is divided into two steps: template mining and constant repair. In order to improve the mining efficiency, a part of the data is extracted from the full data D to form the sample data D s In the sample data D s Template mining is performed on D, while constant repair is performed on the full data D. Template mining comes first, and constant repair comes later. Template mining is performed first, and constant repair is performed based on the mined template.

[0080] Reference Figure 1 , showing a flowchart of the steps of a data processing method based on relational data provided by an embodiment of the present application;

[0081] A data processing method based on relational data, the method being used to repair target data with missing data segments through data relations and verify the validity of the repaired target data, the method comprising:

[0082] S110, acquiring target data, and performing data screening according to the meaning of the target data to determine sampled data, wherein the sampled data is a constant predicate and includes at least one;

[0083] S120, generating a template predicate according to the sampled data, and constructing a target template according to the template predicate;

[0084] S130, filtering the target data according to the constant predicate to construct a predicate set;

[0085] S140, performing association rule mining based on the total set of predicates to generate a total set of candidate rules;

[0086] S150: Determine valid rules in the total set of candidate rules based on the target data, and determine valid data based on the valid rules.

[0087] In an embodiment of the present application, target data is obtained and sampled data is determined based on the meaning of the target data, wherein the sampled data is a constant predicate and includes at least one; a template predicate is generated based on the sampled data, and a target template is constructed based on the template predicate; the target data is filtered based on the constant predicate to construct a predicate set; association rule mining is performed based on the predicate set to generate a candidate rule set; valid rules within the candidate rule set are determined based on the target data, and valid data is determined based on the valid rules. By proposing a data processing method to repair constants, when discovering rules with constants in large-scale relational data, it is not necessary to enumerate all possible constants and valid rules with constants can be discovered, thereby greatly improving the execution efficiency of rule discovery.

[0088] Next, a method for mining predicate combination rules based on reinforcement learning in this exemplary embodiment will be further described.

[0089] As described in step S110 , target data is acquired, and data is screened based on the meaning of the target data to determine sampled data, wherein the sampled data is a constant predicate and includes at least one.

[0090] In one embodiment of the present invention, the specific process of "obtaining target data, and performing data screening to determine sampled data based on the meaning of the target data, wherein the sampled data is a constant predicate and includes at least one" in step S110 can be further explained in combination with the following description.

[0091] Reference Figure 2 , showing a flowchart of the steps of a data processing method based on relational data provided by an embodiment of the present application;

[0092] As described in the following steps,

[0093] S210, obtaining data attributes in the target data;

[0094] S220, determining the word sense type of the target data according to the data attribute, wherein the word sense type includes a constant and a non-constant;

[0095] S230: Filter the target data whose word meaning type corresponds to the constant and determine it as the sampled data.

[0096] It should be noted that the data attributes in the target data are obtained; different target data have corresponding data attributes.

[0097] It should be noted that the semantic type of the target data is determined based on the data attributes, wherein the semantic type includes constants and no constants; the semantic type of the target data is determined by the data attributes; the semantic type includes constants and no constants.

[0098] It should be noted that the target data whose semantic type is the constant corresponding to the target data are screened and determined as the sampled data; the target data whose semantic type is the constant are screened and marked as the sampled data.

[0099] As described in step S120 , a template predicate is generated according to the sampled data, and a target template is constructed according to the template predicate.

[0100] In an embodiment of the present invention, the specific process of "generating a template predicate according to the sampled data, and constructing a target template according to the template predicate" in step S120 can be further explained in combination with the following description.

[0101] Reference Figure 3 , showing a flowchart of the steps of a data processing method based on relational data provided by an embodiment of the present application;

[0102] As described in the following steps,

[0103] S310, generating a template predicate according to the sampled data;

[0104] S320: When the template predicate has a valid value in the target data, determine that the template predicate is a valid predicate;

[0105] S330: Construct a target template according to the valid predicate.

[0106] It should be noted that a template predicate is generated based on the sampled data; wherein a predicate can be formed based on the data attributes of the sampled data; a number of predicates are generated based on the data attributes; the initial state of the predicate generated by the data attributes is an invalid predicate, and it is necessary to determine whether it is a template predicate through its corresponding constant value.

[0107] It should be noted that when verifying a template predicate, as long as at least one sub-data in the target data has a valid value in the data attribute of the template predicate, the predicate is considered a valid predicate, and the template REE, namely the target template, is formed through the valid predicate.

[0108] In a specific implementation, the template REE, ie, the target template, is a REE rule in which all constants in the REE are represented by a wildcard “_”.

[0109] It should be noted that wildcard characters can match any constant value.

[0110] In a specific implementation, the template REE corresponding to the sample REE is as follows:

[0111] Express(t)∧Express(s)∧t.recipient=s.recipient∧t.address=“_”->s.zip=:_”.

[0112] The benefit of template REE is that if one or more REE rules differ only in the constants of the constant predicates, they can be represented by the same template REE.

[0113] Multiple CFDs can be expressed as:

[0114] Express(t)∧t.Address = "_"->t.Zip = "_"

[0115] To distinguish multiple REE rules under the same template REE, each REE rule will have a pattern tuple attached to the template REE for constant assignment.

[0116] For example, the pattern tuple of CFD address = "Guangzhou City, Guangdong Province" -> postal code = "510000" is ("Guangzhou City, Guangdong Province", "510000").

[0117] Multiple pattern tuples form the pattern tableau of REE, as shown in Table 1:

[0118]

[0119] Table 1

[0120] As an example, consider constant predicates that differ only in the constant value of the same attribute.

[0121] In a specific implementation, such as t.address="Shenzhen City, Guangdong Province", t.address="Guangzhou City, Guangdong Province" and t.address="Dongguan City, Guangdong Province", only one template predicate is enumerated, namely t.address="_".

[0122] As described in step S330, constructing a target template according to the valid predicate;

[0123] In an embodiment of the present invention, the specific process of "constructing a target template according to the valid predicate" in step S330 can be further explained in combination with the following description.

[0124] Reference Figure 4 , showing a flowchart of the steps of a data processing method based on relational data provided by an embodiment of the present application;

[0125] As described in the following steps,

[0126] S410, combining the valid predicates to generate permutations and combinations;

[0127] S420, screening the permutations and combinations to determine a template predicate combination;

[0128] S430: Construct the target template according to the template predicate combination and the valid predicate.

[0129] It should be noted that the valid predicates are combined to generate a permutation combination; a number of valid predicates are permuted to generate a permutation combination; and a number of valid predicates are combined to form a permutation combination.

[0130] It should be noted that the permutations and combinations are screened to determine the template predicate combination; instead of enumerating all the permutations and combinations, the concept of free itemsets in the transaction database is used to preliminarily screen the permutations and combinations to determine the template predicate combination.

[0131] As an example, instead of enumerating all permutations and combinations between template predicates, the concept of free itemset in the transaction database is used to preliminarily screen all permutations and combinations between template predicates; only template predicate combinations that pass the screening will form a valid template REE, i.e., the target template.

[0132] In a specific implementation, for example, when the template predicate is t.address="_", there is a courier data in the sampled data, and the address is not a null value (null value means there is no data), then this sampled data has a valid value in the data attribute of this template predicate.

[0133] It should be noted that for a template REE (target template), as long as at least one set of data attributes satisfies the template REE (target template), the rule will be considered a valid rule candidate for subsequent rule verification. For example, if Express(t)∧t.Address="_"->t.ZipCode="_", there is a Express data item in the data, and both the address and zip code are not null values. Therefore, the data attributes of this data item satisfy the template REE (target template).

[0134] As described in step S130, the target data is screened according to the constant predicate to construct a predicate set;

[0135] In an embodiment of the present invention, the specific process of "screening the target data according to the constant predicate to construct a predicate set" in step S130 can be further explained in conjunction with the following description.

[0136] Reference Figure 5, showing a flowchart of the steps of a data processing method based on relational data provided by an embodiment of the present application;

[0137] As described in the following steps,

[0138] S510: Screen the target data according to the constant predicate to determine non-constant predicate data, wherein the non-constant predicate data is a non-constant predicate set;

[0139] S520, supplementing the non-constant predicate data with constant values ​​according to the target template to generate constant predicate data, wherein the constant predicate data is a constant predicate set;

[0140] S530: Construct the total predicate set according to the non-constant predicate set and the constant predicate set.

[0141] It should be noted that in D s That is, after mining the template REE on the sampled data, constant repair is performed on the full data D, i.e., the target data, by using the template REE. Among them, constant repair includes the following four main steps: (1) confirming the enumeration range with non-constant predicates; (2) using the template to supplement constants; (3) generating candidate rules; and (4) rule verification.

[0142] It should be noted that, the target data is screened based on the constant predicate to determine non-constant predicate data, wherein the non-constant predicate data is a non-constant predicate set; the constant predicate data in the target data is screened out by the constant predicate, thereby obtaining target data that is not constant predicate data, and the target data that is not constant predicate data is identified as non-constant predicate data.

[0143] It should be noted that, for the non-constant predicate data screened out, constants are supplemented according to the template predicates in the template REE, ie, the target template, wherein each template predicate is supplemented with constants, thereby constructing constant predicate data that can be enumerated.

[0144] As an example, for the template predicate t.address=“_”, we find all the address attribute values ​​in the data and fill them in the position of the wildcard “_” to form the constant predicate data.

[0145] It should be noted that, based on the constant predicate set and the non-constant predicate set, a total predicate set can be constructed, and the total predicate set includes all the non-constant predicates and all the constant predicates obtained above.

[0146] As an example, given a template REE, i.e., the target template, the sub-data in the entire data D, i.e., the target data, are screened using the constant predicates in the template REE. Only the sub-data that satisfy the non-constant predicates will participate in the next step of constant supplementation. In other words, only the data with non-constant predicates will participate in the next step of constant supplementation. This approach avoids expensive constant supplementation in the entire data D, and greatly improves the execution efficiency of the algorithm while ensuring completeness.

[0147] As described in step S140, association rule mining is performed based on the total set of predicates to generate a candidate rule set;

[0148] In one embodiment of the present invention, the specific process of "mining association rules based on the total set of predicates to generate a candidate rule set" in step S140 can be further explained in conjunction with the following description.

[0149] Reference Figure 6 , showing a flowchart of the steps of a data processing method based on relational data provided by an embodiment of the present application;

[0150] As described in the following steps,

[0151] S610, performing a depth-first search based on the total set of predicates to generate a first candidate rule set; or, S620, performing a breadth-first search based on the total set of predicates to generate a second candidate rule set;

[0152] S630: Generate the total set of candidate rules according to the first candidate rule set or the second candidate rule set.

[0153] It should be noted that a depth-first search or a breadth-first search is performed based on the total set of predicates to obtain a number of candidate rules, and the total set of candidate rules is generated from the several candidate rules.

[0154] As an example, the template REE, ie, the target template, is re-mined based on depth-first or breadth-first rules on the total set of predicates to obtain candidate rules.

[0155] As described in step S150, determining valid rules in the total set of candidate rules based on the target data, and determining valid data based on the valid rules;

[0156] In one embodiment of the present invention, the specific process of "determining valid rules in the candidate rule set according to the target data, and determining valid data according to the valid rules" in step S150 can be further explained in combination with the following description.

[0157] Reference Figure 7 , showing a flowchart of the steps of a data processing method based on relational data provided by an embodiment of the present application;

[0158] As described in the following steps,

[0159] S710, obtaining each sub-candidate rule in the total set of candidate rules;

[0160] S720: Verify each of the candidate sub-rules based on the target data to determine the validity of each of the candidate sub-rules; wherein, when there is a sub-target data in the target data corresponding to the current candidate sub-rule, the current candidate sub-rule is determined to be a valid rule;

[0161] S730: Obtain the sub-target data corresponding to the valid rule, and mark the sub-target data as the valid data.

[0162] It should be noted that the valid rules in the total set of candidate rules are determined based on the target data, and the valid data are determined based on the valid rules; for a candidate rule, it is necessary to determine whether it is a valid rule through the full data D, ie, the target data.

[0163] As an example, for each candidate rule, the validity of the candidate rule is verified on the full data D, ie, the target data; when there is a sub-target data in the target data corresponding to the current sub-candidate rule, then the candidate rule is valid.

[0164] In a specific implementation, for example, if the candidate rule is Express(t) ∧ Express(s) ∧ t.Recipient = s.Recipient ∧ t.Address = "Shenzhen, Guangdong Province" -> s.ZipCode = "510000", if the data contains Express t and Express s with the same recipient, and Express t's address is in "Shenzhen, Guangdong Province", then Express s's zip code must be "510000". Therefore, the data corresponding to Express t and Express s satisfies the candidate rule and the candidate rule is considered valid. The valid rule will form the final output.

[0165] Technical effects of the present invention:

[0166] In CFD, there is no concept similar to templates. Therefore, it is necessary to enumerate all constants on the entire data to generate valid CFD rules with constants. Instead, by using the definition of template REE, a small sample data D is first extracted from the global data. s Template mining is performed. Due to the small amount of data, this process is very fast compared to the CFD method of constant enumeration on global data. Secondly, using the mined template REE, there is no need to perform expensive constant enumeration on the entire data D. Instead, we only need to enumerate constants that are likely to form valid rules. Constants that are unlikely to form valid rules will be excluded during the range confirmation before enumeration, avoiding the possibility of redundant and invalid operations.

[0167] We compared the accuracy and mining efficiency of three rule mining methods in multiple public data, including: (1) s After template mining, the rule mining method of constant repair in D is performed; (2) Directly in D s and (3) a method for performing rule mining directly on D.

[0168] By comparison with the above method (2), the mining method of the present invention improves the rule recall rate by 2%; after constant repair, the mined rules are more accurate.

[0169] Compared with method (3), the present invention can improve the running efficiency by an average of 12.2 times. On a large DBLP dataset with 3 relational tables, 18 attributes, and 1.8 million data items, the running time of the present invention is 406 seconds, while the running time of method (3) is 2096 seconds; in other words, the mining efficiency is higher.

[0170] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0171] Reference Figure 8 , shows a structural block diagram of a data processing device based on relational data provided by an embodiment of the present application;

[0172] The apparatus is used for mining and repairing rules of constants in relational data, wherein the relational data includes full data and sampled data in the full data, specifically including:

[0173] An acquisition module 810 is configured to acquire target data and perform data screening based on the meaning of the target data to determine sampled data, wherein the sampled data is a constant predicate and includes at least one;

[0174] A first construction module 820 is configured to generate a template predicate based on the sampled data, and to construct a target template based on the template predicate;

[0175] A second construction module 830 is configured to filter the target data according to the constant predicate and construct a predicate set;

[0176] A generating module 840 is configured to perform association rule mining based on the predicate set to generate a candidate rule set;

[0177] The determination module 850 is configured to determine valid rules in the total set of candidate rules based on the target data, and determine valid data based on the valid rules.

[0178] In one embodiment of the present invention, the acquisition module 810 includes:

[0179] A first acquisition submodule is used to acquire data attributes in the target data;

[0180] A first determining submodule is configured to determine the word sense type of the target data according to the data attribute, wherein the word sense type includes a constant and a non-constant;

[0181] The second determining submodule is configured to screen the target data whose semantic type corresponds to the constant and determine the target data as the sampled data.

[0182] In one embodiment of the present invention, the first building module 820 includes:

[0183] A first generating submodule, configured to generate a template predicate based on the sampled data;

[0184] a third determining submodule, configured to determine that the template predicate is a valid predicate when a valid value exists for the template predicate in the target data;

[0185] The first construction submodule is configured to construct a target template according to the valid predicate.

[0186] In one embodiment of the present invention, the first building block includes:

[0187] A first generating unit, configured to generate permutations and combinations by combining the valid predicates;

[0188] a first determining unit, configured to screen the permutations and combinations to determine a template predicate combination;

[0189] The first construction unit is configured to construct the target template according to the template predicate combination and the valid predicate.

[0190] In one embodiment of the present invention, the second building module 830 includes:

[0191] a first screening submodule, configured to screen the target data according to the constant predicate to determine non-constant predicate data, wherein the non-constant predicate data is a non-constant predicate set;

[0192] a second generating submodule, configured to supplement the non-constant predicate data with constant values ​​according to the target template to generate constant predicate data, wherein the constant predicate data is a constant predicate set;

[0193] The second construction submodule is configured to construct the total predicate set according to the non-constant predicate set and the constant predicate set.

[0194] In one embodiment of the present invention, the generating module 840 includes:

[0195] A third generating submodule is configured to generate a first candidate rule set by performing a depth-first search based on the total set of predicates; or

[0196] a fourth generating submodule, configured to generate a second candidate rule set by performing a breadth-first search based on the total set of predicates;

[0197] The fifth generating submodule is configured to generate the total set of candidate rules based on the first candidate rule set or the second candidate rule set.

[0198] In one embodiment of the present invention, the determining module 850 includes:

[0199] A second acquisition submodule is used to acquire each sub-candidate rule in the total set of candidate rules;

[0200] a fourth determination submodule, configured to verify the validity of each of the candidate sub-rules based on the target data; wherein, when there is a sub-target data in the target data corresponding to the current candidate sub-rule, the current candidate sub-rule is determined to be a valid rule;

[0201] The third acquisition submodule is configured to acquire the sub-target data corresponding to the valid rule and mark the sub-target data as the valid data.

[0202] Reference Figure 9 , showing a computer device of a data processing method based on relational data of the present invention, which may specifically include the following:

[0203] The computer device 12 is a general-purpose computing device. The components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0204] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor, or a local bus 18 that utilizes any of a variety of bus 18 architectures. Examples of such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.

[0205] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0206] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although Figure 9 Not shown, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to bus 18 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 configured to perform the functions of various embodiments of the present invention.

[0207] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in a memory. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0208] The computer device 12 may also communicate with one or more external devices 14 (e.g., a keyboard, a pointing device, a display 24, a camera, etc.), one or more devices that enable an operator to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, a modem, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN)), a wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Figure 9Not shown, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34.

[0209] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing a data processing method based on relational data provided by an embodiment of the present invention.

[0210] That is, when the processing unit 16 executes the above program, it achieves the following: acquiring target data, and filtering data based on the semantics of the target data to determine sampled data, wherein the sampled data is a constant predicate and includes at least one; generating template predicates based on the sampled data, and constructing a target template based on the template predicates; filtering the target data based on the constant predicates to construct a predicate set; performing association rule mining based on the predicate set to generate a candidate rule set; determining valid rules within the candidate rule set based on the target data, and determining valid data based on the valid rules. Constants are repaired by proposing a data processing method.

[0211] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, a data processing method based on relational data as provided in all embodiments of the present application is implemented:

[0212] That is, when the program is executed by a processor, it is implemented as follows: acquiring target data, filtering data based on the semantics of the target data to determine sampled data, wherein the sampled data is a constant predicate and includes at least one; generating template predicates based on the sampled data, and constructing a target template based on the template predicates; filtering the target data based on the constant predicates to construct a predicate set; performing association rule mining based on the predicate set to generate a candidate rule set; determining valid rules within the candidate rule set based on the target data, and determining valid data based on the valid rules. Constants are repaired by proposing a data processing method.

[0213] Any combination of one or more computer-readable media may be employed. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, apparatus, or device.

[0214] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0215] The computer program code for performing the operations of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the operator's computer, partially on the operator's computer, as a separate software package, partially on the operator's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the operator's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.

[0216] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.

[0217] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.

[0218] The above is a detailed introduction to a data processing method and device based on relational data provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technical personnel in this field, according to the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data processing method based on relational data, the method is used to repair target data with missing data segments through data relations and verify the validity of the repaired target data, characterized in that: include: Acquire target data, and perform data screening based on the meaning of the target data to determine sampled data, wherein the sampled data is a constant predicate and includes at least one, and the sampled data is text data; generating a template predicate according to the sampled data, and constructing a target template according to the template predicate; The specific steps of generating a template predicate based on the sampled data and constructing a target template based on the template predicate are: generating a template predicate based on the sampled data; determining that the template predicate is a valid predicate when the template predicate has a valid value in the target data; and constructing a target template based on the valid predicate; The specific steps of constructing the target template based on the valid predicates are: combining the valid predicates to generate permutations and combinations; screening the permutations and combinations to determine template predicate combinations; constructing the target template based on the template predicate combinations and the valid predicates; when verifying the template predicates, if at least one sub-data in the target data has a valid value for the data attribute of the template predicate, the predicate is regarded as a valid predicate, and the valid predicates constitute a template REE, which is the target template; Performing data screening on the target data according to the constant predicate to construct a predicate set; Performing association rule mining based on the total set of predicates to generate a total set of candidate rules; Determining valid rules in the total set of candidate rules based on the target data, and determining valid data based on the valid rules; The specific steps of determining the valid rules in the total set of candidate rules based on the target data, and determining the valid data based on the valid rules are: obtaining each sub-candidate rule in the total set of candidate rules; verifying each sub-candidate rule based on the target data to determine the validity of each sub-candidate rule; wherein, when there is a sub-target data in the target data corresponding to the current sub-candidate rule, the current sub-candidate rule is determined to be a valid rule; obtaining the sub-target data corresponding to the valid rule, and marking the sub-target data as the valid data.

2. The method according to claim 1, characterized in that The step of obtaining target data and screening the target data according to the meaning of the target data to determine sampled data, wherein the sampled data is a constant predicate, and at least one step is included, including: Obtaining data attributes in the target data; Determining a word meaning type of the target data according to the data attribute, wherein the word meaning type includes a word meaning type with a constant and a word meaning type without a constant; The target data having the word meaning type corresponding to the constant is screened and determined as the sampled data.

3. The method according to claim 1, characterized in that The step of filtering the target data according to the constant predicate and constructing a predicate set includes: Performing data screening on the target data according to the constant predicate to determine non-constant predicate data, wherein the non-constant predicate data is a non-constant predicate set; Supplementing the non-constant predicate data with constant values ​​according to the target template to generate constant predicate data, wherein the constant predicate data is a constant predicate set; The total predicate set is constructed according to the non-constant predicate set and the constant predicate set.

4. The method according to claim 1, wherein The step of performing association rule mining based on the total set of predicates to generate a candidate rule set includes: Performing a depth-first search based on the total set of predicates to generate a first candidate rule set; or, Performing a breadth-first search based on the total set of predicates to generate a second candidate rule set; The total set of candidate rules is generated according to the first candidate rule set or the second candidate rule set.

5. A data processing device based on relational data, the device is used to repair target data with missing data segments through data relations and verify the validity of the repaired target data, characterized in that: include: An acquisition module, configured to acquire target data and perform data screening to determine sampled data based on the meaning of the target data, wherein the sampled data is a constant predicate and includes at least one, and the sampled data is text data; A first construction module is configured to generate a template predicate based on the sampled data, and to construct a target template based on the template predicate; The specific steps of generating a template predicate based on the sampled data and constructing a target template based on the template predicate are: generating a template predicate based on the sampled data; determining that the template predicate is a valid predicate when the template predicate has a valid value in the target data; and constructing a target template based on the valid predicate; The specific steps of constructing the target template based on the valid predicates are: combining the valid predicates to generate permutations and combinations; screening the permutations and combinations to determine template predicate combinations; constructing the target template based on the template predicate combinations and the valid predicates; when verifying the template predicates, if at least one sub-data in the target data has a valid value for the data attribute of the template predicate, the predicate is regarded as a valid predicate, and the valid predicates constitute a template REE, which is the target template; A second construction module is used to perform data screening on the target data according to the constant predicate to construct a predicate set; A generation module, configured to mine association rules based on the predicate set to generate a candidate rule set; a determination module, configured to determine valid rules in the total set of candidate rules based on the target data, and determine valid data based on the valid rules; The specific steps of determining the valid rules in the total set of candidate rules based on the target data, and determining the valid data based on the valid rules are: obtaining each sub-candidate rule in the total set of candidate rules; verifying each sub-candidate rule based on the target data to determine the validity of each sub-candidate rule; wherein, when there is a sub-target data in the target data corresponding to the current sub-candidate rule, the current sub-candidate rule is determined to be a valid rule; obtaining the sub-target data corresponding to the valid rule, and marking the sub-target data as the valid data.

6. A computer device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 4 when executed by the processor.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images