A data screening rule verification method and device based on multi-round sampling

Abstract

The present application provides a data screening rule verification method and device based on multi-round sampling, which is used to screen target data in a large database to determine data screening rules. The method includes obtaining the target data and determining the corresponding data relationship table based on the target data; constructing a relationship graph G based on the tuple; sampling the vertices in the vertex set V K times to generate K sampled data; searching the K sampled data layer by layer to construct an overall synchronous parallel computing model; and screening the K sampled data based on the scheduling node and several working nodes to generate target data screening rules. Multi-round sampling can improve the recall rate of data screening rules; in the process of constructing the relationship graph G, the influence of predicates is considered in the edge construction, so that useful data that can be used for rule discovery will be sampled with a higher priority; ensuring the accuracy of sampling; and improving the efficiency of data screening operations.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a data screening rule verification method and device based on multiple rounds of sampling. Background Art

[0002] Discovering rules in relational data is a time-consuming and laborious process. In today's big data era, the size of data is growing at an exponential rate, which brings unprecedented challenges to rule mining. Using existing algorithms to mine rules may take users several days or even weeks, which is far from meeting the requirements of many scenarios.

[0003] Existing sampling technology is usually uniform sampling on a single table. After sampling, single-machine rule mining is performed on the sampled data.

[0004] Uniform sampling technology has the following main disadvantages: (1) Uniform sampling is mainly performed on single-table rules, which greatly reduces the effect of uniform sampling; (2) The existing uniform sampling is basically single-round sampling, which may cause a large loss of rules, so the recall rate of rules is very low; (3) Uniform sampling does not distinguish whether the data is useful or not, and all data are treated equally. Such a method is unscientific. Some useful data that can be used for rule discovery should be sampled with a higher priority; (4) The existing uniform sampling is basically a heuristic method and cannot guarantee the accuracy of sampling; and the existing rule mining algorithms are mainly stand-alone algorithms; even if more computing resources are used, it is not necessarily guaranteed to improve the computational efficiency of rule mining. Summary of the invention

[0005] In view of the above problems, the present application is proposed to provide a data screening rule verification method based on multi-round sampling and a device thereof to overcome the above problems or at least partially solve the above problems, including:

[0006] A data screening rule verification method based on multiple rounds of sampling is used to screen target data in a large database to determine data screening rules, the method comprising:

[0007] Acquire the target data, and determine a corresponding data relationship table according to the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple;

[0008] Constructing a relationship graph G according to the tuple, wherein the relationship graph G includes a vertex set V and an edge set E;

[0009] Sampling the vertices in the vertex set V K times to generate K sampling data;

[0010] Conducting layer-by-layer search on the K sampled data to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and a plurality of working nodes;

[0011] The K sampled data are screened according to the scheduling node and the plurality of working nodes to generate target data screening rules.

[0012] Furthermore, the step of constructing a relationship graph G according to the tuple, wherein the relationship graph G includes a vertex set V and an edge set E, comprises:

[0013] generating a plurality of vertices according to the tuple;

[0014] A plurality of the vertices are connected to generate a plurality of edges;

[0015] Constructing an edge set E based on the plurality of vertices and the plurality of edges; wherein, when an edge e in the edge set E connects a vertex t and another vertex s in the vertex set v respectively, a tuple pair corresponding to the vertex t and the vertex s satisfies at least one equality predicate;

[0016] The relationship graph G is constructed based on the vertex set V and the edge set E.

[0017] Furthermore, the step of performing K sampling on the vertices in the vertex set V to generate K sampling data includes:

[0018] Performing K random walk sampling on the vertices in the vertex set V to generate K sampling data; or,

[0019] Perform K breadth-first sampling on the vertices in the vertex set V to generate K sampling data.

[0020] Furthermore, the step of performing K random walk sampling on the vertices in the vertex set V to generate K sampling data includes:

[0021] Obtaining an initial vertex in the vertex set V by uniform sampling;

[0022] Iteratively sampling the initial vertex according to the random walk sampling to generate a plurality of sampling tuples, wherein each step of the random walk sampling stops or moves to an adjacent vertex of the current vertex with a preset probability ∈, and the iterative process of the iterative sampling is less than or equal to a preset sample size;

[0023] The K sampling data are generated according to a number of the sampling tuples.

[0024] Furthermore, the step of performing K breadth-first sampling on the vertices in the vertex set V to generate K sampling data includes:

[0025] Acquire an initial tuple pair in the vertex set V by uniform sampling, wherein the initial tuple pair includes vertex z and vertex y;

[0026] Iteratively sampling the initial tuple pair according to the breadth-first sampling to generate a plurality of adjacent vertices, wherein a target distance of the adjacent vertices is less than or equal to a preset distance between the vertex z and the vertex y;

[0027] The K sampling data are generated according to a number of the adjacent vertices.

[0028] Furthermore, the step of performing data screening on K sampled data according to the scheduling node and the plurality of working nodes to generate target data screening rules comprises:

[0029] The scheduling node generates a plurality of tasks corresponding to the K sampled data, wherein each of the plurality of tasks is a triplet, and the triplet includes a selected predicate set, a to-be-selected predicate set, and a valid predicate;

[0030] According to the scheduling node, a plurality of the tasks are evenly distributed to the working nodes, wherein the working node includes at least one; when there are tasks greater than a preset task amount in the working node, half of the task amount in the working node is distributed to other working nodes; or, when there are only tasks with a preset task amount of 1 in the working node, the data corresponding to the tasks are split;

[0031] The target data screening rule is generated according to the working node, the scheduling node and the K sampled data.

[0032] Furthermore, the step of generating the target data screening rule based on the working node, the scheduling node and the K sampled data includes:

[0033] Acquire K of the sampled data, and determine all the candidate predicates in the candidate predicate set in the K of the sampled data;

[0034] The working node iteratively adds the candidate predicate to the selected predicate set;

[0035] When the set of predicates to be selected is an empty set, the working node stops iterating, and determining that the set of predicates to be selected is an empty set is the first data screening rule; or,

[0036] When at least one of the candidate predicates in the candidate predicate set is related to the selected predicate set, the working node stops iterating, and determines that at least one of the candidate predicates in the candidate predicate set is related to the selected predicate set as the second data screening rule;

[0037] The first data screening rule and the second data screening rule are summarized in the scheduling node, and the scheduling node generates the target data screening rule according to the first data screening rule and the second data screening rule.

[0038] The present application also discloses a data screening rule verification device based on multiple rounds of sampling, which is used to perform data screening on target data in a large database to determine data screening rules. The device includes:

[0039] An acquisition module, used for acquiring the target data, and determining a corresponding data relationship table according to the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple;

[0040] A first construction module, configured to construct a relationship graph G according to the tuple, wherein the relationship graph G includes a vertex set V and an edge set E;

[0041] A first generating module, configured to perform K sampling on the vertices in the vertex set V to generate K sampling data;

[0042] A second construction module is used to search the K sampled data layer by layer to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and a plurality of working nodes;

[0043] The second generating module is used to generate target data screening rules by performing data screening on the K sampled data according to the scheduling node and the plurality of working nodes.

[0044] The present application also discloses a device, including a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, the steps of the data screening rule verification method based on multiple rounds of sampling as described above are implemented.

[0045] The present application also discloses a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data screening rule verification method based on multiple rounds of sampling are implemented as described above.

[0046] This application has the following advantages:

[0047] In an embodiment of the present application, the target data is obtained, and the corresponding data relationship table is determined according to the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple; a relationship graph G is constructed according to the tuple, wherein the relationship graph G includes a vertex set V and an edge set E; the vertices in the vertex set V are sampled K times to generate K sampled data; the K sampled data are searched layer by layer to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and several working nodes; the K sampled data are screened according to the scheduling node and several working nodes to generate target data screening rules. The proposed sampling method can be performed on cross-table rules such as REE that support multiple tuple variables; the sampling method is multi-round sampling, which can improve the recall rate of data screening rules; in the process of constructing the relationship graph G, the influence of predicates is considered in the edge construction, so that useful data that can be used for rule discovery will be sampled with higher priority; the accuracy of sampling is ensured; thereby using more computing resources and improving the computational efficiency of rule mining. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the present application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0049] Figure 1 It is a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0050] Figure 2 It is a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0051] Figure 3 It is a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0052] Figure 4 It is a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0053] Figure 5 It is a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0054] Figure 6 It is a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0055] Figure 7 It is a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0056] Figure 8 It is a structural block diagram of a data screening rule verification device based on multi-round sampling provided by an embodiment of the present application;

[0057] Fig. 9 It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the objects, features and advantages of the present application more obvious and understandable, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0059] It should be noted that the rules used in the present invention are entity enhancing rules (hereinafter referred to as REE). The basic component of REE is the predicate p, which is defined as follows:

[0060] p:=R(t)|tA◎c|tA◎sB|M(tA,sB)

[0061] Among them, ◎ is an operator, which can be equal or not equal; R(t) means that t is a tuple variable in the relational table R; tA represents the A attribute of variable t; M is a machine learning model, if tA and sB are related, then the machine learning model will return true (correct), otherwise it returns false (wrong). tA◎c with a constant is called a constant predicate; tA◎sB without a constant is called a variable predicate; M(tA,sB) is called a machine learning predicate.

[0062] Based on predicates, REE is defined as: X->e, where (1) X is a combination of multiple predicates, called the condition of this REE; (2) e is a predicate, called the result of this REE.

[0063] A specific REE example is as follows:

[0064] Express(t)∧Express(s)∧t.Recipient=s.Recipient∧t.Address=“Shenzhen City, Guangdong Province”->s.Postal Code=“510000”

[0065] The scenario described by this REE is that if the recipients of courier t and courier s are the same person, and the address of courier t is in "Shenzhen City, Guangdong Province", then the postal code of courier s must be "510000".

[0066] Existing rule mining algorithms discover REEs in data through a depth-first or breadth-first stand-alone approach. Their essence is a process of enumerating all permutations and combinations of predicates on the entire data D. For all possible predicates, any one or more of them can form a valid rule with the REE result e. Therefore, in order to perform rule mining, existing methods need to try all permutations and combinations of predicates. When the data scale is large, the efficiency of such a brute force stand-alone algorithm is undoubtedly very low.

[0067] The goal of this invention is to propose a parallel rule discovery technology based on a new sampling method; under the premise of ensuring sampling accuracy, it supports parallel discovery of cross-table rules such as REE in big data, and solves the efficiency problem of rule mining in large-scale data.

[0068] By performing rule discovery on the full data D, the process of rule discovery will take a lot of time due to the large amount of data. Therefore, a method has been proposed to sample a small data set D from the full data D. s ;D s The size of D may be only one tenth or even one hundredth of D; instead of performing rule discovery on the entire data D, s Although rule discovery on sampled data may lose some rules, it can greatly improve the efficiency of rule discovery.

[0069] Reference Figure 1 , showing a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0070] A data screening rule verification method based on multiple rounds of sampling is used to screen target data in a large database to determine data screening rules, the method comprising:

[0071] S110, acquiring the target data, and determining a corresponding data relationship table according to the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple;

[0072] S120, constructing a relationship graph G according to the tuple, wherein the relationship graph G includes a vertex set V and an edge set E;

[0073] S130, performing K sampling on the vertices in the vertex set V to generate K sampling data;

[0074] S140, searching the K sample data layer by layer to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and a plurality of working nodes;

[0075] S150, performing data screening on the K sampled data according to the scheduling node and the plurality of working nodes to generate target data screening rules.

[0076] In an embodiment of the present application, the target data is obtained, and the corresponding data relationship table is determined according to the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple; a relationship graph G is constructed according to the tuple, wherein the relationship graph G includes a vertex set V and an edge set E; the vertices in the vertex set V are sampled K times to generate K sampled data; the K sampled data are searched layer by layer to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and several working nodes; the K sampled data are screened according to the scheduling node and several working nodes to generate target data screening rules. The proposed sampling method can be performed on cross-table rules such as REE that support multiple tuple variables; the sampling method is multi-round sampling, which can improve the recall rate of data screening rules; in the process of constructing the relationship graph G, the influence of predicates is considered in the edge construction, so that useful data that can be used for rule discovery will be sampled with higher priority; the accuracy of sampling is ensured; thereby using more computing resources and improving the computational efficiency of rule mining.

[0077] Next, a data screening rule verification method based on multiple rounds of sampling in this exemplary embodiment will be further described.

[0078] It should be noted that the purpose of data sampling is to extract a portion of sampled data D for rule discovery from the entire data D, i.e., the target data. s Specifically, extract the sample data D from the full data D s The problem is transformed into a problem of sampling vertices in the relational graph G.

[0079] It should be noted that the target data is obtained, and the corresponding data relationship table is determined based on the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple; that is, several tuples are determined through the data relationship table corresponding to the target data, i.e., the full data D.

[0080] As described in step S120, a relationship graph G is constructed based on the tuple, wherein the relationship graph G includes a vertex set V and an edge set E.

[0081] In one embodiment of the present invention, the specific process of "constructing a relationship graph G according to the tuple, wherein the relationship graph G includes a vertex set V and an edge set E" in step S120 can be further explained in combination with the following description.

[0082] Reference Figure 2 , showing a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0083] As described in the following steps,

[0084] S210, generating a plurality of vertices according to the tuple;

[0085] S220, connecting a plurality of the vertices to generate a plurality of edges;

[0086] S230, constructing an edge set E based on the plurality of vertices and the plurality of edges; wherein, when an edge e in the edge set E connects a vertex t and another vertex s in the vertex set v respectively, a tuple pair corresponding to the vertex t and the vertex s satisfies at least one equality predicate;

[0087] S240. Construct the relationship graph G based on the vertex set V and the edge set E.

[0088] It should be noted that a number of vertices are generated based on a number of tuples, and a number of vertices are generated from a number of tuples, wherein each vertex corresponds to a tuple in the full data D.

[0089] It should be noted that a number of the vertices determine a number of edges, and different vertices are connected to form a number of edges.

[0090] It should be noted that an edge set E is constructed based on a number of the vertices and a number of the edges; wherein, when an edge e in the edge set E respectively connects a vertex t and another vertex s in the vertex set V, the tuple pair corresponding to the vertex t and the vertex s satisfies at least one equality predicate; a number of vertices are aggregated to form a vertex set V; and a number of edges are aggregated to form an edge set E.

[0091] It should be noted that the relationship graph G is constructed based on the vertex set V and the edge set E; thereby constructing a relationship graph G=(V, E) for the entire data D.

[0092] As described in step S130, K samplings are performed on the vertices in the vertex set V to generate K sampling data.

[0093] In an embodiment of the present invention, the specific process of "sampling the vertices in the vertex set V K times to generate K sampling data" in step S130 can be further explained in combination with the following description.

[0094] Reference Figure 3 , showing a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0095] As described in the following steps,

[0096] S310, performing K random walk sampling on the vertices in the vertex set V to generate K sampling data; or,

[0097] S320 , performing K breadth-first sampling on the vertices in the vertex set V to generate K sampling data.

[0098] It should be noted that based on the constructed relationship graph G, either random walk sampling or breadth-first sampling can be used to sample the vertices in the relationship graph G. The specific use of random walk sampling or breadth-first sampling is random, and the user can decide whether to sample only in the random walk method, only in the breadth-first method, or alternate between random walk sampling and breadth-first sampling.

[0099] In a specific implementation, single-round sampling has the disadvantage of poor recall, that is, a rule that is globally applicable to all data D may not be applicable to sample data D. s Therefore, adopting a multi-round sampling strategy can improve the recall rate. By running K single-round sampling methods (random walk sampling or breadth-first sampling), K sampled data are obtained. We perform rule discovery on each sampled data (rule discovery refers to discovering REEs in the sampled data), and then combine the discovered REEs.

[0100] As described in step S310, K random walk samplings are performed on the vertices in the vertex set V to generate K sampling data.

[0101] In an embodiment of the present invention, the specific process of step S310 of "performing K random walk sampling on the vertices in the vertex set V to generate K sampling data" can be further explained in combination with the following description.

[0102] Reference Figure 4 , showing a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0103] As described in the following steps,

[0104] S410, obtaining an initial vertex in the vertex set V by uniform sampling;

[0105] S420, iteratively sampling the initial vertex according to the random walk sampling to generate a plurality of sampling tuples, wherein each step of the random walk sampling stops or moves to an adjacent vertex of the current vertex with a preset probability ∈, and the iterative process of the iterative sampling is less than or equal to a preset sample size;

[0106] S430 , generating the K sampling data according to a number of the sampling tuples.

[0107] It should be noted that an initial vertex is obtained by uniform sampling in the vertex set V. First, a vertex is uniformly sampled in the vertex set V and the vertex is marked as the initial vertex t, so that several tuples included in the full data D can be sampled with the same probability.

[0108] It should be noted that the initial vertex is iteratively sampled according to the random walk sampling to generate a number of sampling tuples, and more sampling tuples are obtained by iteratively sampling from the initial vertex t through random walk sampling.

[0109] As an example, for random walk sampling, first select an initial vertex t from the vertex set V based on uniform sampling, so that all tuples in the full data D are sampled with the same probability; then iteratively sample more sampling tuples starting from the initial vertex t by simulating random walks, with the longest walk length being len; at each step, the random walk will terminate with probability ∈, or move to the adjacent vertex of the current vertex with probability (1-∈); all vertices in the walk sequence will be included in the sampled data D s The whole process is iterated until |D s |Exceeded maximum sample size.

[0110] As described in step S320, K breadth-first sampling is performed on the vertices in the vertex set V to generate K sampling data.

[0111] In an embodiment of the present invention, the specific process of step S320 of "performing K breadth-first sampling on the vertices in the vertex set V to generate K sampling data" can be further explained in combination with the following description.

[0112] Reference Figure 5 , showing a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0113] As described in the following steps,

[0114] S510, obtaining an initial tuple pair in the vertex set V by uniform sampling, the initial tuple pair including vertex z and vertex y;

[0115] S520, iteratively sampling the initial tuple pair according to the breadth-first sampling to generate a plurality of adjacent vertices, wherein a target distance of the adjacent vertices is less than or equal to a preset distance between the vertex z and the vertex y;

[0116] S530: Generate the K sampling data according to a number of the adjacent vertices.

[0117] It should be noted that an initial tuple pair is obtained by uniform sampling in the vertex set V, and the initial tuple pair includes vertex z and vertex y; the initial tuple pair is obtained by uniform sampling in the vertex set V<y,z> .

[0118] It should be noted that, according to the breadth-first sampling, the initial tuple pair is iteratively sampled to generate a number of adjacent vertices; the initial tuple pair is iteratively sampled by the breadth-first sampling.<y,z> Get vertex z and adjacent nodes whose distances between vertices do not exceed a preset distance, wherein the adjacent nodes include at least one.

[0119] As an example, for breadth-first sampling, by uniformly sampling a tuple pair in the vertex set V<y,z> , and then use breadth-first sampling to get the adjacent nodes whose distance between vertex z and vertex y does not exceed (len-1); that is, if<y,z> If a REE of length len is matched, breadth-first sampling will ensure that at least one data matching the REE will be sampled.

[0120] As described in step S140, the K sample data are searched layer by layer to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and a plurality of working nodes;

[0121] It should be noted that, based on the K sampled data, an overall synchronous parallel computing model is constructed by layer-by-layer search, wherein the overall synchronous parallel computing model includes a scheduling node and several working nodes; by performing a parallel rule mining algorithm, i.e. layer-by-layer search, on the K sampled data, an overall synchronous parallel computing model is constructed, wherein the overall synchronous parallel computing model includes a scheduling node and several working nodes.

[0122] As described in step S150, data screening is performed on the K sampled data according to the scheduling node and the plurality of working nodes to generate target data screening rules.

[0123] In one embodiment of the present invention, the specific process of step S150 "generating target data screening rules by performing data screening on the K sampled data according to the scheduling node and the plurality of working nodes" can be further explained in combination with the following description.

[0124] Reference Figure 6 , showing a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0125] As described in the following steps,

[0126] S610, the scheduling node generates a plurality of tasks corresponding to the K sampled data, wherein each of the plurality of tasks is a triplet, and the triplet includes a selected predicate set, a to-be-selected predicate set, and a valid predicate;

[0127] S620, according to the scheduling node, evenly distribute a plurality of the tasks to the working nodes, wherein the working node includes at least one; when there are tasks greater than a preset task amount in the working node, half of the task amount in the working node is distributed to other working nodes; or, when there are only tasks with a preset task amount of 1 in the working node, the data corresponding to the tasks is split;

[0128] S630: Generate the target data screening rule according to the working node, the scheduling node and the K sampled data.

[0129] It should be noted that the scheduling node generates a number of tasks based on the K sampled data, wherein each of the tasks is a triplet, and the triplet includes a selected predicate set, a to-be-selected predicate set and a discovery result; the scheduling node generates a number of tasks corresponding to the sampled data.

[0130] It should be noted that, according to the scheduling node, several tasks are evenly distributed to several working nodes to generate several discovery rules; the scheduling node then evenly distributes several tasks to several working nodes, and generates several discovery rules, namely REE results e, through several working nodes.

[0131] In a specific implementation, the parallel rule mining algorithm is based on layer-by-layer search and consists of a scheduler, i.e., a scheduling node, and several workers, i.e., working nodes. Under the overall synchronous parallel computing model, the scheduler, i.e., the scheduling node, is responsible for the generation and distribution tasks, the maintenance of rules, and load balancing; while the workers, i.e., the working nodes, are responsible for discovering rules in parallel. The overall computing is divided into multiple supersteps, where each superstep corresponds to each level in the rule mining.

[0132] Specifically, the scheduler first generates several tasks based on the REE result e that needs to be discovered. Each task consists of a triple, where the triple is marked as <P sel , P re , e>. Each task consists of a triple: <P sel , P re , e>; where P sel What is stored is the predicate that has been selected to form the REE condition, that is, the selected predicate, and P re What is stored are the predicates to be selected; each worker transmits the newly discovered rules to the scheduler, and the scheduler generates the target rules based on the summary.

[0133] In each superstep, the tasks of workers whose workload is greater than the preset workload are split, and half of the tasks are assigned to other idle workers; assuming that this worker is called W, that is, work node W, if there are multiple tasks in W, W assigns half of the tasks to them; if W only has one task left, that is, the preset task volume is 1, the task is split by splitting the data corresponding to this task.

[0134] As described in step S630, the target data screening rule is generated according to the working node, the scheduling node and the K sampled data.

[0135] In one embodiment of the present invention, the specific process of step S630 "generating the target data screening rule according to the working node, the scheduling node and the K sampled data" can be further explained in combination with the following description.

[0136] Reference Figure 7 , showing a flowchart of the steps of a data screening rule verification method based on multi-round sampling provided by an embodiment of the present application;

[0137] As described in the following steps,

[0138] S710, obtaining K sampled data, and determining all the predicates to be selected in the predicate set to be selected in the K sampled data;

[0139] S720, the working node iteratively adds the candidate predicate to the selected predicate set;

[0140] S730: When the set of predicates to be selected is an empty set, the working node stops iterating, and determining that the set of predicates to be selected is an empty set is a first data screening rule; or,

[0141] S740: When at least one predicate in the predicate set to be selected is related to the selected predicate set, the working node stops iterating, and determines that at least one predicate in the predicate set to be selected is related to the selected predicate set as the second data screening rule;

[0142] S750: Aggregate the first data screening rule and the second data screening rule to the scheduling node, and the scheduling node generates the target data screening rule according to the first data screening rule and the second data screening rule.

[0143] It should be noted that the working node iteratively adds the candidate predicates to the selected predicate set; that is, the working node first iteratively determines the candidate predicates in the candidate predicate set, and then iteratively adds the candidate predicates to the selected predicate set; by extracting K of the sampled data and iteratively selecting P re The candidate predicates in P are added to sel middle.

[0144] It should be noted that the candidate predicate is added to the selected predicate set. When the candidate predicate set is an empty set or the selected predicate meets the preset condition, the iteration is stopped to generate a plurality of discovery rules. The preset condition is P sel →e is a valid REE.

[0145] In a specific implementation, initially, P sel That is, the set of selected predicates is an empty set, P re That is, the set of candidate predicates is the set of all possible predicates. After all tasks are evenly distributed to all workers, each worker performs parallel rule discovery by extracting the required sample data and iteratively selecting P re The predicate in P sel until the following conditions are met: re becomes an empty set, that is, the set of predicates to be selected is an empty set, and determining that the set of predicates to be selected is an empty set is the first data screening rule; or, P sel →e is a valid REE, i.e., at least one of the candidate predicates in the candidate predicate set is related to the selected predicate set, and it is determined that at least one of the candidate predicates in the candidate predicate set is related to the selected predicate set as the second data screening rule; by aggregating the first data screening rule and the second data screening rule to the scheduling node, the scheduling node generates the target data screening rule according to the first data screening rule and the second data screening rule.

[0146] Technical effects of the present invention:

[0147] (1) The proposed sampling method can be performed on cross-table rules such as REE that support multiple tuple variables.

[0148] (2) The sampling method is multi-round sampling, which has a better recall rate of data screening rules than single-round sampling. That is, a data screening rule that is globally applicable to D can be used to select the best data from a small sample D. s was discovered.

[0149] (3) In the process of constructing the relationship graph G, the influence of predicates is considered in edge construction. In this way, useful data that can be used for rule discovery will be sampled with higher priority.

[0150] (4) Ensure the accuracy of sampling; specifically, be able to accurately calculate the required size of the sampling data, as well as the lower bounds of the recall and accuracy of the sampling rules.

[0151] (5) The mining algorithm is a parallel rule discovery algorithm under the overall synchronous parallel computing model.

[0152] This allows more computing resources to be used and improves the computational efficiency of data screening rule mining.

[0153] In multiple public data, not only the efficiency of using full data D for data screening rule mining is compared, but also the accuracy of using uniform sampling for data screening rule mining is compared; according to the results,

[0154] (1) In the sampling data D s The rule discovery can improve the operation efficiency by 12.2 times on average. On a large DBLP data set with 3 relational tables, 18 attributes, and 1.8 million data items, the running time is reduced to 406 seconds by using the method of the present invention, while the running time of data screening rule mining using the full data D is 2096 seconds.

[0155] (2) After sampling only 10% of the data, the average accuracy can reach 90% and the average recall can reach 82% by using the method of the present invention, which are 2% and 7% higher than those of uniform sampling, respectively.

[0156] (3) The rule discovery algorithm is parallel and scalable: if the number of processors used increases from 4 to 20, the efficiency increases by 3.38 times.

[0157] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0158] Reference Figure 8 , shows a structural block diagram of a data screening rule verification device based on multi-round sampling provided by an embodiment of the present application;

[0159] A data screening rule verification device based on multi-round sampling is used for parallel rule mining of target data in a large database, specifically comprising:

[0160] An acquisition module 810 is used to acquire the target data and determine a corresponding data relationship table according to the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple;

[0161] A first construction module 820, configured to construct a relationship graph G according to the tuple, wherein the relationship graph G includes a vertex set V and an edge set E;

[0162] A first generating module 830 is used to perform K sampling on the vertices in the vertex set V to generate K sampling data;

[0163] A second construction module 840 is used to search the K sample data layer by layer to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and a plurality of working nodes;

[0164] The second generating module 850 is used to generate target data screening rules by performing data screening on the K sampled data according to the scheduling node and the plurality of working nodes.

[0165] In one embodiment of the present invention, the first building module 820 includes:

[0166] A first generating submodule, used for generating a plurality of vertices according to the tuple;

[0167] A second generating submodule is used to connect a plurality of the vertices to generate a plurality of edges;

[0168] A first construction submodule is used to construct an edge set E based on the plurality of vertices and the plurality of edges; wherein, when an edge e in the edge set E connects a vertex t and another vertex s in the vertex set v respectively, a tuple pair corresponding to the vertex t and the vertex s satisfies at least one equality predicate;

[0169] The second construction submodule is used to construct the relationship graph G according to the vertex set V and the edge set E.

[0170] In one embodiment of the present invention, the first generating module 830 includes:

[0171] A third generation submodule is used to perform K random walk sampling on the vertices in the vertex set V to generate K sampling data; or,

[0172] The fourth generating submodule is used to perform K breadth-first sampling on the vertices in the vertex set V to generate K sampling data.

[0173] In one embodiment of the present invention, the third generation submodule includes:

[0174] A first acquisition unit, configured to acquire an initial vertex in the vertex set V by uniform sampling;

[0175] A first generating unit, configured to perform iterative sampling on the initial vertex according to the random walk sampling to generate a plurality of sampling tuples, wherein each step in the random walk sampling stops or moves to an adjacent vertex of the current vertex with a preset probability ∈, and the iterative process of the iterative sampling is less than or equal to a preset sample size;

[0176] The second generating unit is used to generate the K sampling data according to a plurality of the sampling tuples.

[0177] In one embodiment of the present invention, the fourth generation submodule includes:

[0178] A second acquisition unit, configured to acquire an initial tuple pair in the vertex set V by uniform sampling, wherein the initial tuple pair includes a vertex z and a vertex y;

[0179] A third generating unit, configured to iteratively sample the initial tuple pair according to the breadth-first sampling to generate a plurality of adjacent vertices, wherein a target distance of the adjacent vertices is less than or equal to a preset distance between the vertex z and the vertex y;

[0180] The fourth generating unit is used to generate the K sampling data according to a number of the adjacent vertices.

[0181] In one embodiment of the present invention, the second generating module 850 includes:

[0182] A fifth generation submodule is used for the scheduling node to generate a plurality of tasks corresponding to the K sampled data, wherein each of the plurality of tasks is a triplet, and the triplet includes a selected predicate set, a to-be-selected predicate set, and a valid predicate;

[0183] The first allocation submodule is used to evenly allocate a plurality of the tasks to the working nodes according to the scheduling node, wherein the working node includes at least one; when there are tasks greater than a preset task amount in the working node, half of the task amount in the working node is allocated to other working nodes; or, when there are only tasks with a preset task amount of 1 in the working node, the data corresponding to the tasks is split;

[0184] The sixth generation submodule is used to generate the target data screening rules according to the working node, the scheduling node and the K sampled data.

[0185] In one embodiment of the present invention, the sixth generation submodule includes:

[0186] A third acquisition unit is used to acquire K sampled data and determine all the candidate predicates in the candidate predicate set in the K sampled data;

[0187] An iteration unit, used for the working node to iteratively add the candidate predicate to the selected predicate set;

[0188] A first determining unit is configured to, when the set of predicates to be selected is an empty set, stop the iteration of the working node, and determine that the set of predicates to be selected is an empty set as a first data screening rule; or,

[0189] a second determining unit, configured to, when at least one of the candidate predicates in the candidate predicate set is related to the selected predicate set, stop the iteration of the working node, and determine that at least one of the candidate predicates in the candidate predicate set is related to the selected predicate set as the second data screening rule;

[0190] The fifth generating unit is used to summarize the first data screening rule and the second data screening rule to the scheduling node, and the scheduling node generates the target data screening rule according to the first data screening rule and the second data screening rule.

[0191] Reference Fig. 9 , shows a computer device of a data screening rule verification method based on multiple rounds of sampling of the present invention, which may specifically include the following:

[0192] The computer device 12 is in the form of a general-purpose computing device, and the components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0193] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor or a local bus 18 using any of a variety of bus 18 architectures. These architectures include, by way of example, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.

[0194] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0195] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be used to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although Fig. 9 Not shown, a disk drive for reading and writing to a removable non-volatile disk (such as a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (such as a CD-ROM, DVD-ROM or other optical medium) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42, which are configured to perform the functions of various embodiments of the present invention.

[0196] A program / utility 40 having a set (at least one) of program modules 42 may be stored in, for example, a memory, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which or some combination may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0197] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, cameras, etc.), one or more devices that enable an operator to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., local area networks (LANs)), wide area networks (WANs), and / or public networks (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Fig. 9Not shown, other hardware and / or software modules may be used in conjunction with the computer device 12, including but not limited to: microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34, etc.

[0198] The processing unit 16 executes various functional applications and data mining by running the programs stored in the system memory 28, such as implementing the data screening rule verification method based on multi-round sampling provided in the embodiment of the present invention.

[0199] That is, when the processing unit 16 executes the program, it implements: obtaining the target data, and determining the corresponding data relationship table based on the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple; constructing a relationship graph G based on the tuple, wherein the relationship graph G includes a vertex set V and an edge set E; sampling the vertices in the vertex set V K times to generate K sampling data; searching the K sampling data layer by layer to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and a plurality of working nodes; and screening the K sampling data based on the scheduling node and the plurality of working nodes to generate target data screening rules.

[0200] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the data screening rule verification method based on multi-round sampling provided in all embodiments of the present application is implemented:

[0201] That is, when the program is executed by the processor, it is implemented as follows: obtaining the target data, and determining the corresponding data relationship table based on the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple; constructing a relationship graph G based on the tuple, wherein the relationship graph G includes a vertex set V and an edge set E; sampling the vertices in the vertex set V K times to generate K sampling data; searching the K sampling data layer by layer to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and a plurality of working nodes; and screening the K sampling data based on the scheduling node and the plurality of working nodes to generate target data screening rules.

[0202] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples of computer-readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0203] Computer-readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0204] The computer program code for performing the operation of the present invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" language or similar programming languages. The program code can be executed entirely on the operator's computer, partially on the operator's computer, as an independent software package, partially on the operator's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the operator's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other.

[0205] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present application.

[0206] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or terminal device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or terminal device including the elements.

[0207] The above is a detailed introduction to a data screening rule verification method and device based on multi-round sampling provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for general technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A data screening rule verification method based on multi-round sampling, used to screen target data in a large database to determine data screening rules, characterized in that: include: Acquire the target data, and determine a corresponding data relationship table according to the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple; A relationship graph G is constructed based on the tuple, wherein the relationship graph G includes a vertex set V and an edge set E; a plurality of vertices are generated based on the tuple; a plurality of edges are generated by connecting the plurality of vertices; an edge set E is constructed based on the plurality of vertices and the plurality of edges; when an edge e in the edge set E connects a vertex t and another vertex s in the vertex set v respectively, a tuple pair constructed corresponding to the vertex t and the vertex s satisfies at least one equality predicate; the relationship graph G is constructed based on the vertex set V and the edge set E; Sampling the vertices in the vertex set V K times to generate K sampling data; Conducting layer-by-layer search on the K sampled data to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and a plurality of working nodes; According to the scheduling node and the plurality of working nodes, the K sampled data are screened to generate target data screening rules; the scheduling node generates a plurality of tasks corresponding to the K sampled data, each of the plurality of tasks is a triple, and the triple includes a selected predicate set, a to-be-selected predicate set and a valid predicate; according to the scheduling node, the plurality of tasks are evenly distributed to the working nodes, and the working nodes include at least one; when there are tasks greater than a preset task amount in the working node, half of the task amount in the working node is distributed to other working nodes; or, when there are only tasks with a preset task amount of 1 in the working node, the data corresponding to the task is distributed to the working node. Splitting; generating the target data screening rule according to the working node, the scheduling node and the K sampled data; obtaining the K sampled data and determining all the candidate predicates in the candidate predicate set within the K sampled data; the working node iteratively adds the candidate predicates to the selected predicate set; when the candidate predicate set is an empty set, the working node stops iterating, and determining that the candidate predicate set is an empty set is a first data screening rule; or, when at least one candidate predicate in the candidate predicate set is related to the selected predicate set, the working node stops iterating, and determining that at least one candidate predicate in the candidate predicate set is related to the selected predicate set is a second data screening rule.

2. The method according to claim 1, characterized in that The step of performing K sampling on the vertices in the vertex set V to generate K sampling data comprises: Performing K random walk sampling on the vertices in the vertex set V to generate K sampling data; or, Perform K breadth-first sampling on the vertices in the vertex set V to generate K sampling data.

3. The method according to claim 2, characterized in that The step of performing K random walk sampling on the vertices in the vertex set V to generate K sampling data comprises: Obtaining an initial vertex in the vertex set V by uniform sampling; The initial vertex is iteratively sampled according to the random walk sampling to generate a plurality of sampling tuples, wherein each step in the random walk sampling is performed with a preset probability Stop or move to a vertex adjacent to the current vertex, where the iterative sampling process is less than or equal to a preset sample size; The K sampling data are generated according to a number of the sampling tuples.

4. The method according to claim 2, characterized in that: The step of performing K breadth-first sampling on the vertices in the vertex set V to generate K sampling data comprises: Obtain an initial tuple pair by uniform sampling in the vertex set V, wherein the initial tuple pair includes vertex z and vertex y; Iteratively sampling the initial tuple pair according to the breadth-first sampling to generate a plurality of adjacent vertices, wherein a target distance of the adjacent vertices is less than or equal to a preset distance between the vertex z and the vertex y; The K sampling data are generated according to a number of the adjacent vertices.

5. A data screening rule verification device based on multi-round sampling, used for data screening of target data in a large database to determine data screening rules, characterized in that: include: An acquisition module, used for acquiring the target data, and determining a corresponding data relationship table according to the target data, wherein each row in the data relationship table generates a tuple and includes at least one tuple; A first construction module is used to construct a relationship graph G according to the tuple, wherein the relationship graph G includes a vertex set V and an edge set E; generate a plurality of vertices according to the tuple; generate a plurality of edges by connecting the plurality of vertices; construct an edge set E according to the plurality of vertices and the plurality of edges; when an edge e in the edge set E respectively connects a vertex t and another vertex s in the vertex set v, a tuple pair corresponding to the vertex t and the vertex s satisfies at least one equality predicate; construct the relationship graph G according to the vertex set V and the edge set E; A first generating module, configured to perform K sampling on the vertices in the vertex set V to generate K sampling data; A second construction module is used to search the K sampled data layer by layer to construct an overall synchronous parallel computing model, wherein the overall synchronous parallel computing model includes a scheduling node and a plurality of working nodes; The second generation module is used to perform data screening on K sampled data according to the scheduling node and several working nodes to generate target data screening rules; the scheduling node generates several tasks corresponding to the K sampled data, each of the several tasks is a triple, and the triple includes a selected predicate set, a to-be-selected predicate set and a valid predicate; according to the scheduling node, several tasks are evenly distributed to the working nodes, and the working nodes include at least one; when there are tasks greater than a preset task amount in the working node, half of the task amount in the working node is distributed to other working nodes; or, when there are only tasks with a preset task amount of 1 in the working node, the tasks are evenly distributed to the working nodes; The corresponding data is split; the target data screening rule is generated according to the working node, the scheduling node and the K sampled data; the K sampled data are obtained, and all the candidate predicates in the candidate predicate set within the K sampled data are determined; the working node iteratively adds the candidate predicates to the selected predicate set; when the candidate predicate set is an empty set, the working node stops iterating, and determines that the candidate predicate set is an empty set as the first data screening rule; or, when at least one candidate predicate in the candidate predicate set is related to the selected predicate set, the working node stops iterating, and determines that at least one candidate predicate in the candidate predicate set is related to the selected predicate set as the second data screening rule.

6. A computer device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 4 when executed by the processor.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images