Entity enhancement rule mining method and apparatus applied to big data
By dividing entity enhancement rules into complete and incomplete rules, and awakening incomplete rules when needed to generate the next set of target rules, the problem of inefficient rule mining in the existing technology is solved, and rapid response to user needs and efficient results are achieved rule output.
Patent Information
- Application Number
- PCT/CN2023/096333
- Authority / Receiving Office
- WO · WO
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-05-18
- Filing Date
- 2023-05-25
- Publication Date
- 2024-11-21
AI Technical Summary
The existing top-k rule mining algorithm is inefficient in big data. It needs to traverse all possible rules for verification and cannot be effectively parallelized. When the user is not satisfied with the current rules, all rules need to be re-enumerated, which affects efficiency.
Divide entity enhancement rules into complete rules and incomplete rules, record the predicates to be selected for incomplete rules, generate the first K complete rules as the target rule set, and wake up the incomplete rules and the first K rules when receiving the update request. Generate the next set of target rulesets to avoid duplicate enumerations.
It improves the efficiency of rule mining, can output top-k rules without traversing all rules, quickly respond to user needs, and reduces the consumption of computing resources.
Smart Images

Figure CN2023096333_21112024_PF_FP_ABST
Abstract
Description
A method and device for mining entity enhancement rules applied to big data Technical Field
[0001] The present application relates to the field of computers, and in particular to a method and device for mining entity enhancement rules applied to big data. Background Art
[0002] Rule discovery in big data means, given a data set D, identifying all the rules that hold true on the data set. A major problem faced by rule discovery is that a large number of rule candidates will be generated. Therefore, an effective strategy is to perform top-k rule discovery. Specifically, given a data scoring function score(), Indicates the rules, for each rule Calculate a ranking score using Then, on the dataset D, the top k rules with the highest ranking scores can be returned as the results of top-k rule discovery.
[0003] Existing top-k rule mining algorithms use a depth-first or breadth-first standalone approach to discover entity enhancement rules in data. Essentially, this is a process of enumerating all permutations and combinations of predicates on the entire data D. For all possible predicates, any one or more of them can potentially form a valid rule with the REE result e. Therefore, in order to perform rule mining, existing methods need to test all permutations and combinations of predicates. For each valid rule, Calculate his ranking score After all the ranking scores of the established rules are recorded, the top k rules with the highest scores can be obtained and returned as the result of the top-k rule mining algorithm.
[0004] When users are dissatisfied with the currently returned rules, they typically continue searching for the next set of top-k rules. However, to meet this user requirement, existing algorithms must initially enumerate and verify all possible rules that match the data. However, in practice, users are often only interested in the top-ranked rules and do not need to iterate through all rules. This also reduces rule discovery efficiency.
[0005] Summary of the Invention
[0006] In view of the above problems, the present application is proposed to provide a method and apparatus for entity enhancement rule mining applied to big data that overcomes the above problems or at least partially solves the above problems. The method includes:
[0007] Acquire a data set, and determine an entity enhancement rule set in the data set; the rule types of the entity enhancement rule set include complete rules and incomplete rules;
[0008] Recording the candidate predicates corresponding to the incomplete rules in the entity enhancement set;
[0009] Determine the ranking scores corresponding to the entity enhancement rules in the entity enhancement rule set, generate a first target rule set from the top K complete rules in the ranking scores, and delete the entity enhancement rules included in the first target rule set in the entity enhancement rule set;
[0010] When an output request for updating the target rule set is received, the rule types of the top K entity enhancement rules in the ranking scores are determined;
[0011] When the rule types of the first K entity enhancement rules include the incomplete rule, the corresponding incomplete rule is awakened according to the candidate predicate, and a second target rule set is generated according to the awakened incomplete rule and the remaining first K entity enhancement rules.
[0012] Furthermore, determining the entity enhancement rule set in the data set includes:
[0013] determining a set of candidate condition predicates and a result predicate in the data set;
[0014] Iterate at least one of the candidate condition predicate sets to generate a target condition predicate set in a preset manner;
[0015] Determining entity enhancement rules according to the target condition predicate set and the result predicate;
[0016] An entity enhancement rule set is generated according to at least one entity enhancement rule.
[0017] Furthermore, the method further comprises:
[0018] For any entity enhancement rule to be identified, determine whether there is a subset of the target condition predicate set in the entity enhancement rule to be identified, and the subset matches the result predicate of the entity enhancement rule to be identified;
[0019] If the subset does not exist, the rule type of the entity enhancement rule to be identified is a complete rule.
[0020] Furthermore, the method further comprises:
[0021] For any entity enhancement rule to be identified, if a preset stop expansion condition is met in the process of iteratively generating a target condition predicate set from at least one of the candidate condition predicate sets in a preset manner, the iterative generation of the target predicate set is stopped, and the rule type of the entity enhancement rule to be identified is determined to be an incomplete rule;
[0022] The preset stop expansion condition includes that the ranking score of the to-be-identified entity enhancement rule is after the Kth one.
[0023] Furthermore, the method further comprises:
[0024] Constructing a preset heap; the heap is used to store the candidate predicates corresponding to the incomplete rules in the entity enhancement set, and to store the entity enhancement rule set after deleting the target rule set.
[0025] Furthermore, the method further comprises:
[0026] When the rule types of the current K entity enhancement rules do not include the incomplete rule, a second target rule set is generated according to the current first K entity enhancement rules.
[0027] Furthermore, determining the ranking score of the entity enhancement rule in the entity enhancement rule set includes:
[0028] Based on the result predicate, dividing the data set into a plurality of to-be-processed sets corresponding to entity enhancement rules in the entity enhancement set;
[0029] Distributing the set to be processed to multiple processing units; the processing units are configured to output preliminary sorting information of corresponding entity enhancement rules according to the set to be processed received;
[0030] The preliminary ranking information of the multiple processing units is processed to obtain ranking scores corresponding to the entity enhancement rules.
[0031] An entity enhancement rule mining device applied to big data, comprising:
[0032] An acquisition module is used to acquire a data set and determine an entity enhancement rule set in the data set; the rule types of the entity enhancement rule set include complete rules and incomplete rules;
[0033] A candidate predicate recording module, used for recording the candidate predicates corresponding to the incomplete rules in the entity enhancement set;
[0034] a first target rule set processing module, configured to determine ranking scores corresponding to entity enhancement rules in the entity enhancement rule set, generate a first target rule set from the top K complete rules in the ranking scores, and delete the entity enhancement rules included in the first target rule set from the entity enhancement rule set;
[0035] A rule type determination module is configured to determine the rule types of the top K entity enhancement rules in the ranking scores when receiving an output request for updating a target rule set;
[0036] The second target rule set generation module is used to wake up the corresponding incomplete rule according to the to-be-selected predicate when the rule type of the current K entity enhancement rules includes the incomplete rule, and generate the second target rule set according to the woken incomplete rule and the remaining first K entity enhancement rules.
[0037] A computer device comprises a processor, a memory and a computer program stored in the memory and capable of running on the processor, wherein when the computer program is executed by the processor, the steps of the entity enhancement rule mining method applied to big data are implemented as described above.
[0038] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the entity enhancement rule mining method applied to big data as described above.
[0039] This application has the following advantages:
[0040] In the embodiments of the present application, compared to the prior art in which the user usually continues to look for the next set of top-k rules when he is not satisfied with the currently returned rules, the existing algorithm needs to enumerate and verify all possible rules that meet the data at the very beginning. However, in practice, users are often only interested in the top-ranked rules and do not need to traverse all the rules, and traversing all the rules will affect the efficiency of rule discovery. The present application provides a method that can, when mining entity enhancement rules in data, divide the rule types of entity enhancement rules into complete rules and incomplete rules, and record the corresponding candidate predicates of the incomplete rules, and output the top K complete rules of the ranking score of the data set as the first target rule set, that is, output the top-k rules for mining the data set without traversing all the rules. And when receiving the user's output request for updating the target rule set, it is determined that the user needs to output the next k entity enhancement rules and the types of the current first K entity enhancement rules include incomplete rules. The corresponding incomplete rules can be awakened according to the predicate to be selected, and the solution of the second target rule set can be generated according to the awakened incomplete rules and the remaining first K entity enhancement rules, avoiding the need to enumerate and verify all possible rules that meet the data. Specifically, the method includes: obtaining a data set, and determining the entity enhancement rule set in the data set; the rule types of the entity enhancement rule set include complete rules and incomplete rules; recording the corresponding incomplete rules in the entity enhancement set. ; determine the ranking score corresponding to the entity enhancement rule in the entity enhancement rule set, and generate a first target rule set with the top K complete rules in the ranking score, and delete the entity enhancement rules included in the first target rule set in the entity enhancement rule set; when an output request for updating the target rule set is received, determine the rule types of the top K entity enhancement rules in the ranking score; when the rule types of the current K entity enhancement rules include the incomplete rule, wake up the corresponding incomplete rule according to the candidate predicate, and generate a second target rule set, i.e., the next group of top-k rules, based on the woken-up incomplete rule and the remaining top K entity enhancement rules. By waking up the corresponding incomplete rules according to the candidate predicate when receiving an output request for updating the target rule set, and generating a second target rule set based on the woken-up incomplete rules and the remaining top K entity enhancement rules, the disadvantage of needing to enumerate and verify all possible rules that meet the data at the beginning of rule mining when the user is not satisfied with the currently returned rules and needs to obtain the next set of top-k rules is solved. This achieves the goal of outputting the top-k rules without traversing all the rules in the data set, and makes it possible to return the next k rules with the highest ranking scores at any time according to user needs without having to obtain all the rules in advance, thereby improving the mining efficiency of rules in the data. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] In order to more clearly illustrate the technical solution of the present application, the following is a brief introduction to the drawings required for the description of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0042] FIG1 is a flowchart of the steps of entity enhancement rule mining applied to big data provided by an embodiment of the present application;
[0043] FIG2 is a flow chart of a method for mining entity enhancement rules applied to big data, provided by an embodiment of the present application;
[0044] FIG3 is a structural block diagram of an entity enhancement rule mining device applied to big data provided by an embodiment of the present application;
[0045] FIG4 is a schematic structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION
[0046] To make the objectives, features, and advantages of this application more readily apparent, the present application is further described below in conjunction with the accompanying drawings and specific embodiments. It is apparent that the embodiments described are only a portion of the embodiments of this application, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments in this application without inventive effort are also within the scope of protection of this application.
[0047] The inventors found through analysis of the prior art that:
[0048] Rules for Entity Enhancing (REE) are the basic components of REE, which are predicates p, defined as follows: p: = R(t) | tA◎c | tA◎sB | M(tA,sB)
[0049] Here, ◎ is an operator, which can be equal to or not equal to; R(t) indicates that t is a tuple variable in the relational table R; tA represents the attribute A of the variable t; and M is a machine learning model that returns true if tA and sB are related, and false otherwise. tA◎c with a constant is called a constant predicate; tA◎sB without a constant is called a variable predicate; M(tA,sB) is called a machine learning predicate.
[0050] Based on predicates, REEφ is defined as: X→e, where (1) X is a combination of multiple predicates, which is called the condition of this REE; (2) e is a predicate, which is called the result of this REE.
[0051] A specific REE example is as follows:
[0052] Express(t)∧Express(s)∧t.Recipient=s.Recipient∧t.Address="Province A, City B"→s.Postal Code="XYZZZZ"
[0053] The scenario described by this REE is that if the recipients of two express packages t and s are the same person, and t's address is in "Province A, City B", then s's zip code must be "XYZZZZ".
[0054] Existing top-k rule mining algorithms discover REE rules in data using a depth-first or breadth-first standalone approach. Their essence is a process of enumerating all permutations and combinations of predicates on the entire data D. For all possible predicates, any one or more of them can potentially form a valid rule with the REE result e. Therefore, in order to perform rule mining, existing methods need to try all permutations and combinations of predicates. For each valid rule, Calculate his ranking score The top k rules are then retrieved and recorded. Once the ranking scores of all valid rules are recorded, the top k rules are retrieved and returned as the result of the top-k rule mining algorithm. Since the complete rule ranking is known, if the user is dissatisfied with the current top k rules, the next k highest-scoring rules can be retrieved from the sequence and returned as the result. When the data scale is large, such a brute-force standalone algorithm is undoubtedly inefficient.
[0055] To obtain the top-k rules, existing methods not only require enumerating and verifying all possible rules that satisfy data D, but also cannot effectively parallelize the computational process. If a user is dissatisfied with the current top-k rules, they cannot effectively return the next k highest-scoring rules, as a search engine would. Consequently, the efficiency of existing top-k rule discovery (mining efficiency) is very low.
[0056] It should be noted that, in any embodiment of the present invention, different entity enhancement rules consisting of one or more predicates may be applicable, and the embodiment of the present invention does not limit the applicable entity enhancement rules.
[0057] 1 , a method for mining entity enhancement rules applied to big data is shown in accordance with an embodiment of the present application. The method includes:
[0058] S110, obtaining a data set, and determining an entity enhancement rule set in the data set; the rule types of the entity enhancement rule set include complete rules and incomplete rules;
[0059] S120, recording the candidate predicates corresponding to the incomplete rules in the entity enhancement set;
[0060] S130: Determine the ranking scores corresponding to the entity enhancement rules in the entity enhancement rule set, generate a first target rule set from the top K complete rules in the ranking scores, and delete the entity enhancement rules included in the first target rule set in the entity enhancement rule set;
[0061] S140: When an output request for updating a target rule set is received, the rule types of the top K entity enhancement rules in the ranking scores are determined;
[0062] S150: When the rule types of the current K entity enhancement rules include the incomplete rule, the corresponding incomplete rule is awakened according to the candidate predicate, and a second target rule set is generated according to the awakened incomplete rule and the remaining first K entity enhancement rules.
[0063] In an embodiment of the present application, a data set is obtained and an entity enhancement rule set in the data set is determined; the rule types of the entity enhancement rule set include complete rules and incomplete rules; the candidate predicates corresponding to the incomplete rules in the entity enhancement set are recorded; the ranking scores corresponding to the entity enhancement rules in the entity enhancement rule set are determined, and the first K complete rules in the ranking scores are used to generate a first target rule set, and the entity enhancement rules included in the first target rule set in the entity enhancement rule set are deleted; when an output request for updating the target rule set is received, the rule types of the first K entity enhancement rules in the ranking scores are determined; when the rule types of the current K entity enhancement rules include the incomplete rules, the corresponding incomplete rules are awakened according to the candidate predicate, and the second target rule set, i.e., the next group of top-k rules, is generated according to the awakened incomplete rules and the remaining first K entity enhancement rules. By waking up the corresponding incomplete rules according to the candidate predicate when receiving an output request for updating the target rule set, and generating a second target rule set based on the woken-up incomplete rules and the remaining top K entity enhancement rules, the disadvantage of needing to enumerate and verify all possible rules that meet the data at the beginning of rule mining when the user is not satisfied with the currently returned rules and needs to obtain the next set of top-k rules is solved. This achieves the goal of outputting the top-k rules without traversing all the rules in the data set, and makes it possible to return the next k rules with the highest ranking scores at any time according to user needs without having to obtain all the rules in advance, thereby improving the mining efficiency of rules in the data.
[0064] Next, the entity enhancement rule mining method applied to big data in this exemplary embodiment will be further described.
[0065] As described in step S110 , a data set is acquired, and an entity enhancement rule set in the data set is determined; the rule types of the entity enhancement rule set include complete rules and incomplete rules.
[0066] The embodiment of the present application does not limit the method of obtaining the data set, and the data set can be read from a local storage medium or received online. In addition, the data set can also be in a dynamically updated state, that is, the data set can be data that changes regularly or irregularly.
[0067] In one embodiment of the present invention, the specific process of "determining the entity enhancement rule set in the data set" described in step S110 can be further explained in combination with the following description, which can specifically include: determining a candidate condition predicate set and a result predicate in the data set; iterating at least one of the candidate condition predicate sets to generate a target condition predicate set in a preset manner; determining an entity enhancement rule based on the target condition predicate set and the result predicate; and generating an entity enhancement rule set based on at least one of the entity enhancement rules.
[0068] The candidate condition predicate set P can be constructed re and target condition predicate set P sel Extract the result predicate e from the data set and store all possible predicates identified into the candidate condition predicate set P re , construct the target condition predicate set P sel When it is an empty set, it is used to store the predicates selected as the REE conditions for renting a car. By traversing the search space in a breadth-first manner, iteratively select P re The predicate in P is added to sel This process is also called expanding P sel The process continues until one of the following stopping conditions is met: (1) P re becomes an empty set; or (2)P sel →e is a valid REE rule, which is expressed as The entity enhancement rules mined from the data set are used to generate an entity enhancement rule set and stored.
[0069] In the embodiment of the present application, the rule types of the entity enhancement rule set are divided into two categories, one is complete rules and the other is incomplete rules.
[0070] In an embodiment of the present application, the following steps are used to identify whether any entity enhancement rule is a complete rule: for any entity enhancement rule to be identified, determine whether there is a subset of the target condition predicate set in the entity enhancement rule to be identified, and the subset matches the result predicate of the entity enhancement rule to be identified; if the subset does not exist, the entity enhancement rule to be identified is a complete rule.
[0071] The complete rule is also called the minimal rule Specifically, if there is a known REE rule Let X' be a subset of X, then It is not a minimization rule, that is, the current is not a complete rule. Similarly, if the rule There is no subset X' of X such that but For complete rules.
[0072] In an embodiment of the present application, the following steps are used to identify whether any entity enhancement rule is an incomplete rule: for any entity enhancement rule to be identified, if a preset stop expansion condition is met in the process of iteratively generating a target condition predicate set from at least one of the to-be-identified condition predicate sets in a preset manner, the iterative generation of the target predicate set is stopped, and the entity enhancement rule to be identified is determined to be an incomplete rule; the preset stop expansion condition includes that the ranking score of the entity enhancement rule to be identified is after the Kth one.
[0073] Incomplete is also called unexpanded (target predicate set expansion) completed rule. If in the expansion process, continue to expand X, the rule The score of will not be higher than the top k rules in the current heap Σ. Even if the stopping condition of the expansion has not been met, stop the expansion of X and As an incomplete rule.
[0074] In the embodiment of the present application, the above-mentioned identification of whether any entity enhancement rule is a complete rule and the identification of whether any entity enhancement rule is an incomplete rule can be called at any time and can be called multiple times, and the embodiment of the present application does not limit this.
[0075] The following description may be used to further illustrate the step S120 of "recording the candidate predicates corresponding to the incomplete rules in the entity enhancement set."
[0076] Although in the above case, the rule is not completely It is impossible to expand to become the top-k rules with the highest ranking scores. However, in the subsequent instant discovery (step S150), if some high-scoring rules have been returned, then It is still possible to become a top-k rule with a higher score through expansion. Therefore, it is necessary to store the candidate predicate set corresponding to the incomplete rule φ so that it can be continued at any time. This is an extension of The strategy of expanding at any time is also called the lazy wake-up strategy.
[0077] In an embodiment of the present application, the entity enhancement rule mining method applied to big data also includes: constructing a preset heap; the heap is used to store the candidate predicates corresponding to the incomplete rules in the entity enhancement set, and to store the entity enhancement rule set after deleting the target rule set.
[0078] The embodiment of the present application can construct a heap Σ and store complete rules and / or incomplete rules, as well as the candidate predicates corresponding to the incomplete rules in the heap Σ. When any rule is output as a target rule set, the output target rule set is deleted from the heap Σ, thereby avoiding the problem of repeatedly outputting the same rules when the user needs to output the next set of top-k rules, resulting in the output results not matching the user's needs and causing waste of resources.
[0079] In one embodiment of the present invention, the specific process of step S150, "when the rule types of the first K entity enhancement rules include the incomplete rule, waking up the corresponding incomplete rule according to the to-be-selected predicate, and generating a second target rule set based on the awakened incomplete rule and the remaining first K entity enhancement rules" can be further explained in combination with the following description.
[0080] Since the candidate predicates corresponding to the incomplete rules are stored in the aforementioned steps, the continued expansion of the incomplete rules can be completed very quickly, that is, the target predicate set is continuously iterated and updated, thereby awakening the incomplete rules. The awakened incomplete rules will continue to exist in the heap Σ as complete rules until the top k rules with the highest ranking scores stored in the heap Σ are all complete rules, and the top k complete rules with the current ranking scores are returned to the user as the second target rule set. Through step S150, when the user is not satisfied with the currently returned rules, the next set of top-k rules with the highest ranking scores can be found for them immediately, which is also called instant discovery.
[0081] In an embodiment of the present invention, the method further includes: when the rule types of the current K entity enhancement rules do not include the incomplete rule, generating a second target rule set based on the current first K entity enhancement rules.
[0082] If only complete rules exist in the current heap Σ, there's no need to activate the incomplete rules, or continue iterating over the target predicate sets corresponding to the incomplete rules. Instead, the top k complete rules by current ranking score are simply returned to the user as the second target rule set. Combining this step with step S150 can further accelerate the determination of the next set of top-k rules with the highest ranking scores if the user is dissatisfied with the currently returned rules.
[0083] At the same time, in actual applications, in order to ensure that the rules discovered each time are non-redundant, it is necessary to perform additional checks on the output complete rules. A rule will only be output if it cannot be deduced from other rules that have already been output.
[0084] In the embodiments of the present application, the specific calculation formula for the ranking score is not limited.
[0085] In addition, the inventors found through analysis of existing technologies that the existing top-k rule mining algorithm finds REE rules in data through a depth-first or breadth-first standalone method. Its essence is a process of enumerating all permutations and combinations of predicates on the entire data D. For all possible predicates, any one or more of them may form a valid rule with the REE result e. Therefore, in order to perform rule mining, the existing method needs to try all permutations and combinations of predicates. For each valid rule Calculate his ranking score and record them. Once the ranking scores of all valid rules are recorded, we can obtain the top k rules by score and return them as the result of the top-k rule mining algorithm. Since the complete rule ranking is known, if the user is not satisfied with the current top-k rules, we can obtain the next k rules with the highest scores from the sequence and return them as the result. When the data scale is large, the efficiency of such a brute force stand-alone algorithm is undoubtedly very low. Even using more computing resources does not necessarily guarantee an improvement in the computational efficiency of rule mining.
[0086] In an optional embodiment of the present application, the "calculating the ranking scores of each entity enhancement rule" in step S130 may include: dividing the data set into multiple sets to be processed based on the result predicate; distributing the sets to be processed to multiple processing units; the processing units are used to output preliminary ranking information of the corresponding entity enhancement rules according to the sets to be processed received; and processing the preliminary ranking information of the multiple processing units to obtain ranking scores.
[0087] Step S130 can be further explained in conjunction with the following description. Referring to FIG2 , a flow chart of an entity enhancement rule mining method for big data provided by an embodiment of the present application is shown. To improve the processing efficiency of the embodiment of the present application, an overall synchronous parallel computing model can be introduced in practice to achieve parallel discovery of entity rules. The details are as follows:
[0088] The overall synchronous parallel computing model is based on breadth-first search and consists of a scheduling unit (coordinator) and n processing units (workers). In this model, the scheduling unit is responsible for generating and distributing tasks, maintaining the overall top-k rules, and load balancing, while the processing units are responsible for parallel and real-time rule discovery. The overall computation is divided into multiple supersteps, each bounded by a fixed time.
[0089] In general, the scheduling unit maintains a global heap Σ. In the i-th superstep, the scheduling unit records the ranking score of the rule with the k-th highest score in the heap Σ, denoted by Ti.
[0090] The scheduling unit first splits the rule discovery into multiple small tasks based on the REE result e that needs to be discovered. Each task consists of a triple:<Psel,Pre,e> . Psel stores the predicates that have been selected to form the REE conditions, while Pre stores the predicates to be selected. Initially, Psel is an empty set, and Pre is the set of all possible predicates. After the scheduling unit distributes the value of Ti and all tasks evenly to all processing units, each processing unit performs top-k rule discovery of the i-th superstep at the same time: they will maintain a heap locally to store the rules discovered locally. When the processing units perform local rule expansion, they will extract the required data based on the distributed tasks and iteratively select the predicates in Pre to add to Psel until at least one of the following three conditions is met: (1) P re becomes an empty set; (2)P sel →e is a complete rule; (3)P sel →e is an incomplete rule.
[0091] At the end of each superstep, the processing units will transmit the top-k rules discovered to the scheduling unit for aggregation. The scheduling unit adjusts the overall heap Σ based on the collected rules and calculates the ranking score (ranking score) Ti+1 to be used in the next superstep. In each superstep, the scheduling unit will also perform load balancing, splitting the tasks of the processing unit W with the heaviest workload and assigning half of them to idle processing units. If there are multiple tasks in W, W will assign half of the tasks. If W only has one task left, it will be split into multiple subtasks by splitting the corresponding data of this task for distribution. When all processing units have completed all calculations, the parallel rule discovery ends. When the parallel rule discovery ends, the ranking score of each entity enhancement rule can be obtained, and then the top-k rules can be output based on the ranking score.
[0092] In an embodiment of the present application, by instantly discovering relevant processing steps, the next k rules with the highest ranking scores can be returned at any time based on user needs without having to obtain all rules in advance. By discovering relevant processing steps in parallel, based on an overall synchronous parallel computing model, tasks are generated and distributed, and tasks are adjusted in a load-balanced manner. The scheduling unit and the processing unit collaborate to perform parallel top-k rule discovery. This ensures that the runtime of top-k rule discovery is reduced when more computing resources are used, thus achieving parallel scalability.
[0093] The following is an example to illustrate the embodiment of the present application: Assume that k=3, and the rule for sorting the scores from high to low in the current heap Σ is: Among them, if the superscript of the rule is c, it means that the rule is a complete rule; if the superscript of the rule is p, it means that the rule is an incomplete rule. Since there are incomplete rules in the top three rules, we will continue to expand / wake up the incomplete rules in descending order. We will first continue to expand Assume that after expansion Finally, we have three new rules The heap Σ is also updated to At this time, the top three rules in Σ are all complete rules. Then these three rules are returned as the results of top-k rule discovery.
[0094] In practical applications, the entity-enhanced rule mining method for big data provided by this application and existing methods were used to mine rules for the same dataset. Experiments were conducted on multiple datasets, and the efficiency of the instant top-k rule discovery algorithm, the parallel rule discovery algorithm, and the existing rule discovery algorithm were comprehensively compared. The results showed:
[0095] (1) When users continuously want to find the k highest-scoring rules, the instant rule discovery algorithm has a clear advantage. The instant discovery algorithm effectively maintains incomplete rules and can quickly wake up and restore the expansion of incomplete rules, making the instant discovery algorithm 95 times faster than traditional algorithms. The next k rules may have already been enumerated and stored. Therefore, when the instant discovery algorithm has accumulated enough incomplete rules, the time to find the next k rules can be further reduced.
[0096] (2) When more computing resources are used, the improvement in parallel rule discovery is very obvious. For example, when performing rule discovery on a dataset, the processing efficiency of rule discovery increased by 3.15 times when the number of computing devices increased from 4 to 20. This confirms that the parallel rule discovery algorithm is parallel scalable.
[0097] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0098] 3 , an entity enhancement rule mining device for big data provided by an embodiment of the present application is shown; specifically, the device includes:
[0099] An acquisition module 310 is configured to acquire a data set and determine an entity enhancement rule set in the data set; the entity enhancement rule set may include complete rules and incomplete rules.
[0100] A candidate predicate recording module 320 is used to record the candidate predicates corresponding to the incomplete rules in the entity enhancement set;
[0101] A first target rule set processing module 330 is configured to determine ranking scores corresponding to entity enhancement rules in the entity enhancement rule set, generate a first target rule set from the top K complete rules in the ranking scores, and delete the entity enhancement rules included in the first target rule set from the entity enhancement rule set;
[0102] A rule type determination module 340 is configured to determine the rule types of the top K entity enhancement rules in the ranking scores when receiving an output request for updating a target rule set;
[0103] The second target rule set generation module 350 is used to wake up the corresponding incomplete rule according to the to-be-selected predicate when the rule type of the current K entity enhancement rules includes the incomplete rule, and generate a second target rule set based on the woken incomplete rule and the remaining first K entity enhancement rules.
[0104] In one embodiment of the present invention, the acquisition module 310 includes:
[0105] A predicate determination submodule, configured to determine a set of candidate condition predicates and a result predicate in the data set;
[0106] A target condition predicate iteration submodule is used to iterate at least one of the candidate condition predicate sets to generate a target condition predicate set in a preset manner;
[0107] The rule determination submodule is configured to determine an entity enhancement rule based on the target condition predicate set and the result predicate.
[0108] In one embodiment of the present invention, the apparatus further comprises:
[0109] The complete rule identification module is used to determine, for any entity enhancement rule to be identified, whether there is a subset of the target condition predicate set in the entity enhancement rule to be identified, and the subset matches the result predicate of the entity enhancement rule to be identified; if the subset does not exist, the entity enhancement rule to be identified is a complete rule.
[0110] In one embodiment of the present invention, the apparatus further comprises:
[0111] an incomplete rule identification module configured to, for any entity enhancement rule to be identified, stop iterating the target predicate set and determine that the entity enhancement rule to be identified is an incomplete rule if a preset stop expansion condition is satisfied during the process of iteratively generating the target predicate set from at least one of the candidate condition predicate sets in a preset manner;
[0112] The preset stop expansion condition includes that the ranking score of the to-be-identified entity enhancement rule is after the Kth one.
[0113] In one embodiment of the present invention, the apparatus further comprises:
[0114] The heap construction module is used to construct a preset heap; the heap is used to store the candidate predicates corresponding to the incomplete rules in the entity enhancement set, and to store the entity enhancement rule set after deleting the target rule set.
[0115] In one embodiment of the present invention, the apparatus further comprises:
[0116] The direct generation module is configured to generate a second target rule set based on the first K entity enhancement rules when the rule types of the current K entity enhancement rules do not include the incomplete rule.
[0117] In one embodiment of the present invention, the first target rule set processing module 330 includes:
[0118] a partitioning submodule, configured to partition the data set into a plurality of to-be-processed sets corresponding to entity enhancement rules in the entity enhancement set based on the result predicate;
[0119] A distribution submodule, configured to distribute the set to be processed to a plurality of processing units; the processing units are configured to output preliminary sorting information of corresponding entity enhancement rules according to the set to be processed received;
[0120] The integration submodule is used to process the preliminary ranking information of the multiple processing units to obtain ranking scores corresponding to the entity enhancement rules.
[0121] 4 , a computer device for an entity enhancement rule mining method for big data according to the present invention is shown, which may specifically include the following:
[0122] The computer device 12 is a general-purpose computing device. The components of the computer device 12 may include but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).
[0123] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or memory controller, a peripheral bus 18, an accelerated graphics port, a processor, or a local bus 18 that utilizes any of a variety of bus 18 architectures. Examples of such architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus 18, a Micro Channel Architecture (MAC) bus 18, an Enhanced ISA bus 18, an Audio Video Electronics Standards Association (VESA) local bus 18, and a Peripheral Component Interconnect (PCI) bus 18.
[0124] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.
[0125] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 may be configured to read and write to non-removable, non-volatile magnetic media (commonly referred to as a "hard drive"). Although not shown in FIG. 4 , a magnetic disk drive may be provided for reading and writing to a removable non-volatile magnetic disk (e.g., a "floppy disk"), as well as an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, DVD-ROM, or other optical media). In these cases, each drive may be connected to bus 18 via one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 configured to perform the functions of various embodiments of the present invention.
[0126] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in a memory. Such program modules 42 include, but are not limited to, an operating system, one or more application programs, other program modules 42, and program data, each of which, or some combination thereof, may include an implementation of a network environment. The program modules 42 generally perform the functions and / or methods of the embodiments described herein.
[0127] The computer device 12 can also communicate with one or more external devices 14 (e.g., a keyboard, pointing device, display 24, camera, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., a network card, modem, etc.). Such communication can occur via an input / output (I / O) interface 22. Furthermore, the computer device 12 can communicate with one or more networks (e.g., a local area network (LAN)), a wide area network (WAN), and / or a public network (e.g., the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via a bus 18. It should be understood that, although not shown in FIG. 4 , other hardware and / or software modules can be used in conjunction with the computer device 12, including, but not limited to, microcode, device drivers, redundant processing units 16, external disk drive arrays, RAID systems, tape drives, and data backup storage systems 34.
[0128] The processing unit 16 executes various functional applications and data processing by running programs stored in the system memory 28, such as implementing the entity enhancement rule mining method applied to big data provided by an embodiment of the present invention.
[0129] That is, when the processing unit 16 executes the program, it implements: obtaining a data set, and determining an entity enhancement rule set in the data set; the rule types of the entity enhancement rule set include complete rules and incomplete rules; recording the candidate predicates corresponding to the incomplete rules in the entity enhancement set; determining the ranking scores corresponding to the entity enhancement rules in the entity enhancement rule set, and generating a first target rule set with the top K complete rules in the ranking scores, and deleting the entity enhancement rules included in the first target rule set in the entity enhancement rule set; when receiving an output request for updating the target rule set, determining the rule types of the top K entity enhancement rules in the ranking scores; when the rule types of the current K entity enhancement rules include the incomplete rules, waking up the corresponding incomplete rules according to the candidate predicate, and generating a second target rule set based on the woken-up incomplete rules and the remaining top K entity enhancement rules.
[0130] In an embodiment of the present invention, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the entity enhancement rule mining method applied to big data provided in all embodiments of the present application:
[0131] That is, when the program is executed by the processor, it is implemented as follows: obtaining a data set, and determining an entity enhancement rule set in the data set; the rule types of the entity enhancement rule set include complete rules and incomplete rules; recording the candidate predicates corresponding to the incomplete rules in the entity enhancement set; determining the ranking scores corresponding to the entity enhancement rules in the entity enhancement rule set, and generating a first target rule set with the top K complete rules in the ranking scores, and deleting the entity enhancement rules included in the first target rule set in the entity enhancement rule set; when an output request for updating the target rule set is received, determining the rule types of the top K entity enhancement rules in the ranking scores; when the rule types of the current K entity enhancement rules include the incomplete rules, waking up the corresponding incomplete rules according to the candidate predicate, and generating a second target rule set based on the woken-up incomplete rules and the remaining top K entity enhancement rules.
[0132] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.
[0133] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.
[0134] The computer program code for performing the operations of the present invention can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, C++, and conventional procedural programming languages such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (for example, using an Internet service provider to connect through the Internet). The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referenced to each other.
[0135] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the embodiments of the present invention.
[0136] Finally, it should be noted that, in this document, relational terms such as first and second, etc., are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or terminal device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or terminal device that includes the element.
[0137] The above is a detailed introduction to the entity enhancement rule mining method and device applied to big data provided by this application. This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method of this application and its core idea; at the same time, for general technical personnel in this field, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on this application.
Claims
1. A method for mining entity enhancement rules applied to big data, characterized in that: Acquire a data set, and determine an entity enhancement rule set in the data set; the rule types of the entity enhancement rule set include complete rules and incomplete rules; Recording the candidate predicates corresponding to the incomplete rules in the entity enhancement set; Determine the ranking scores corresponding to the entity enhancement rules in the entity enhancement rule set, generate a first target rule set from the first K complete rules in the ranking scores, and delete the entity enhancement rules included in the first target rule set in the entity enhancement rule set; When an output request for updating the target rule set is received, the rule types of the top K entity enhancement rules in the ranking scores are determined; When the rule types of the current K entity enhancement rules include the incomplete rule, the corresponding incomplete rule is awakened according to the candidate predicate, and a second target rule set is generated according to the awakened incomplete rule and the remaining first K entity enhancement rules.
2. The method according to claim 1, characterized in that: Determining the entity enhancement rule set in the data set includes: Determining a set of candidate condition predicates and a result predicate in the data set; Iterate at least one of the candidate condition predicate sets to generate a target condition predicate set in a preset manner; Determine entity enhancement rules according to the target condition predicate set and the result predicate; An entity enhancement rule set is generated according to at least one of the entity enhancement rules.
3. The method according to claim 2, characterized in that The method further comprises: For any entity enhancement rule to be identified, determine whether there is a subset of the target condition predicate set in the entity enhancement rule to be identified, and the subset matches the result predicate of the entity enhancement rule to be identified; If the subset does not exist, the rule type of the to-be-identified entity enhancement rule is a complete rule.
4. The method according to claim 3, characterized in that The method further comprises: For any entity enhancement rule to be identified, if a preset stop expansion condition is met in the process of iteratively generating a target condition predicate set from at least one of the candidate condition predicate sets in a preset manner, the iterative generation of the target predicate set is stopped, and the rule of the entity enhancement rule to be identified is determined. The type is not completely regular; The preset stop expansion condition includes that the ranking score of the to-be-identified entity enhancement rule is after the Kth one.
5. The method according to claim 4, characterized in that The method further comprises: Constructing a preset heap; the heap is used to store the candidate predicates corresponding to the incomplete rules in the entity enhancement set, and to store the entity enhancement rule set after deleting the target rule set.
6. The method according to claim 5, characterized in that The method further comprises: When the rule types of the current K entity enhancement rules do not include the incomplete rule, a second target rule set is generated according to the current first K entity enhancement rules.
7. The method according to claim 2, characterized in that Determining the ranking scores of entity enhancement rules in the entity enhancement rule set includes: Based on the result predicate, the data set is divided into a plurality of to-be-processed sets corresponding to entity enhancement rules in an entity enhancement set; Distributing the to-be-processed set to a plurality of processing units; the processing units are used to output preliminary sorting information of corresponding entity enhancement rules according to the to-be-processed set received by them; The preliminary sorting information of the multiple processing units is processed to obtain sorting scores corresponding to the entity enhancement rules.
8. An entity enhancement rule mining device applied to big data, characterized in that: include: An acquisition module, used for acquiring a data set and determining an entity enhancement rule set in the data set; The rule types of the entity enhancement rule set include complete rules and incomplete rules; A candidate predicate recording module, used for recording the candidate predicates corresponding to the incomplete rules in the entity enhancement set; The first target rule set processing module is used to determine the ranking score corresponding to the entity enhancement rule in the entity enhancement rule set, generate the first target rule set with the first K complete rules in the ranking score, and delete the entity enhancement rules included in the first target rule set in the entity enhancement rule set. but; A rule type determination module, configured to determine the rule types of the top K entity enhancement rules in the ranking scores when receiving an output request for updating a target rule set; The second target rule set generation module is used to wake up the corresponding incomplete rule according to the to-be-selected predicate when the rule types of the current K entity enhancement rules include the incomplete rule, and generate the second target rule set according to the awakened incomplete rule and the remaining first K entity enhancement rules.
9. A computer device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program implements the method according to any one of claims 1 to 7 when executed by the processor.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.
Citation Information
Patent Citations
Rule dynamic expansion method and system
CN109858906A
Method for discovering association rule from time sequence diagram of event and electronic equipment
CN114330719A
Data processing method and device based on relational data
CN115033650A
Data screening method and device based on reinforcement learning
CN115048425A
Data classification method of association rule based on decision tree pruning enhancement
CN115495497A
Cited By
Self-strengthening total element mining method, device and equipment based on spiral iteration, medium and element mining system
CN121009128A
