A similarity model retrieval method based on improved Apriori algorithm

Abstract

The present invention relates to a similarity model retrieval method based on an improved Apriori algorithm, and belongs to the technical field of similarity retrieval. To improve the efficiency of Apriori association rule mining, the present invention uses an improved Apriori algorithm to achieve rapid incremental updates and threshold rapid updates. Based on the principle that "any subset of a frequent item set must be a frequent item set," existing frequent item sets are used to prune newly added candidate item sets, then new frequent item sets are generated. Association rules are then calculated using the frequent item sets. Finally, a cosine similarity algorithm is used to mine potential similarity models. The present invention provides an efficient technical solution for similarity algorithm model retrieval and has important practical application value.

Description

Technical Field

[0001] The present invention belongs to the technical field of similarity retrieval, and in particular relates to a similarity model retrieval method based on an improved Apriori algorithm. Background Art

[0002] With the rapid development of computer science, algorithm models have become an important achievement of human wisdom. In addition, the development of artificial intelligence is in full swing. As the "soul" of artificial intelligence, machine learning models have attracted more and more attention for their intellectual property protection. However, there are still disputes over copyright and patent protection in the current intellectual property protection of algorithm models. Algorithm models can only be effectively developed in the long term through inheritance and protection. In addition, in this era of knowledge explosion, algorithm model achievements have a huge output every year. The algorithm similarity retrieval task faced with a large number of algorithm model data sets is particularly arduous. When using retrieval technology to retrieve similar algorithm models, its efficiency must be an important consideration. Therefore, the present invention uses an improved Apriori algorithm to mine association rules between algorithm models according to the usage of the algorithm model, improves the mining efficiency of association rules, and then uses the cosine similarity algorithm to mine potential similar models. Finally, the potential similar models are similarity calculated or manually identified at the source code level. This method provides a strong technical guarantee for the inheritance and protection of the algorithm, avoiding the similarity calculation of low similarity correlation algorithm models at the text level, resulting in a waste of computing resources.

[0003] In computer science and data mining, the Apriori algorithm is a classic algorithm for learning association rules. It uses a breadth-first search algorithm and a tree structure to efficiently count candidate itemsets. It generates a candidate itemset of length k from a candidate itemset of length k-1, then removes candidates containing uncommon subpatterns. According to the downward closure theorem, this candidate itemset contains all frequent itemsets of length k. Frequent itemsets within the candidate itemset are then determined by scanning the transaction database.

[0004] Association rule mining was proposed by Rakesh Agrawal et al. at the SIDMOD conference in 1993. After more than two decades of development, it is now widely used in the Internet, e-commerce, medical, finance and other industries. DWCheung et al. proposed the association rule fast update algorithm FUP, which solves the problem of updating association rules when the data set is increased while the minimum support and minimum confidence remain unchanged. In China, Feng Yucai et al. proposed the IUA and PIUA algorithms, which solve the problem of fast updating of association rules when the minimum support and minimum confidence change. The above research shows that there is still a lot of room for improvement in the mining efficiency of Apriori in frequent item sets, and there is also a lot of room for exploration in its application scenarios. The present invention uses the Apriori association rule algorithm to apply to potential similarity model mining, opening up a pioneering application of the Apriori association rule algorithm in the field of similarity retrieval. Summary of the Invention

[0005] (1) Technical issues to be resolved

[0006] The technical problem to be solved by the present invention is: how to design a fast and efficient potential similarity model retrieval solution.

[0007] (2) Technical solution

[0008] In order to solve the above technical problems, the present invention provides a similarity model retrieval method based on an improved Apriori algorithm, comprising the following steps:

[0009] S1. Select a model dataset D, remove duplicates and sort the model dataset D, and create a candidate set C1;

[0010] S2, scan the data set D, calculate the support of a candidate set C1 in the data set D, and determine whether the support is greater than the minimum support minSupport. The item sets greater than the minimum support minSupport form a frequent set L1;

[0011] S3. Perform combination calculations based on L1 to obtain all possible candidate item sets C2. Calculate the support of the two-item candidate set C2 in the data set D. The item sets with a support greater than the minimum support minSupport form the two-item frequent set L2. Iterate this step to obtain the k-item frequent set Lk.

[0012] S4. Recursively calculate the credibility through the frequent item set Lk, record the set whose credibility is greater than the minimum credibility minConf, and generate association rules;

[0013] S5. Add a new dataset Dn and use the Apriori incremental fast update algorithm to quickly generate a new frequent item set Lk. Then recursively calculate the credibility and generate association rules based on the credibility count.

[0014] S6. Update the minimum support minSupport, use the Apriori threshold fast update algorithm to generate a new frequent item set Lk, then recursively calculate the credibility, and generate association rules through the credibility count;

[0015] S7. Use the cosine similarity algorithm to calculate the similarity of the association rules in S6, set a threshold for the cosine similarity to eliminate non-potential similarity models, and obtain the potential similarity model.

[0016] Preferably, the Apriori incremental fast update algorithm is to traverse and calculate the support count of the newly added candidate set when there is a new Dn in the data set D, and the support count of the original candidate set is no longer traversed and calculated, and the candidate set is then used to obtain the frequent set according to the support threshold.

[0017] Preferably, the specific implementation process of the Apriori incremental fast update algorithm is as follows:

[0018] First, the Apriori incremental fast update algorithm stores the support count after each traversal and calculation of the candidate set's support count. Then, when there is a new candidate set in the data set, it traverses and calculates the support count of the new candidate set, then reads the support count of the original candidate set, adds the support count of the original candidate set to the support count of the new candidate set, and finally, divides the support count of the merged candidate set by the total length of the updated data set to obtain the support of each candidate set in the updated data set. The candidate set then obtains the frequent set based on the support threshold.

[0019] Preferably, in the update model of the data set, ssCnt is set as the support, lenOld is the length of the original data set, and lenNew is the length of the new data set after the newly added data set is inserted. The support of the original frequent item set is recalculated through the formula newCnt = ssCnt / (LenOld+LenNew), the original infrequent item set and the newly added candidate set are combined to calculate the support, the support of the original frequent item set and the support calculated by the union are added, and then a threshold judgment is performed to obtain the new frequent item set.

[0020] Preferably, the Apriori threshold fast update algorithm only traverses and calculates the new candidate sets generated by the threshold update after the support threshold is updated, and the candidate sets that originally meet the support threshold are no longer traversed and calculated, and the candidate sets are then used to derive frequent sets based on the new support threshold.

[0021] Preferably, the support threshold update includes two cases: Case 1, the new support threshold is greater than the original support threshold. In this case, the support count of the frequent itemsets is used to delete the frequent itemsets that do not meet the newSupport by traversing; Case 2, the new support threshold is smaller than the original support threshold. In this case, the frequent item set newL1 is recalculated to obtain a new frequent item set L1′=newL1-L1, and the frequent item sets Lk that meet the new minimum support newSupport are divided into three categories:

[0022] ①The first frequent set Lk1: the candidate set is obtained by apriori_gen(Lk1-1);

[0023] ② The second frequent set Lk2: the candidate set is obtained by apriori_gen(Lk2-1′);

[0024] ③The third frequent set Lk3: the candidate set is obtained by apriori_gen(Lk3-1,Lk3-1′);

[0025] These three types of frequent sets are processed separately, and the apriori_gen() function is used to calculate all possible candidate sets composed of frequent item sets.

[0026] Preferably, for type ①, this part of the candidate set is pruned by apriori_gen(Lk1-1)-Lk1, and the support of the remaining candidate data sets in data set D is calculated.

[0027] Preferably, for type ②, the support of the candidate set apriori_gen(Lk2-1′) in the data set D is directly calculated.

[0028] Preferably, for type ③, the first frequent set Lk1 and the second frequent set Lk2 are concatenated, and the principle of "any subset of a frequent item set must be a frequent item set" is used to detect whether the subset of the candidate set is a frequent item set. If not, this part of the candidate set is pruned.

[0029] The present invention also provides an application of the method in the technical field of similarity retrieval.

[0030] (3) Beneficial effects

[0031] To improve the efficiency of Apriori association rule mining, this paper uses an improved Apriori algorithm to achieve rapid incremental updates and threshold updates. Based on the principle that "any subset of a frequent itemset must be a frequent itemset," this paper uses existing frequent itemsets to prune new candidate itemsets, then generates new frequent itemsets. Association rules are then calculated using these frequent itemsets. Finally, a cosine similarity algorithm is used to mine potential similarity models.

[0032] The present invention adopts the improved Apriori algorithm to mine model association rules, and then uses the obtained association rules to mine similar models, which has the following advantages and beneficial effects:

[0033] 1. The present invention improves the Apriori algorithm by using the Apriori incremental fast update algorithm and the Apriori threshold fast update algorithm, which can mine potential similar models in a targeted manner according to the usage of the model, and has the characteristics of high efficiency and speed.

[0034] 2. The present invention uses support and credibility thresholds to control data generation and achieve rapid support updates. Users can mine data to obtain potential similarity models according to their needs.

[0035] The present invention provides an efficient technical solution for similar algorithm model retrieval and has important practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flowchart of potential similarity model mining of the present invention;

[0037] Figure 2 This is a diagram of a dynamic update model of the support threshold of the present invention. DETAILED DESCRIPTION

[0038] In order to make the purpose, content, and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0039] The present invention provides a similarity model retrieval method based on an improved Apriori algorithm, which is a fast and efficient potential similarity model retrieval solution. Addressing the issues of copyright and patent protection for algorithmic models, the present invention proposes a similarity model retrieval method based on an improved Apriori algorithm. Based on the usage of the model, the method quickly and targetedly mines potential similarity models to improve the efficiency of similarity model retrieval. The improved Apriori algorithm is used to mine model association rules, and the cosine similarity algorithm is used to calculate the similarity of association vectors to mine potential similarity models. Some basic concepts related to the present invention are:

[0040] 1. Apriori Algorithm: The Apriori algorithm is a frequent itemset algorithm for mining association rules. It was proposed by Rakesh Agrawal and Ramakrishnan Skrikant. It uses an iterative method called level-by-level search, where k itemsets are used to explore (k+1) itemsets. First, a set of frequent one-itemsets is found. This set is denoted as L1. L1 is used to find a set of frequent two-itemsets, L2, which is then used to find L3. This process continues until no frequent k-itemsets can be found. Finding each Lk requires a database scan.

[0041] 2. Candidate Set: The k-dimensional candidate set is denoted as Ck. One candidate set C1 is obtained by deduplicating and sorting the dataset. In addition, Ck+1 candidate sets are obtained by combining the frequent set Lk. The calculation formula is expressed as Ck = apriori_gen(Lk-1).

[0042] 3. Frequent set: The candidate set with greater than the minimum support is the frequent item set. Among them, the necessary condition for the k-dimensional data item set Lk to be a frequent item set is that all its k-1-dimensional sub-item sets are also frequent item sets, denoted as Lk-1.

[0043] 4. Association rule: {A}->{B} is an association rule, which means that after an individual uses model A, he also uses model B.

[0044] 5. Support: By scanning the data set, calculate the proportion of records in the data set that contain the item set. This proportion is the support. The support of data set A is expressed as support(A).

[0045] 6. Credibility: Defined for a specific association rule such as {A}->{B}, the credibility of a rule {A}->{B} is defined as support(A|B) / support(A).

[0046] 7. k-itemset: If event A contains k elements, then this event A is called a k-itemset, and the event A that meets the minimum support threshold is called a frequent k-itemset.

[0047] 8. Cosine Similarity: Cosine similarity, also known as cosine similarity, evaluates the similarity between two vectors by calculating the cosine of the angle between them. Cosine similarity plots vectors into a vector space, such as the most common two-dimensional space, based on their coordinate values.

[0048] In order to solve the copyright and patent protection issues of model algorithms, there are problems such as large amount of data in existing algorithm models and difficulty in identifying similarities. The present invention uses a method based on the improved Apriori algorithm to mine potential similar models. It can achieve efficient and rapid updates of frequent item sets when the data set and support are dynamically updated. Frequent item sets are then used to calculate association rules. Then, the cosine similarity algorithm is used to calculate the association rule vector to mine potential similar models. Finally, text similarity calculation or manual identification are used to identify whether the models are similar. Figure 1 As shown, the present invention includes the following steps:

[0049] S1. Select a model dataset D, remove duplicates and sort the model dataset D, and create a candidate set C1;

[0050] S2, scan the data set D, calculate the support of a candidate set C1 in the data set D, and determine whether the support is greater than the minimum support minSupport. The item sets greater than the minimum support minSupport form a frequent set L1;

[0051] S3. Perform combination calculations based on L1 to obtain all possible candidate item sets C2. Calculate the support of the two-item candidate set C2 in the data set D. The item sets with a support greater than the minimum support minSupport form the two-item frequent set L2. Iterate this step to obtain the k-item frequent set Lk.

[0052] S4. Recursively calculate the credibility through the frequent item set Lk, record the set whose credibility is greater than the minimum credibility minConf, and generate association rules rules;

[0053] S5. Add a new dataset Dn and use the Apriori incremental fast update algorithm to quickly generate a new frequent item set Lk. Then recursively calculate the credibility and generate association rules based on the credibility count.

[0054] S6. Update the minimum support minSupport, use the Apriori threshold fast update algorithm to generate a new frequent item set Lk, then recursively calculate the credibility, and generate association rules rules through the credibility count;

[0055] S7. Use the cosine similarity algorithm to calculate the similarity of the association rules in S6, set a threshold for the cosine similarity to eliminate non-potential similarity models, and obtain the potential similarity model.

[0056] In this embodiment, the specific processes of the Apriori incremental fast update algorithm and the Apriori threshold fast update algorithm are as follows:

[0057] (1) Apriori incremental fast update algorithm

[0058] The purpose of incremental fast update is to traverse and calculate the support counts of only the newly added candidate sets Dn in the dataset D. The support counts of the existing candidate sets are no longer traversed and calculated. The candidate sets are then used to determine the frequent set based on the support threshold. This method avoids repeated traversal calculations and achieves rapid updates of frequent sets.

[0059] First, after each traversal and calculation of the support count of the candidate set by the Apriori incremental fast update algorithm, the support count is stored. This step is to avoid recalculating the support count of the candidate sets that have already been calculated when new data is added to the dataset. Then, when new candidate sets are added to the dataset, traverse and calculate the support count of the new candidate sets. Then read the support count of the original candidate sets, and add the support count of the original candidate sets to the support count of the new candidate sets. Finally, divide the support count of the merged candidate sets by the total length of the updated dataset to obtain the support of each candidate set in the updated dataset. The candidate sets are then used to obtain the frequent sets based on the support threshold.

[0060] The dataset dynamic update model is as Figure 1 shown. Among them, ssCnt is the support count, lenOld is the length of the original dataset, lenNew is the length of the new dataset after inserting the new added dataset. By the formula newCnt = ssCnt / (LenOld + LenNew), recalculate the support of the original frequent item sets. The union of the original infrequent item sets and the new candidate sets is calculated for support. Add the support of the original frequent item sets to the support of the above union and then make a threshold judgment to obtain the new frequent item sets.

[0061] (2) Apriori threshold fast update algorithm

[0062] The purpose of the threshold fast update is to only traverse and calculate the new candidate sets generated due to the threshold update after the support threshold is updated. The candidate sets that originally met the support threshold are no longer traversed and calculated. The candidate sets are then used to obtain the frequent sets based on the new support threshold, thus improving the update efficiency of the frequent sets. The support threshold update includes two cases: the new support threshold is larger than the original support threshold, and the new support threshold is smaller than the original support threshold.

[0063] The support threshold dynamic update is as Figure 1 shown in the dataset dynamic update model shown. Case 1, newSupport > oldSupport. Some of the original frequent item sets may lose the minimum support. Use the support count of the frequent item sets and delete the frequent item sets that do not meet newSupport by traversing. Case 2, newSupport < oldSupport. Recalculate the frequent one-item set newL1, and obtain the newly added frequent one-item set L1′ = newL1 - L1. Divide the frequent item sets Lk that meet the new minimum support newSupport into three categories:

[0064] ① The first type of frequent set Lk: The candidate sets are obtained through apriori_gen(Lk - 1)

[0065] ② The second type of frequent set Lk: The candidate set is obtained through apriori_gen(Lk-1′).

[0066] ③ The third type of frequent set Lk: The candidate set is obtained through apriori_gen(Lk-1, Lk-1′).

[0067] Among them, the apriori_gen() function is used to calculate all possible candidate sets composed of frequent item sets;

[0068] For type ①, since newSupport < oldSupport, the original frequent sets that meet the minimum support oldSuppport must meet newSupport. Therefore, the candidate sets in this part are pruned by the way of apriori_gen(Lk-1) - Lk, and the support degrees of the remaining candidate data sets in the data set D are calculated. For type ②, the support degree of the candidate set apriori_gen(Lk-1′) in the data set D is directly calculated. For type ③, the first type of frequent set Lk and the second type of frequent set Lk are concatenated, and whether the subsets of the candidate sets are frequent item sets is detected according to the principle that any subset of a frequent item set must be a frequent item set. If not, the candidate set is pruned.

[0069] The above is only the preferred implementation manner of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and deformations can be made, and these improvements and deformations should also be regarded as the protection scope of the present invention.

Claims

1. A similarity model retrieval method based on an improved Apriori algorithm, characterized in that: The model is an algorithmic model, and the method includes the following steps: S1. Select a model dataset D, remove duplicates and sort the model dataset D, and create a candidate set C1; S2, scan the data set D, calculate the support of a candidate set C1 in the data set D, and determine whether the support is greater than the minimum support minSupport. The item sets greater than the minimum support minSupport form a frequent set L1; S3. Perform combination calculations based on L1 to obtain all possible candidate item sets C2. Calculate the support of the two-item candidate set C2 in the data set D. The item sets with a support greater than the minimum support minSupport form the two-item frequent set L2. Iterate this step to obtain the k-item frequent set Lk. S4. Recursively calculate the credibility through the frequent item set Lk, record the set whose credibility is greater than the minimum credibility minConf, and generate association rules; S5. Add a new data set Dn and use the Apriori incremental fast update algorithm to generate a new frequent item set Lk. Then recursively calculate the credibility and generate association rules based on the credibility count. S6. Update the minimum support minSupport, use the Apriori threshold fast update algorithm to generate a new frequent item set Lk, then recursively calculate the credibility, and generate association rules through the credibility count; S7, using the cosine similarity algorithm to calculate the similarity of the association rules in S6, setting a threshold for the cosine similarity to eliminate non-potential similar models, and obtaining potential similar models; finally, determining whether the models are similar by calculating text similarity; The Apriori incremental fast update algorithm is that when a new Dn is added to the dataset D, only the support count of the newly added candidate set is traversed and calculated, and the support count of the original candidate set is no longer traversed and calculated. The candidate set is then used to obtain the frequent set according to the support threshold; The Apriori threshold fast update algorithm is that after the support threshold is updated, only the new candidate sets generated by the threshold update are traversed and calculated. The candidate sets that originally meet the support threshold are no longer traversed and calculated, and the candidate sets are then used to obtain the frequent sets based on the new support threshold.

2. The method according to claim 1, wherein There are two cases for updating the support threshold: Case 1: the new support threshold is larger than the original support threshold. In this case, the support count of the frequent itemset is used to delete the frequent itemsets that do not meet the newSupport by traversing. Case 2: the new support threshold is smaller than the original support threshold. In this case, the frequent itemset newL1 is recalculated to obtain the newly added frequent itemset L1′=newL1-L1. The frequent itemsets Lk that meet the new minimum support newSupport are divided into three categories: ①The first frequent set Lk1: the candidate set is obtained by apriori_gen(Lk1-1); ② The second frequent set Lk2: the candidate set is obtained by apriori_gen(Lk2-1′); ③The third frequent set Lk3: the candidate set is obtained by apriori_gen(Lk3-1,Lk3-1′); These three types of frequent sets are processed separately, and the apriori_gen() function is used to calculate all possible candidate sets composed of frequent item sets.

3. The method according to claim 2, wherein For type ①, prune this part of the candidate set by apriori_gen(Lk1-1)-Lk1, and calculate the support of the remaining candidate data sets in data set D.

4. The method according to claim 2, wherein For type ②, directly calculate the support of the candidate set apriori_gen(Lk2-1′) in the dataset D.

5. The method according to claim 2, wherein For type ③, concatenate the first frequent set Lk1 and the second frequent set Lk2, and use the principle of "any subset of a frequent item set must be a frequent item set" to detect whether the subset of the candidate set is a frequent item set. If not, prune this part of the candidate set.

6. Application of the method according to any one of claims 1 to 5 in the field of similarity retrieval technology.

Images