A commodity association mining method based on quick frequent item set generation of vertical and horizontal coding

Abstract

The present application relates to a kind of commodity association mining method based on quick frequent item set generation of longitudinal and transverse coding, belong to data mining technical field.The present application utilizes Spark big data processing framework, generates frequent-1 item set by the shopping list data of commodity statistics, generates corresponding length transverse 1 bit string according to the length of frequent-1 item set, according to the order of frequent-1 item set, shopping list is washed and sorted and is converted into 0-1 bit string BitMap, and it is converted into the longitudinal coding corresponding to frequent-1 item set, item set is generated by corresponding length transverse 1 bit string bit operation, simultaneously, item set is calculated using item longitudinal coding, and frequent item set is obtained.The present application compared with traditional Aprior algorithm, mainly solves the problem that Aprior needs to generate a large number of candidate item set, occupies a large amount of memory space and needs to scan database multiple times, and consumes a large amount of time.The present application is applied to the analysis in actual shopping commodity field, and the mining efficiency of frequent item set is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to a kind of commodity association mining method based on fast frequent item set generation of vertical-horizontal coding, belong to data mining technical field. BACKGROUND

[0002] By mining the potential commodity combination relationship in the user's purchase commodity combination in the commodity shopping list, the enterprise can be provided with sales strategy, product association combination, which can improve the sales of enterprise, and also can provide more suitable commodity recommendation for consumers. The frequent item set mining in the field of association rules is a good method to deal with such problems. The traditional Apriori frequent item set mining algorithm needs to generate a large number of candidate item sets, occupies a large amount of memory space and needs to scan the database multiple times, and consumes a lot of time. In view of the above problems, the present application uses bit string, converts each transaction into vertical bit string coding of frequent-1 item, obtains item set by bit string fast iteration algorithm on horizontal 1 bit string, and uses vertical bit string coding to quickly calculate support to obtain frequent item set. And deploy it to distributed platform, greatly speed up its generation rate. SUMMARY

[0003] The present application provides a kind of commodity association mining method based on fast frequent item set generation of vertical-horizontal coding, to solve the problem that Aprior needs to generate a large number of candidate item sets, occupies a large amount of memory space and needs to scan the database multiple times, and consumes a lot of time. The present application is applied to the analysis in the actual shopping commodity field, increases the judgment of item set whether frequent when generating item set, and finally integrates in distributed environment, improves the mining efficiency of frequent item set.

[0004] The technical scheme of the present application is: a kind of commodity association mining method based on fast frequent item set generation of vertical-horizontal coding, the specific steps of the method are as follows:

[0005] Step1 utilizes Spark distributed platform to read commodity shopping list data, sets minimum support degree;

[0006] Step2 commodity shopping list data is split, mapped and combined, and item set with support degree greater than or equal to minimum support degree is screened out, and frequent-1 item set is outputted according to support degree size;

[0007] Step3 commodity shopping list is cleaned and sorted according to frequent-1 item set;

[0008] Step4 0-1 bit string coding is carried out to the cleaned and sorted commodity shopping list data, BitMap is obtained, and BitMap is converted into vertical 0-1 bit string coding corresponding to each frequent-1 item set;

[0009] Step5 corresponding to the length of the frequent-1 item set horizontal 1 bit string encoding generation and iteration through the bit string fast iteration algorithm to get the frequent bit string;

[0010] Step6 frequent bit string conversion to commodity items.

[0011] Further, the specific steps of Step 1 are:

[0012] (1) create a SparkConf object, set the number of partitions in the object, set the memory running size, the number of node running, the main node running and other necessary parameters;

[0013] (2) generate a SparkContext object according to the SparkConf object, which is the interface for the application program to communicate between the Spark cluster;

[0014] (3) read the original commodity shopping list from HDFS into memory through SparkContext, and specify the number of partitions when reading in parallel;

[0015] (4) set the minimum support threshold min_suppport.

[0016] Further, the specific steps of Step 2 are:

[0017] (1) use count() to count the number of elements in RDD;

[0018] (2) use filter() to filter the elements, and the filtering method is to judge whether the support is greater than or equal to the minimum support, if true, the element is filtered through filter and retained in the newly generated RDD;

[0019] (3) use sortWith() to sort the filtered elements, and the sorting method is to judge the size of the element support, descending order.

[0020] Further, the specific steps of Step 3 are:

[0021] Step 1: use filter() to filter the elements, and the filtering method is to judge whether the element exists in the frequent-1 item set, if true, the element is filtered through filter and retained in the newly generated RDD;

[0022] Step 2: use sortWith() to sort the elements, and the sorting method is to judge the size of the element index in the frequent-1 item set, ascending order.

[0023] Further, the specific steps of Step 4 are:

[0024] According to the order of the frequent-1 item set, the item of the frequent-1 item set is 1 if it exists in the shopping list data, and 0 if it does not exist, to generate a BitMap, each column of the BitMap is a vertical encoding of a frequent-1 item set, and thus the BitMap is converted into a vertical 0-1 bit string corresponding to each frequent-1 item set.

[0025] Further, the specific steps of Step 5 are as follows:

[0026] (1) generating a 1-bit string of a corresponding length according to the length of the frequent-1 item set;

[0027] (2) defining i = 1 to represent that the current output contains all subsets of i elements, and defining n to represent the length of the frequent-1 item set;

[0028] (3) determining whether i is less than n, if yes, go to (4), otherwise end;

[0029] (4) defining c = (1 << i) - 1, c is the first subset of all subsets containing i elements;

[0030] (5) determining whether c is less than or equal to (1 << n) - 1, if yes, go to (6), otherwise go to (7);

[0031] (6) performing an AND operation using the vertical encoding of the item, and obtaining the support degree by counting the number of 1s, if it is frequent, storing it, and then calculating the next subset corresponding to the number using c = NextN(c), and then going to (5);

[0032] (7) executing i++, and then going to (3).

[0033] Further, the specific steps of Step 6 are as follows:

[0034] According to the order of the frequent-1 item set, the item of the frequent-1 item set is 1 if it exists in the shopping list data, and 0 if it does not exist, to generate a BitMap, each column of the BitMap is a vertical encoding of a frequent-1 item set, and thus the BitMap is converted into a vertical 0-1 bit string corresponding to each frequent-1 item set.

[0035] The beneficial effects of the present application are: the present application utilizes the Spark big data processing framework, generates frequent-1 item set by counting the shopping list data of the goods, generates corresponding length 1 bit string according to the length of the frequent-1 item set, cleans and sorts the shopping list according to the order of the frequent-1 item set and converts it into the form of 0-1 bit string, generates item set through corresponding length 1 bit string bit operation, and calculates the support of the item set by using item coding, and obtains the frequent item set. Compared with the prior art, the present application mainly solves the problems of generating a large number of candidate item sets, occupying a large amount of memory space, and needing to scan the database multiple times, consuming a large amount of time, etc. In the analysis of the actual shopping commodity field, it can be judged whether the item set is frequent while generating the item set, and finally integrated in the distributed environment, which improves the mining efficiency of the frequent item set. BRIEF DESCRIPTION OF DRAWINGS

[0036] Fig. 1 is the general flowchart of the present application;

[0037] Fig. 2 is the 0-1 bit string encoding process flowchart of the shopping list data of the goods;

[0038] Fig. 3 is the frequent bit string generation flowchart. DETAILED DESCRIPTION

[0039] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application will be further described in detail in combination with the following drawings. The following described examples are only part of the embodiments of the present application, not all the embodiments. Based on the examples in the present application, all belong to the protection scope of the present application.

[0040] As shown in Figs. 1-3 , a commodity association mining method based on fast frequent item set generation of vertical and horizontal coding, the specific steps of the method are as follows:

[0041] Step 1: as shown in Table 1, there are ten commodity shopping list data records, and the commodity shopping list is read by using the Spark distributed platform, which is converted into Row type RDD. Set the minimum support degree to 0.2.

[0042] Table 1 Commodity shopping list

[0043]

[0044] Step 2: Splitting, mapping and merging the data can get {bread=>7, milk=>8, juice=>3, cheese=>1, cheese=>2, egg=>6, potato=>1, coke=>1}. According to the minimum support, the item set with support greater than or equal to the minimum support is filtered out {bread=>7, milk=>8, juice=>3, cheese=>2, egg=>6}, and the frequent-1 item set is output in order of support size, as shown in Table 2.

[0045] Table 2 frequent-1 item set

[0046]

[0047] Step 3: The shopping list is cleaned according to the frequent-1 item set, and only the items existing in the frequent-1 item set are retained. The cleaned data is sorted according to the order of the frequent-1 item set. As shown in Table 3.

[0048] Table 3 shopping list after cleaning and sorting

[0049]

[0050] Step 4: The shopping list after cleaning and sorting is encoded by 0-1 bit string. If it exists, it is 1, and if it does not exist, it is 0. The corresponding BitMap is obtained. For example, the first data {milk, bread, juice} exists in the frequent-1 item set {milk, bread, juice}, {egg, cheese} does not exist, so the encoding is {1, 1, 0, 1, 0}. The specific encoding is shown in Table 4. According to the BitMap, the vertical encoding of the frequent-1 item set is obtained, as shown in Table 5.

[0051] Table 4 BitMap of shopping list after cleaning and sorting

[0052]

[0053] Table 5 frequent-1 item set vertical encoding

[0054]

[0055] Step 5: The length of the frequent-1 item set is 5, so the 1-bit string encoding corresponding to the length of the frequent-1 item set is {1, 1, 1, 1, 1}. The 0-1 subset generation is performed on the encoding {1, 1, 1, 1, 1}, and 2 5 subsets are generated. The vertical encoding of each item is used for AND operation, and the number of 1s is calculated to obtain the support of the item set. The infrequent item set is discarded, and 8 frequent item sets are generated. The specific process is shown in Table 6.

[0056] Table 6 bit string generation and support calculation

[0057]

[0058] Step 6: The frequent item sets generated in Step 5 are mapped in order by the frequent-1 item sets, and the frequent item sets {{cheese, milk}, {juice, milk}, {juice, bread}, {milk, bread}, {milk, egg}, {bread, egg}, {juice, milk, bread}, {milk, bread, egg}} are obtained. The frequent item sets are combined with the frequent-1 item sets, and the final item sets of the goods {{milk}, {bread}, {egg}, {juice}, {cheese}, {cheese, milk}, {juice, milk}, {juice, bread}, {milk, bread}, {milk, egg}, {bread, egg}, {juice, milk, bread}, {milk, bread, egg}} are obtained.

[0059] The specific embodiments of the present application are described in detail above with reference to the accompanying drawings, but the present application is not limited to the above-described embodiments, and various changes can be made within the knowledge of those skilled in the art without departing from the spirit of the present application.

Claims

1. A commodity association mining method based on fast frequent itemset generation of vertical coding, characterized in that, Comprising the following steps: Step1 reading commodity shopping list data using Spark distributed platform, setting minimum support; Step2 split mapping and merging commodity shopping list data, screening out item set with support greater than or equal to minimum support, and sorting output frequent-1 item set according to support size; Step3 cleaning and sorting commodity shopping list according to frequent-1 item set; Step4 0-1 bit string encoding of cleaned and sorted commodity shopping list data, obtaining BitMap, and converting BitMap into vertical 0-1 bit string encoding corresponding to each frequent-1 item set; Step5 horizontal 1 bit string encoding corresponding to frequent-1 item set length is generated, and frequent bit string is obtained by bit string fast iteration algorithm; Step6 converting frequent bit string into commodity item; The specific steps of Step4 are: According to the order of frequent-1 item set, each commodity shopping list data is encoded, and the existence of frequent-1 item set in commodity shopping list data is 1, and the nonexistence is 0, BitMap is generated, and each column of BitMap is a vertical encoding of a frequent-1 item set, and BitMap is converted into vertical 0-1 bit string encoding corresponding to each frequent-1 item set; The specific steps of Step5 are: (1) generating 1 bit string corresponding to the length of frequent-1 item set according to the length of frequent-1 item set; (2) defining i = 1 represents the current output containing i elements, and defining n represents the length of frequent-1 item set; (3) judging whether i is less than n, if yes, turning to (4), otherwise ending; (4) defining c = (1 << i) - 1, c is the first subset of all subsets containing i elements; (5) judging whether c is less than or equal to (1 << n) - 1, if yes, turning to (6), otherwise turning to (7); (6) performing AND operation with the vertical encoding of the item, and obtaining the support by counting the number of 1, if frequent, storing, then calculating the next subset corresponding to the number by c = NextN(c), and then turning to (5); (7) executing i++, and then turning to (3). 2.The commodity association mining method based on fast frequent itemset generation using vertical coding according to claim 1, wherein, The specific steps of Step 1 are: (1) creating a SparkConf object, setting the number of partitions in the object, setting the memory running size, the number of node running, and the master node running; (2) generating SparkContext object according to SparkConf object, which is the interface for application program to communicate between Spark clusters; (3) reading the original commodity shopping list from HDFS into memory through SparkContext, and specifying the partition number when reading in parallel; (4) setting minimum support threshold min_suppport. 3.The commodity association mining method based on fast frequent itemset generation using vertical coding according to claim 1, wherein, The specific steps of Step2 are: (1) using count() to count the number of RDD elements; (2) Using filter(), filtering the elements, the filtering method is to judge whether the support is greater than or equal to the minimum support, if true, the element passes through the filter and is retained in the newly generated RDD; (3) Using sortWith(), sorting the filtered elements, the sorting method is to judge the element support size, descending order sorting. 4.The commodity association mining method based on fast frequent itemset generation using vertical coding according to claim 1, wherein, The specific steps of Step 3 are: Step 1: Using filter(), filtering the elements, the filtering method is to judge whether the element exists in the frequent-1 item set, if true, the element passes through the filter and is retained in the newly generated RDD; Step 2: Using sortWith(), sorting the elements, the sorting method is to judge the element index position size in the frequent-1 item set, ascending order sorting. 5.The commodity association mining method based on fast frequent itemset generation using vertical coding according to claim 1, wherein, The specific steps of Step 6 are: According to the order of the frequent-1 item set, map each frequent bit string, if the bit is 1, add the item of the frequent-1 item set to it; if the bit is 0, do nothing.

Images