Data mining method, apparatus, device, and medium

Abstract

This disclosure provides a data mining method, apparatus, device, and storage medium, relating to the field of artificial intelligence technology, specifically to big data, data analysis, and other technical fields. The data mining method includes: using a first query statement to obtain the sorting result of target items in an original dataset; using a second query statement to call a preset function, and using the preset function to perform data mining processing on the original dataset based on the sorting result of the target items, to obtain data mining results. This disclosure allows for simple data mining processing.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, specifically to the fields of computer vision, deep learning, and more particularly to a data mining method, apparatus, device, and medium. Background Technology

[0002] Frequent Pattern Growth (FP-Growth) is an association rule mining algorithm. Association rule mining algorithms can discover frequent itemsets and their relationships from large amounts of data stored in databases or other information.

[0003] In related technologies, the FP-Growth algorithm is mainly implemented by programming languages ​​such as Python, and then the FP-Growth algorithm is used to mine frequent itemsets and their associations. Summary of the Invention

[0004] This disclosure provides a data mining method, apparatus, device, and storage medium.

[0005] According to one aspect of this disclosure, a data mining method is provided, comprising: using a first query statement to obtain the sorting result of target items in an original dataset; using a second query statement to call a preset function, and using the preset function to perform data mining processing on the original dataset according to the sorting result of the target items, so as to obtain data mining results.

[0006] According to another aspect of this disclosure, a data mining apparatus is provided, comprising: a sorting module, configured to use a first query statement to obtain a sorting result of target items in an original dataset; and a mining module, configured to use a second query statement to call a preset function, and use the preset function to perform data mining processing on the original dataset according to the sorting result of the target items, so as to obtain data mining results.

[0007] According to another aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to said at least one processor; wherein the memory stores instructions executable by said at least one processor, said instructions being executed by said at least one processor to enable said at least one processor to perform the method as described in any of the foregoing aspects.

[0008] According to another aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method according to any of the preceding aspects.

[0009] According to another aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method according to any of the preceding aspects.

[0010] According to the technical solution disclosed herein, data mining processing can be performed easily.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0014] Figure 2 This is a schematic diagram illustrating an application scenario provided according to embodiments of this disclosure;

[0015] Figure 3 This is a schematic diagram of the overall architecture of a data mining system provided according to an embodiment of this disclosure;

[0016] Figure 4 This is a schematic diagram of the execution flow of the SQL module provided according to an embodiment of this disclosure;

[0017] Figure 5 This is a schematic diagram of the execution flow of a UDF module provided according to an embodiment of this disclosure;

[0018] Figure 6 This is a schematic diagram according to the second embodiment of the present disclosure;

[0019] Figure 7 This is a schematic diagram according to the third embodiment of the present disclosure;

[0020] Figure 8 This is a schematic diagram of an electronic device used to implement the data mining method of the embodiments of this disclosure. Detailed Implementation

[0021] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0022] In related technologies, the FP-Growth algorithm is implemented independently using programming languages ​​such as Python. This requires users to have programming skills, relies on programming languages ​​and frameworks, has high coding costs, and is relatively complex to implement.

[0023] To simplify data mining processes, the present disclosure provides the following embodiments.

[0024] To better understand the embodiments of this disclosure, the terminology used in these embodiments is explained as follows:

[0025] A dataset is a collection of data. A dataset can contain at least one transaction, and each transaction can contain at least one data item.

[0026] Taking a shopping scenario as an example, each data item can be a category of goods. For example, milk is a data item, beer is a data item, and diapers are a data item.

[0027] Suppose there are four transactions, labeled T1, T2, T3, and T4, and the various products involved are labeled a, b, c, d, and e. The dataset can record the information shown in Table 1.

[0028] Table 1

[0029] transactions Product categories T1 a、b T2 a, b, c T3 b, c, d, e T4 a, b, c

[0030] An itemset is a set of data items. An itemset containing k data items is called a k-itemset. For example, the set {milk, beer, diapers} is a three-itemset.

[0031] The occurrence count of an itemset refers to the number of transactions in the dataset that contain that itemset. Taking an itemset (i.e., each data item) as an example, based on the dataset in Table 1, the occurrence counts of each itemset (data item) are as follows: a = 3, b = 4, c = 3, d = 1, and e = 1. For another example, taking a two-item set {a, b}, the occurrence count of this two-item set is 3.

[0032] The support of an itemset is the ratio of the number of transactions containing that itemset to the total number of transactions in the itemset. Taking the itemset 'a' above as an example, the support of 'a' is 3 / 4.

[0033] A frequent itemset is an itemset whose frequency of occurrence (or support) is greater than or equal to a preset threshold. If the threshold for frequency of occurrence is 2, then in the example above, a, b, and c are frequent itemsets.

[0034] Association rules are used to represent the implicit relationships between data. Taking the association rule between itemsets as an example, suppose two itemsets are represented by X and Y respectively. If X is a prerequisite and Y is the association result, then the association rule between the two can be represented as X → Y.

[0035] The confidence score of an association rule indicates the probability that an association outcome (e.g., Y) will occur given certain preconditions (e.g., X). Based on the example above, the confidence score for a → {a, b} is 3 / 3.

[0036] Figure 1 Based on a schematic diagram of the first embodiment of this disclosure, this embodiment provides a data mining method, which includes:

[0037] 101. Use the first query statement to obtain the sorting results of the target items in the original dataset.

[0038] 102. Using a second query statement, a preset function is invoked, and the preset function performs data mining processing on the original dataset based on the sorting result of the target item to obtain data mining results.

[0039] Query statements, in this context, refer to general, basic statements capable of manipulating datasets, such as Structured Query Language (SQL) statements. SQL is a database language with multiple functions, including data manipulation and definition. This interactive language provides great convenience to users, and database management systems can utilize SQL to improve the quality and efficiency of computer application systems.

[0040] Since the query statement is general and basic, it does not require the user to understand programming languages ​​such as Python, and the coding ability requirement for the user is low, making it easier to implement.

[0041] To distinguish between them, datasets include the original dataset and local datasets. The original dataset refers to the entire dataset, such as one containing 100 transactions. The local dataset refers to the dataset after the original dataset has been grouped. For example, if the 100 transactions are divided into 10 groups, then each group contains 10 transactions, and each group of 10 transactions can form a local dataset.

[0042] A target item refers to a data item in the original dataset that needs to be processed. Each data item can be designated as a target item based on user needs, or data items that meet preset conditions can be designated as target items. Based on the example in Table 1, each product (a, b, c, d, e) can be a target item. Alternatively, a target item can refer to a frequent item set; assuming a threshold of occurrence frequency = 2, then a, b, and c can be considered target items.

[0043] Taking each data item in the original dataset as the target item as an example, the target items can be sorted in descending order of their frequency of occurrence. The sorted data items are then used as the sorting result for the target items. Furthermore, the sorting result can be in array form. Therefore, based on the above example, the sorting result for the target items is [b, a, c, d, e]. The sorting order of two data items with the same frequency is not limited. For example, if 'a' and 'c' have the same frequency of occurrence, the sorting order for 'a' and 'c' can be either a, c, or c, a.

[0044] The data mining method in this embodiment can be executed by a computing engine. A preset function refers to a function set within the computing engine. This preset function can be built into the computing engine itself, or it can be provided by another provider and registered by the user (the user) within the computing engine. When provided by another provider, the preset function can be a user-defined function (UDF) provided by that provider. A UDF is a function written by the provider itself to extend the logical functionality of the computing engine, specifically providing data mining functionality.

[0045] Users can register the UDF provided by the provider into the computing engine, and then call the UDF for data mining processing. Since the UDF is provided by the provider, users can call the UDF without having to code it themselves.

[0046] After the user calls the UDF, they can use the UDF and the sorting results of the target items to perform data mining processing on the dataset to obtain data mining results.

[0047] Data mining results can specifically include: frequent itemsets in the dataset, as well as association rules between frequent itemsets and their confidence levels.

[0048] In this embodiment, the sorting result of the target item is obtained through the first query statement, and the preset function is called through the second query statement. The preset function is used to obtain the data mining result based on the sorting result. For the user, the user only needs to use the query statement and call the already implemented preset function to perform data mining processing. There is no need to use programming languages ​​such as Python. Therefore, data mining processing can be performed easily without the user needing to master programming languages ​​such as Python.

[0049] To better understand the embodiments of this disclosure, the application scenarios to which the embodiments of this disclosure are applicable are described below.

[0050] like Figure 2As shown, the data mining system includes a user terminal 201 and a server 202. The user terminal 201 can transmit the raw dataset to the server 202, and the server 202 can use data mining algorithms to perform data mining processing on the raw dataset to obtain data mining results. Examples of data mining algorithms include the Apriori algorithm or the FP-Growth algorithm. Figure 2 Taking the FP-Growth algorithm as an example, data mining results can include: frequent itemsets in the dataset, association rules between frequent itemsets and their confidence levels, etc. Figure 2 Taking frequent itemsets as an example, server 202 can then return the data mining results to user terminal 201 for display. User terminals may include, for example, personal computers, laptops, and mobile devices (such as mobile phones). The server can be a local server or a cloud server. The user terminal and server can communicate via wired and / or wireless networks. It is understood that if the user terminal itself has data mining capabilities, it can also perform data mining processing locally on the user terminal to obtain and display the data mining results.

[0051] Taking the FP-Growth algorithm as an example, in related technologies, the FP-Growth algorithm is implemented based on a specific programming language, such as Python.

[0052] Taking SQL statements as the query statement and UDF as the preset function as the example, this embodiment mainly implements data mining processing based on SQL and UDF (SQL+UDF).

[0053] like Figure 3 As shown, the overall structure of the data mining system includes an SQL module 301 and a UDF module 302. The SQL module 301 uses a first query statement to obtain the sorting results of the target items in the original dataset; the UDF module 302 uses a second query statement to call a preset function, and uses the preset function to perform data mining processing on the original dataset based on the sorting results of the target items to obtain data mining results. Figure 3 Taking the data mining results as frequent itemsets as an example.

[0054] For the SQL module:

[0055] The ranking result of the target items can be obtained based on the number of times the target items appear.

[0056] Accordingly, the step of using the first query statement to obtain the sorting result of the target items in the dataset may include: using the first query statement to obtain the occurrence count of the target items in the dataset; and using the first query statement to sort the target items based on the occurrence count to obtain the sorting result of the target items.

[0057] In this embodiment, the ranking result of the target item is obtained based on the occurrence frequency of the target item, which can easily and efficiently obtain the ranking result of the target item, thereby improving the efficiency of data mining.

[0058] Specifically, the first query statement includes a grouping clause and a counting clause; obtaining the occurrence count of a target item in the dataset using the first query statement includes: using the grouping clause to divide the same target item in the dataset into the same group; and using the counting clause to count the occurrence count of the target item in each group to obtain the occurrence count of the target item.

[0059] In this embodiment, the number of occurrences of the target item can be easily obtained using SQL clauses through grouping and counting clauses, providing a data foundation for subsequent processing.

[0060] Further, the first query statement also includes: a window clause, a first concatenation clause, a sorting clause, a transformation clause, a second concatenation clause, a matching clause, and an array clause; the step of using the first query statement to sort the target items based on the occurrence count to obtain the sorting result of the target items includes: using the window clause to assign a sequence number to the target items based on the occurrence count, and constructing an item header table based on the sequence number and occurrence count of the target items, wherein each row element of the item header table includes the sequence number and occurrence count of each target item; using the first... A concatenation clause is used to concatenate the sequence number and the number of trips for each row element in the item header table using a separator; the sorting clause and the transformation clause are used to sort each row element based on the sequence number contained in each row element; the second concatenation clause and the matching clause are used to concatenate each sorted row element, and the sequence number and the separator are removed from each concatenated row element to obtain a string composed of sorted target items; the array clause is used to convert the string into an array, and the array is used as the sorting result of the target items.

[0061] In this embodiment, the sorting result of the target item can be obtained based on the number of times the target item appears through the above clause in the SQL statement, so as to obtain the sorting result of the target item based on the SQL statement without the need for the user to use programming languages ​​such as Python, which is more convenient.

[0062] For details on obtaining the sorting results of the target items based on SQL statements, please refer to [link / reference]. Figure 4 .

[0063] like Figure 4 As shown, it mainly includes:

[0064] 1) Use grouping clauses (such as those containing the GroupBy function) and counting clauses (such as those containing the count function) to count the occurrences of each data item in the original dataset.

[0065] If the target item is a data item, the number of times each data item appears can be used as the number of times each target item appears; or, if the target item is a frequent item set, a filtering function in the SQL statement can be used to remove data items whose number of occurrences is less than a threshold in order to obtain the target item and its number of occurrences.

[0066] Assuming the original dataset contains 5 data items A, B, C, D, and E, and taking the target item as an example, the target item and its frequency can be obtained as follows: Figure 4 As shown, the target items are represented by itemsets and the number of occurrences are represented by counts. For example, the target items include A, and the number of occurrences of A is 100. The target items also include B, and the number of occurrences of B is 150. The rest are similar.

[0067] 2) Use a window clause (such as including the row_number function) to sort each target item in descending order of the count in 1) and assign an index (rn). Construct an item header table based on the index and the target item, where each row of the item header table includes the target item and its index.

[0068] 3) For each row of elements, use a concatenation clause (for distinction, this concatenation clause can be called the first concatenation clause, which can specifically include the `concat_ws` function) to concatenate the sequence number and the corresponding target item. Additionally, the sequence number can be set to a fixed number of digits; if insufficient, `lpad` can be used to pad the number of digits to ensure the accuracy of subsequent sorting. For example: `concat_ws(':',lpad(cast(rn as string),3,'0'),xxx)`.

[0069] After assigning serial numbers and concatenating them, we can obtain, for example: Figure 4 The item header table shown has each row containing a sequence number (rn) and the corresponding target item. For example, the first row contains the element 004:A, and the rest are similar.

[0070] 4) Group by dimension field, and use transformation clauses (such as those containing the collect_set function) and sorting clauses (such as those containing the sort_array function) to sort the target items under the same dimension according to the prefix rn index.

[0071] 5) Use the concatenation clause again (for distinction, this concatenation clause can be called the second concatenation clause, which may include the concat_ws function) to concatenate the results in 4), and use the matching clause (such as including the regexp_replace regular expression matching function) to remove the redundant sequence number (rn) and separator (:) before the target item.

[0072] Among them, such as Figure 4 As shown, after concatenation, we can obtain strings like 001:C,002:E,003:B,004:A,005:D. After removing the numbers and delimiters, we can obtain strings like C,E,B,A,D.

[0073] 6) Finally, use an array clause (such as including the split function) to convert the above string into an array.

[0074] Among them, such as Figure 4 As shown, the array is [C,E,B,A,D], which is the sorting result of the target items, and then used as the input parameter of the UDF.

[0075] For the UDF module:

[0076] After obtaining the sorting result of the target item, the sorting result can be used as the input parameter of the UDF, and the UDF can be called to perform data mining processing.

[0077] The second query statement may include a selection clause; the step of using the second query statement to call a preset function includes using the selection clause to call the preset function.

[0078] For example, if the default function is represented by fp_tree and the selection clause is identified by the keyword select, then the UDF can be called using the following clause: select fp_tree(data,[C,E,B,A,D],...), where data is the information of the original dataset, [C,E,B,A,D] is the sorting result of the target items, and the input parameters can also include other parameters, such as minimum support, minimum confidence, number of association rules, etc.

[0079] In this embodiment, a preset function is called based on the second query statement, which can be used to obtain data mining results. This method has low coding costs and is simpler and more efficient to implement.

[0080] Data mining based on preset functions can be implemented using a distributed system.

[0081] Accordingly, the preset function is deployed on multiple computing nodes;

[0082] The step of using the preset function to perform data mining processing on the original dataset based on the sorting result of the target item to obtain data mining results includes:

[0083] Using the preset functions on the multiple computing nodes, data mining processing is performed on the original dataset in parallel according to the sorting results of the target items to obtain data mining results.

[0084] In this embodiment, data mining processing is performed in parallel by multiple computing nodes, which can improve the computing speed and data mining efficiency.

[0085] Further, the plurality of computing nodes include: a merging node and a plurality of adding nodes; the preset functions include: an adding function, a merging function, and a transformation function, wherein the adding function is deployed on each adding node, and the merging function and the transformation function are deployed on the merging node; the step of using the preset functions on the plurality of computing nodes to perform data mining processing on the original dataset in parallel according to the sorting result of the target item to obtain data mining results includes: obtaining a plurality of local datasets according to the sorting result of the target item and the original dataset, and assigning the plurality of local datasets to the plurality of adding nodes respectively; using the adding function deployed on each adding node in parallel to construct a local frequent pattern tree corresponding to each adding node based on the local dataset; using the merging function deployed on the merging node to merge the local frequent pattern trees corresponding to each adding node to obtain a final frequent pattern tree; and using the transformation function deployed on the merging node to obtain the data mining results based on the final frequent pattern tree.

[0086] In this embodiment, multiple local Frequent Pattern Trees (FP trees) can be obtained in parallel by adding functions on the adding nodes; multiple local FP trees can be merged to obtain the final FP tree by merging functions on the merging nodes; and data mining results can be obtained based on the final FP tree according to the transformation functions on the merging nodes. Since the above-mentioned adding, merging, and transformation functions are all preset functions, users do not need to program them themselves, which makes it easy for users to use, simplifies user operations, reduces coding overhead, and makes it easy to obtain data mining results.

[0087] This embodiment uses UDF as an example of a preset function.

[0088] UDFs are code logic written by the provider that can accept one or more parameters to implement certain specific functions that the original computing engine does not have. They are mainly divided into three categories:

[0089] *UDF (User Defined Function): Single-line input, single-line output; User-defined function.

[0090] *UDAF (User Defined Aggregate Function): Inputs multiple rows and outputs a single row, used for scenarios involving multi-row data aggregation;

[0091] *UDTF (User Defined Table generating Function): Takes one line as input and outputs multiple lines, used for data splitting scenarios.

[0092] This embodiment uses a custom UDAF approach to implement the FP-Growth calculation logic.

[0093] For this UDAF, such as Figure 5 As shown, multiple computing nodes include: multiple added nodes (such as...) Figure 5 The compute nodes shown on the upper side) and merge nodes (such as Figure 5 (The computation node shown on the lower side).

[0094] Multiple added nodes can be computed in parallel. Each added node can build a local Frequent Pattern Tree (FP tree) based on the local dataset. The merging node can merge the local FP trees obtained from each added node to obtain the final FP tree, and obtain the data mining results based on the final FP tree.

[0095] like Figure 5 As shown, the input for each added node is a local dataset. For example, the local dataset for the first added node includes: data-11, data-12, data-13..., and the local dataset for the nth (n is a positive integer greater than or equal to 2, which can be set) added node includes: data-n1, data-n2, data-n3....

[0096] The local dataset for each added node is obtained based on the ranking result of the target item and the original dataset. Specifically, the original dataset can be divided into n groups. For example, if the original dataset contains 100 transactions, and n=10 and the transactions are evenly distributed (the specific grouping strategy can be set), then each group contains 10 transactions. Within each group, each transaction in that group is sorted according to the ranking result of the target item, and the sorted transactions are combined to form the corresponding local dataset.

[0097] For example, based on the original dataset shown in Table 1, assuming it is to be divided into two groups, the first group includes the first two transactions (T1 and T2), and the second group includes the last two transactions (T3 and T4), with the sorting result of each data item (target item) as [b, a, c, d, e], then the first local dataset can be obtained based on this sorting result and the first group. Specifically, for each transaction, the data items in that transaction are arranged according to the sorting result. Similarly, the second local dataset can be obtained based on this sorting result and the second group. The first and second local datasets are shown in Tables 2 and 3, respectively.

[0098] Table 2

[0099]

[0100]

[0101] Table 3

[0102] transactions Product categories T3 b, c, d, e T4 b, a, c

[0103] Combined Figure 5 Assuming the first local dataset for adding a node is the first local dataset shown in Table 2, then data-11 = {b, a}, and data-12 = {b, a, c}. The rest are similar.

[0104] FP-growth is an association rule mining algorithm that can discover frequent itemsets in a dataset and the association rules between frequent itemsets.

[0105] The FP-growth algorithm mainly includes: constructing the FP tree and mining frequent itemsets and their association rules from the FP tree.

[0106] Each added node can use its added function to process the local dataset to build an FP-tree. The FP-tree obtained by adding nodes can be called a local FP-tree.

[0107] The provider can write the addition function based on the FP-tree construction logic of the FP-growth algorithm. The specific content can be written by the provider itself or obtained from existing functions. After the provider writes the addition function or obtains it from existing functions, it can encapsulate it and then deploy it to each addition node through the user's registration.

[0108] UDAF provides a data structure called AggregationBuffer, in which the local FP-tree built by each added node can be stored.

[0109] Merge nodes and use the merge function on them to merge the local FP trees obtained by each added node to obtain the final FP tree; and use the transformation function on them to perform data mining processing on the final FP tree to obtain data mining results. The processing logic can adopt the content of obtaining data mining results based on FP trees in the FP-growth algorithm.

[0110] The specific content of the merge and transformation functions can be written by the provider or obtained from existing functions. After the provider writes the merge and transformation functions or obtains them from existing functions, they can be encapsulated and then deployed to the merge node through the user's registration.

[0111] like Figure 5 As shown, a merging node can also have an AggregationBuffer, which can store the final FP-tree in the AggregationBuffer on the merging node.

[0112] like Figure 5 As shown, the transformation function for obtaining data mining results based on the final FP-tree is represented by FPGrowthModel. Its input information includes the FP-tree (specifically, the final FP-tree), and may also include other parameters such as minimum support, minimum confidence, and number of rules. These parameters can be specified by the user, such as those included in the input parameters of a predefined function call. FPGrowthModel integrates functions for generating and displaying data mining results. Figure 5 In this dataset, the function for generating data mining results is represented by `generateAssociationRules()`, and the function for displaying results is represented by `FormOutput()`. Based on the function for generating data mining results, the data mining results can be obtained based on the input information mentioned above. The data mining results can include frequency itemsets and their association rules in the original dataset. Based on the function for displaying results, the data mining results can be displayed in a specified format (such as JSON format).

[0113] Regarding the UDAF provided by the provider, for ease of explanation, the provider has named this custom UDAF fp_growth. The main steps are as follows:

[0114] 1) Define the parameter validation for fp_growth UDAF, including the parameter quantity, parameter type, and value range;

[0115] 2) Define the FP-Tree and FPGrowthModel data structures, or you can directly reference some of the pre-encapsulated data structures in the computing framework;

[0116] 3) Define an AggregationBuffer for the built-in FP-Tree. Each computing node performs parallel distributed computation to fill the transaction dataset into the FP-Tree and merge them in pairs.

[0117] 4) The final AggregationBuffer is calculated in the terminate method and converted into FPGrowthModel;

[0118] 5) Calculate the association rule results by combining parameters such as minimum support, minimum confidence, and limit on the number of association rules in the UDF, including frequent itemsets before and after order, corresponding support, confidence, lift, etc.

[0119] Therefore, the input parameters of the UDF (specifically, UDAF) provided by the provider include: the ranking result of the target items, and may also include minimum support, minimum confidence, and the number of association rules returned by the UDF calculation. The data mining results returned by the UDF may include: frequent itemsets, association rules between frequent itemsets, and may also include the support of frequent itemsets, the confidence of association rules, lift, etc.

[0120] In conjunction with the above application scenarios, this disclosure also provides a data mining method.

[0121] Figure 6 Based on the schematic diagram of the second embodiment of this disclosure, this embodiment provides a data mining method. Taking SQL statements as the query statement and UDFs as the preset function as an example, the method includes:

[0122] 601. Use the first SQL statement to obtain the sorting result of the target item in the original dataset.

[0123] Among them, combined Figure 4 The first SQL statement can specifically include: grouping clause and counting clause (GroupBy+count), window clause (row_number), first concatenation clause (concat_ws), sorting clause (sort_array) and transformation clause (collect_set), second concatenation clause (concat_ws), matching clause (regexp_replace), and array clause (split). Using the above SQL clauses, the sorting result of the target item can be obtained, such as... Figure 4 The values ​​shown are [C,E,B,A,D].

[0124] 602. Use the second SQL statement to call the preset UDF.

[0125] The second SQL statement can be a selection clause, which can then invoke a User-Defined Function (UDF). This UDF is provided by the provider; it can be integrated into the computing engine itself, or it can be registered into the computing engine by the user.

[0126] 603. Using the preset UDF on multiple computing nodes, data mining processing is performed on the original dataset in parallel according to the sorting result of the target item to obtain data mining results.

[0127] This data mining method can be implemented by a distributed system, that is, the pre-defined UDF can be deployed on multiple computing nodes, so that multiple computing nodes can perform parallel computing to obtain data mining results.

[0128] In this embodiment, the implementation is based on SQL statements and preset UDFs. For the user, using SQL statements and calling the preset UDFs to obtain data mining results eliminates the need for coding in programming languages ​​such as Python, making the implementation simpler and more efficient. Furthermore, parallel execution across multiple computing nodes can improve data mining efficiency.

[0129] Figure 7 This is a schematic diagram based on the third embodiment of the present disclosure. This embodiment provides a data mining device 700, which includes a sorting module 701 and a mining module 702.

[0130] The sorting module 701 is used to obtain the sorting result of the target items in the original dataset using a first query statement; the mining module 702 is used to call a preset function using a second query statement, and use the preset function to perform data mining processing on the original dataset according to the sorting result of the target items, so as to obtain data mining results.

[0131] In this embodiment, the sorting result of the target item is obtained through the first query statement, and the preset function is called through the second query statement. The preset function is used to obtain the data mining result based on the sorting result. For the user, the user only needs to use the query statement and call the already implemented preset function to perform data mining processing. There is no need to use programming languages ​​such as Python. Therefore, data mining processing can be performed easily without the user needing to master programming languages ​​such as Python.

[0132] In some embodiments, the sorting module 701 is further configured to: use the first query statement to obtain the occurrence count of the target item in the dataset; use the first query statement to sort the target item based on the occurrence count to obtain the sorting result of the target item.

[0133] In this embodiment, the ranking result of the target item is obtained based on the occurrence frequency of the target item, which can easily and efficiently obtain the ranking result of the target item, thereby improving the efficiency of data mining.

[0134] In some embodiments, the first query statement includes a grouping clause and a counting clause; the sorting module 701 is further configured to: use the grouping clause to divide the same target item in the dataset into the same group; and use the counting clause to count the target items in each group to obtain the number of times the target item appears.

[0135] In this embodiment, the number of occurrences of the target item can be easily obtained using SQL clauses through grouping and counting clauses, providing a data foundation for subsequent processing.

[0136] In some embodiments, the first query statement further includes: a window clause, a first concatenation clause, a sorting clause, a transformation clause, a second concatenation clause, a matching clause, and an array clause; the sorting module 701 is further configured to: use the window clause to assign an index to the target item based on the occurrence count, and construct an item header table based on the index and occurrence count of the target item, wherein each row element of the item header table includes the index and occurrence count of each target item; use the first concatenation clause to concatenate the index and occurrence count of each row element of the item header table using a separator; use the sorting clause and the transformation clause to sort each row element based on the index contained in each row element; use the second concatenation clause and the matching clause to concatenate each sorted row element, and remove the index and the separator from each concatenated row element to obtain a string composed of sorted target items; use the array clause to convert the string into an array, and use the array as the sorting result of the target items.

[0137] In this embodiment, the sorting result of the target item can be obtained based on the number of times the target item appears through the above clause in the SQL statement, so as to obtain the sorting result of the target item based on the SQL statement without the need for the user to use programming languages ​​such as Python, which is more convenient.

[0138] In some embodiments, the second query statement includes a selection clause; the mining module 702 is further configured to: use the selection clause to call the preset function.

[0139] In this embodiment, a preset function is called based on the second query statement, which can be used to obtain data mining results. This method has low coding costs and is simpler and more efficient to implement.

[0140] In some embodiments, the preset function is deployed on multiple computing nodes; the mining module 702 is further configured to: use the preset function on the multiple computing nodes to perform data mining processing on the original dataset in parallel according to the sorting result of the target item, so as to obtain data mining results.

[0141] In this embodiment, data mining processing is performed in parallel by multiple computing nodes, which can improve the computing speed and data mining efficiency.

[0142] In some embodiments, the plurality of computing nodes include: a merging node and a plurality of adding nodes; the preset function includes: an adding function, a merging function, and a transformation function, wherein the adding function is deployed on each adding node, and the merging function and the transformation function are deployed on the merging node; the mining module 702 is further configured to: obtain a plurality of local datasets based on the sorting result of the target item and the original dataset, and allocate the plurality of local datasets to the plurality of adding nodes respectively; use the adding function deployed on each adding node in parallel to construct a local frequent pattern tree corresponding to each adding node based on the local dataset; use the merging function deployed on the merging node to merge the local frequent pattern trees corresponding to each adding node to obtain a final frequent pattern tree; and use the transformation function deployed on the merging node to obtain the data mining result based on the final frequent pattern tree.

[0143] In this embodiment, multiple local Frequent Pattern Trees (FP trees) can be obtained in parallel by adding functions on the adding nodes; multiple local FP trees can be merged to obtain the final FP tree by merging functions on the merging nodes; and data mining results can be obtained based on the final FP tree according to the transformation functions on the merging nodes. Since the above-mentioned adding, merging, and transformation functions are all preset functions, users do not need to program them themselves, which makes it easy for users to use, simplifies user operations, reduces coding overhead, and makes it easy to obtain data mining results.

[0144] It is understood that the same or similar content in different embodiments of this disclosure can be referred to each other.

[0145] It is understood that the terms "first" and "second" in the embodiments of this disclosure are only used for distinction and do not indicate the degree of importance or the order of events.

[0146] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0147] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0148] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. Electronic device 800 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, servers, blade servers, mainframe computers, and other suitable computers. Electronic device 800 may also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0149] like Figure 8 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. The RAM 803 may also store various programs and data required for the operation of the electronic device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0150] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0151] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as data mining methods. For example, in some embodiments, the data mining method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the data mining method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform data mining methods by any other suitable means (e.g., by means of firmware).

[0152] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0153] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable load balancing device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on the machine, partially on the machine, as a standalone software package partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0154] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0155] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0156] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0157] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0158] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0159] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data mining method, comprising: The first query statement is used to obtain the sorting results of the target items in the original dataset; The second query statement is used to call a preset function, and the preset function is used to perform data mining processing on the original dataset according to the sorting result of the target item to obtain data mining results; The first query statement includes: a window clause, a first concatenation clause, a sort clause, a transformation clause, a second concatenation clause, a matching clause, and an array clause; The sorting result was obtained in the following way: Using the aforementioned windowing clause, an index is assigned to the target item based on the frequency of its occurrence, and an item header table is constructed based on the index of the target item and the target item itself. Each row of the item header table includes the index of each target item and the target item itself. Using the first concatenation clause, for each element in the item header table, the sequence number and the target item are concatenated using a delimiter; Using the sorting clause and the transformation clause, the elements in each row are sorted based on the sequence number contained in each element; Using the second concatenation clause and the matching clause, each row of sorted elements is concatenated, and the sequence number and the delimiter are removed from each row of concatenated elements to obtain a string composed of sorted target items; The string is converted into an array using the array clause, and the array is used as the sorting result of the target item.

2. The method according to claim 1, wherein, The first query statement also includes: a grouping clause and a counting clause; The number of occurrences was obtained in the following way: The grouping clause is used to group the same target item in the dataset into the same group; The counting clause is used to count the number of times the target item appears in each group.

3. The method according to any one of claims 1-2, wherein, The second query statement includes: a select clause; The step of using a second query statement to call a preset function includes: The preset function is invoked using the selection clause.

4. The method according to any one of claims 1-2, wherein, The preset function is deployed on multiple computing nodes; The step of using the preset function to perform data mining processing on the original dataset based on the sorting result of the target item to obtain data mining results includes: Using the preset functions on the multiple computing nodes, data mining processing is performed on the original dataset in parallel according to the sorting results of the target items to obtain data mining results.

5. The method according to claim 4, wherein, The plurality of computing nodes includes: a merging node and a plurality of adding nodes; The preset functions include: an add function, a merge function, and a transform function. The add function is deployed on each add node, and the merge function and the transform function are deployed on the merge node. The process involves using the preset functions on the multiple computing nodes to perform data mining processing on the original dataset in parallel based on the sorting results of the target items, in order to obtain data mining results, including: Based on the sorting results of the target items and the original dataset, multiple local datasets are obtained, and the multiple local datasets are respectively assigned to the multiple added nodes; The addition function deployed on each addition node is used in parallel to construct a local frequent pattern tree corresponding to each addition node based on the local dataset; The merging function deployed on the merging node is used to merge the local frequent pattern trees corresponding to each added node to obtain the final frequent pattern tree. The data mining results are obtained based on the final frequent pattern tree by using the transformation function deployed on the merging node.

6. A data mining apparatus, comprising: The sorting module is used to obtain the sorting results of the target items in the original dataset using the first query statement; The mining module is used to call a preset function using a second query statement, and to perform data mining processing on the original dataset based on the sorting result of the target item using the preset function to obtain data mining results; The first query statement includes: a window clause, a first concatenation clause, a sort clause, a transformation clause, a second concatenation clause, a matching clause, and an array clause; The sorting result was obtained in the following way: Using the aforementioned windowing clause, an index is assigned to the target item based on the frequency of its occurrence, and an item header table is constructed based on the index of the target item and the target item itself. Each row of the item header table includes the index of each target item and the target item itself. Using the first concatenation clause, for each element in the item header table, the sequence number and the target item are concatenated using a delimiter; Using the sorting clause and the transformation clause, the elements in each row are sorted based on the sequence number contained in each element; Using the second concatenation clause and the matching clause, each row of sorted elements is concatenated, and the sequence number and the delimiter are removed from each row of concatenated elements to obtain a string composed of sorted target items; The string is converted into an array using the array clause, and the array is used as the sorting result of the target item.

7. The apparatus according to claim 6, wherein, The first query statement also includes: a grouping clause and a counting clause; The sorting module is further used for: The grouping clause is used to group the same target item in the dataset into the same group; The counting clause is used to count the number of times the target item appears in each group.

8. The apparatus according to any one of claims 6-7, wherein, The second query statement includes: a select clause; The mining module is further used for: The preset function is invoked using the selection clause.

9. The apparatus according to any one of claims 6-7, wherein, The preset function is deployed on multiple computing nodes; The mining module is further used for: Using the preset functions on the multiple computing nodes, data mining processing is performed on the original dataset in parallel according to the sorting results of the target items to obtain data mining results.

10. The apparatus according to claim 9, wherein, The plurality of computing nodes includes: a merging node and a plurality of adding nodes; The preset functions include: an add function, a merge function, and a transform function. The add function is deployed on each add node, and the merge function and the transform function are deployed on the merge node. The mining module is further used for: Based on the sorting results of the target items and the original dataset, multiple local datasets are obtained, and the multiple local datasets are respectively assigned to the multiple added nodes; The addition function deployed on each addition node is used in parallel to construct a local frequent pattern tree corresponding to each addition node based on the local dataset; The merging function deployed on the merging node is used to merge the local frequent pattern trees corresponding to each added node to obtain the final frequent pattern tree. The data mining results are obtained based on the final frequent pattern tree by using the transformation function deployed on the merging node.

11. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.

12. A non-transitory computer readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

13. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

Images