A fuzzy decision tree construction method, device and equipment and computer storage medium

Abstract

The application discloses a fuzzy decision tree construction method and device, equipment and a computer storage medium. In the case that a construction data set is acquired, a first mapping reduction model is used to perform mapping reduction processing on data in the construction data set, a plurality of fuzzy partitions in the construction data set are obtained, a second mapping reduction model is used to perform mapping reduction processing on data of each fuzzy partition in the plurality of fuzzy partitions, a plurality of fuzzy sub-partitions of each fuzzy partition are obtained, a root node set and a sub-node set are determined according to the obtained fuzzy partitions and fuzzy sub-partitions, and finally, a fuzzy decision tree of the above construction data set is constructed based on the root node set and the sub-node set. In this way, through twice mapping reduction processing, a fuzzy decision tree classifier can be distributed constructed from big data based on distributed parallel computing, data classification processing of a data set with high dimension and large data quantity is realized, and the efficiency of inductive classification of related fuzzy data is improved.

Description

Technical Field

[0001] This application belongs to the field of classification algorithms, and in particular relates to a method, apparatus, device and computer storage medium for constructing fuzzy decision trees. Background Technology

[0002] In the context of big data, there is a need to classify and summarize data with high dimensionality, large volume, and fuzzy relationships. Traditional rule-centered decision tree classification methods require domain experts to rely on their experience and knowledge to construct decision trees by providing fuzzy membership functions for each attribute.

[0003] Traditional decision tree inductive classification methods, when dealing with datasets with high dimensionality and large data volume, may involve millions of calculations of fuzzy entropy increases and the merging of statistical information during the process of selecting fuzzy partitions and constructing decision trees. This results in low efficiency in inductive classification of data with fuzzy correlations. Summary of the Invention

[0004] This application provides a method, apparatus, device, and computer storage medium for constructing fuzzy decision trees, which can improve the efficiency of constructing fuzzy decision trees.

[0005] In a first aspect, embodiments of this application provide a fuzzy decision tree construction method, which may include:

[0006] Given the constructed dataset, the data in the constructed dataset is mapped and reduced using the first mapping and reduction model to obtain multiple fuzzy partitions in the constructed dataset, each of which includes multiple data points in the constructed dataset.

[0007] The data of each fuzzy partition in multiple fuzzy partitions are mapped and reduced by the second mapping and reduction model to obtain multiple fuzzy sub-partitions of each fuzzy partition, and each fuzzy sub-partition includes at least one piece of data in the fuzzy partition.

[0008] Each fuzzy partition is taken as the root node to obtain a root node set including multiple root nodes; and each fuzzy sub-partition of the fuzzy partition is taken as the child node to obtain a child node set corresponding to the root node of the fuzzy partition, the child node set including multiple child nodes.

[0009] Based on the root node set and child node set, construct a fuzzy decision tree for the dataset.

[0010] In one embodiment, after the step of taking the fuzzy sub-partitions of each fuzzy partition as child nodes to obtain the set of child nodes of the root node corresponding to the fuzzy partition, the method may further include:

[0011] Split detection is performed on the child nodes in the child node set to obtain the detection results. The detection results are used to indicate whether there are splittable child nodes in the child node set.

[0012] If the detection results indicate that there are splittable child nodes in the child node set, move the splittable child nodes to the set of nodes to be split.

[0013] The fuzzy sub-partitions corresponding to each node in the set of nodes to be split are used as fuzzy partitions and input into the second mapping reduction model to obtain multiple fuzzy sub-partitions output by the second mapping reduction model.

[0014] The multiple fuzzy sub-partitions output by the second mapping reduction model are added as child nodes to the child node set, and the step of re-performing the split detection of the child nodes in the child node set is performed until the detection result indicates that there are no splittable child nodes in the child node set. Then, the step of constructing a fuzzy decision tree for the dataset based on the root node set and the child node set is performed.

[0015] In one embodiment, the step described above, which involves performing mapping and reduction processing on the data in the construction dataset using a first mapping and reduction model to obtain multiple fuzzy partitions in the construction dataset after obtaining the construction dataset, may include:

[0016] Given the constructed dataset, the fuzzy entropy information corresponding to each attribute is calculated based on multiple attributes of the data in the constructed dataset using the first mapping reduction model.

[0017] Based on the calculated fuzzy entropy information corresponding to each attribute, the fuzzy partition corresponding to each attribute is determined.

[0018] In one embodiment, the step of performing mapping and reduction processing on the data of each fuzzy partition in multiple fuzzy partitions using a second mapping and reduction model to obtain multiple fuzzy sub-partitions of each fuzzy partition may include:

[0019] The second mapping reduction model is used to segment the data of each fuzzy partition in multiple fuzzy partitions based on multiple attributes of the data in each fuzzy partition, thereby obtaining multiple fuzzy sub-partitions of each fuzzy partition.

[0020] Secondly, embodiments of this application provide a fuzzy decision tree construction apparatus, which may include:

[0021] The first mapping and reduction module is used to perform mapping and reduction processing on the data in the construction dataset through the first mapping and reduction model when the construction dataset is obtained, so as to obtain multiple fuzzy partitions in the construction dataset, each fuzzy partition including multiple data in the construction dataset.

[0022] The second mapping and reduction module is used to perform mapping and reduction processing on the data of each fuzzy partition in multiple fuzzy partitions through the second mapping and reduction model, so as to obtain multiple fuzzy sub-partitions of each fuzzy partition, and each fuzzy sub-partition includes at least one piece of data in the fuzzy partition.

[0023] The determination module is used to take each fuzzy partition as the root node to obtain a root node set including multiple root nodes; and to take the fuzzy sub-partitions of each fuzzy partition as child nodes to obtain a child node set of the root node corresponding to the fuzzy partition, wherein the child node set includes multiple child nodes.

[0024] The building module is used to construct a fuzzy decision tree for the dataset based on the root node set and the child node set.

[0025] Thirdly, embodiments of this application provide an electronic device, the device comprising:

[0026] processor;

[0027] Memory used to store processor-executable instructions;

[0028] The processor is configured to execute instructions to implement the fuzzy decision tree construction method as shown in any embodiment of the first aspect.

[0029] Fourthly, embodiments of this application provide a computer storage medium on which a computer program is stored, which, when executed by a processor, implements the fuzzy decision tree construction method as shown in any embodiment of the first aspect.

[0030] Fifthly, embodiments of this application also provide a computer program product comprising a computer program stored in a readable storage medium, wherein at least one processor of the device reads from the storage medium and executes the computer program, causing the device to perform the fuzzy decision tree construction method shown in any embodiment of the first aspect.

[0031] This application provides a fuzzy decision tree construction method, apparatus, device, and computer storage medium. Compared with the prior art, this application has the following advantages:

[0032] This application discloses a fuzzy decision tree construction method, apparatus, device, and computer storage medium. Upon obtaining a construction dataset, a first mapping and reduction model is used to perform mapping and reduction processing on the data in the construction dataset to obtain multiple fuzzy partitions. Then, a second mapping and reduction model is used to perform mapping and reduction processing on the data in each of the multiple fuzzy partitions to obtain multiple fuzzy sub-partitions for each fuzzy partition. Based on the obtained fuzzy partitions and fuzzy sub-partitions, a root node set and a child node set are determined. Finally, based on the root node set and the child node set, a fuzzy decision tree for the construction dataset is constructed.

[0033] By performing two mapping and reduction processes on the data in the constructed dataset through the first mapping and reduction model and the second mapping and reduction model respectively, a fuzzy decision tree classifier can be constructed in a distributed manner from big data based on distributed parallel computing. This enables data classification processing for datasets with high dimensionality and large data volume, thereby improving the efficiency of summarizing and classifying data with fuzzy correlation. Attached Figure Description

[0034] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a flowchart illustrating a fuzzy decision tree construction method provided in an embodiment of this application;

[0036] Figure 2 This is a flowchart illustrating another fuzzy decision tree construction method provided in an embodiment of this application;

[0037] Figure 3 This is a flowchart illustrating another fuzzy decision tree construction method provided in an embodiment of this application;

[0038] Figure 4 This is a flowchart illustrating another fuzzy decision tree construction method provided in an embodiment of this application;

[0039] Figure 5 This is a schematic diagram illustrating the construction of a fuzzy decision tree according to an embodiment of this application;

[0040] Figure 6 This is a schematic diagram of the structure of a fuzzy decision tree construction device provided in an embodiment of this application;

[0041] Figure 7 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] The features and exemplary embodiments of various aspects of this application will be described in detail below. To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only intended to explain this application and not to limit it. For those skilled in the art, this application can be implemented without some of these specific details. The following description of the embodiments is merely to provide a better understanding of this application by illustrating examples.

[0043] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0044] As can be seen from the background section, for a large amount of data with low value density, ambiguous correlation, and imprecise data, the traditional rule-centered decision tree inductive classification method requires domain experts to rely on experience knowledge to give fuzzy set membership functions for each attribute, which cannot meet the requirements of the system to automatically extract information from datasets with high dimensionality and large data volume.

[0045] In the context of big data, the demand for processing large amounts of fuzzy and imprecise data is growing, and fuzzy classification models play an important role due to their inherent interpretability and ability to automatically extract information. However, currently, only a few works on distributed implementations of classification algorithms integrate fuzzy theory.

[0046] On the other hand, for datasets with high dimensionality and large data volume, the calculation of fuzzy entropy increase and the merging of statistical information for selecting fuzzy partitions and constructing decision trees can reach millions. Therefore, traditional single-machine implementation methods of fuzzy decision trees are unsuitable for managing big data due to their inability to meet the huge time and space overhead.

[0047] Fuzzy pattern tree (FPT) is a machine learning classification method similar to a binary decision diagram (BDD). Its internal nodes are labeled using generalized (fuzzy) logic and arithmetic operators, and its leaf nodes are associated with (unary) fuzzy predicates on a given set of input attributes.

[0048] The MapReduce programming model allows for the processing of large datasets by automatically parallelizing computation across a set of machines. Based on functional programming, it primarily implements the application of a map function to a set of input data, returning a...<key,Value> The key-value pair structure serves as an intermediate dataset, and data with the same key is input into a reduce function for execution, ultimately returning the processed result. This allows the computational model to be parallelized to process large datasets. The execution processes of map and reduce can run on multiple machines without interdependence or coordination between them.

[0049] The MapReduce execution flow is as follows: First, the MapReduce library in the user program splits the input file into M blocks, the size of each block being controlled by user-selectable parameters. Then, multiple replicas of the program are started on several machines in a cluster. The cluster has a master node responsible for distributing Map and Reduce tasks to idle worker nodes, with each node handling only one task. The worker nodes executing the Map tasks read the corresponding input block content and parse it...<key,Value> Key-value pairs are passed to the user-defined map function for business logic processing. The resulting intermediate key-value pairs are cached in memory and periodically divided into R blocks by a partitioning function and written to the local disk. The master node is responsible for passing the location of these cached pairs on the local disk to the reduce worker nodes and notifying the worker nodes executing the reduce tasks after all map tasks have been completed. The worker nodes executing the reduce tasks use RPC calls to read the cached data from the local disk of the map worker nodes, merge and sort this data according to the keys of the intermediate key-value pairs, and pass each key and its associated list of values ​​to the user's reduce function. The output of the reduce function is written to the end of the final output file created by the reduce partitioning process.

[0050] Fuzzy decision tree classification can be applied to the field of big data and is suitable for most scenarios that require classification of big data, such as the State Grid's detection of abnormal electricity consumption, personal credit assessment of telecom operators' data, and product classification in e-commerce scenarios. It is understood that the dataset constructed in this embodiment can also include electricity consumption data, personal credit data, e-commerce product data, etc.

[0051] To address the problems existing in the prior art, embodiments of this application provide a fuzzy decision tree construction method, apparatus, device, and computer storage medium. Upon obtaining a construction dataset, a first mapping and reduction model is used to perform mapping and reduction processing on the data in the construction dataset to obtain multiple fuzzy partitions. Then, a second mapping and reduction model is used to perform mapping and reduction processing on the data in each of the multiple fuzzy partitions to obtain multiple fuzzy sub-partitions for each fuzzy partition. Based on the obtained fuzzy partitions and fuzzy sub-partitions, a root node set and a child node set are determined. Finally, based on the root node set and the child node set, a fuzzy decision tree for the construction dataset is constructed.

[0052] By performing two mapping and reduction processes on the data in the constructed dataset through the first mapping and reduction model and the second mapping and reduction model respectively, a fuzzy decision tree classifier can be constructed in a distributed manner from big data based on distributed parallel computing. This enables data classification processing for datasets with high dimensionality and large data volume, thereby improving the efficiency of summarizing and classifying data with fuzzy correlation.

[0053] The following section first introduces the fuzzy decision tree construction method provided in the embodiments of this application. For example... Figure 1 As shown in the embodiments of this application, the fuzzy decision tree construction method includes the following steps:

[0054] S101: Given the constructed dataset, the data in the constructed dataset is mapped and reduced using the first mapping and reduction model to obtain multiple fuzzy partitions in the constructed dataset, each fuzzy partition including multiple data in the constructed dataset;

[0055] S102: The data of each fuzzy partition in multiple fuzzy partitions are mapped and reduced by the second mapping and reduction model to obtain multiple fuzzy sub-partitions of each fuzzy partition, and each fuzzy sub-partition includes at least one piece of data in the fuzzy partition.

[0056] S103: Take each fuzzy partition as the root node to obtain a root node set including multiple root nodes; and take the fuzzy sub-partitions of each fuzzy partition as child nodes to obtain a child node set corresponding to the root node of the fuzzy partition, the child node set including multiple child nodes.

[0057] S104: Construct a fuzzy decision tree for the dataset based on the root node set and child node set.

[0058] The above describes a fuzzy decision tree construction method provided in this application. Given a construction dataset, a first mapping reduction model is used to perform mapping reduction processing on the data in the construction dataset to obtain multiple fuzzy partitions. Then, a second mapping reduction model is used to perform mapping reduction processing on the data in each of the multiple fuzzy partitions to obtain multiple fuzzy sub-partitions for each fuzzy partition. Based on the obtained fuzzy partitions and fuzzy sub-partitions, a root node set and a child node set are determined. Finally, based on the root node set and the child node set, a fuzzy decision tree for the construction dataset is constructed.

[0059] By performing two mapping and reduction processes on the data in the constructed dataset through the first mapping and reduction model and the second mapping and reduction model respectively, a fuzzy decision tree classifier can be constructed in a distributed manner from big data based on distributed parallel computing. This enables data classification processing for datasets with high dimensionality and large data volume, thereby improving the efficiency of summarizing and classifying data with fuzzy correlation.

[0060] In S101, given the constructed dataset, the data in the constructed dataset is mapped and reduced using a first mapping and reduction model to obtain multiple fuzzy partitions of the constructed dataset. Each fuzzy partition includes multiple data points from the constructed dataset. In one example, the mapping and reduction model includes the mapping and reduction MapReduce programming model.

[0061] In one example, the constructed dataset is divided into multiple data blocks, which serve as input to the first mapping reduction model. By calculating and minimizing the weighted fuzzy entropy, the optimal candidate fuzzy partition is selected, resulting in multiple fuzzy partitions corresponding to multiple data blocks in the constructed dataset.

[0062] In S102, the data of each fuzzy partition in multiple fuzzy partitions are mapped and reduced using the second mapping reduction model to obtain multiple fuzzy sub-partitions for each fuzzy partition. Each fuzzy sub-partition includes at least one piece of data from the fuzzy partition. In one example, the data of each fuzzy partition in the constructed dataset calculated in S101 is used as the input to the second mapping reduction model. By calculating the fuzzy information gain of the attributes corresponding to the data, each fuzzy partition is further divided to obtain the fuzzy sub-partitions corresponding to each fuzzy partition.

[0063] In S103, each fuzzy partition is taken as the root node, resulting in a root node set containing multiple root nodes; and each fuzzy sub-partition of the fuzzy partition is taken as the child node, resulting in a child node set corresponding to the root node of the fuzzy partition, where each child node set contains multiple child nodes. It can be understood that the node set is the set of data to be partitioned, and the initial set is the entire dataset to be partitioned (i.e., the data in the constructed dataset).

[0064] In S104, a fuzzy decision tree for the dataset is constructed based on the root node set and child node set. In one example, the fuzzy partition obtained through the first mapping reduction model is a fuzzy partition with five triangular fuzzy sets. Starting from the initial set to be partitioned, each partition generates two child nodes, and the instances in the two subsets belong to two disjoint groups divided by the parent node's fuzzy set. The fuzzy decision tree is then constructed using the parent and child nodes.

[0065] To improve the efficiency of constructing fuzzy decision trees, such as Figure 2 As shown, after S103, it may also include:

[0066] S201: Perform split detection on the child nodes in the child node set and obtain the detection result. The detection result is used to indicate whether there are splittable child nodes in the child node set.

[0067] S202: If the detection result indicates that there are splittable child nodes in the child node set, move the splittable child nodes to the set of nodes to be split;

[0068] S203: Input the fuzzy sub-partitions corresponding to each node in the set of nodes to be split as fuzzy partitions into the second mapping reduction model to obtain multiple fuzzy sub-partitions output by the second mapping reduction model;

[0069] S204: Add the multiple fuzzy sub-partitions output by the second mapping reduction model as child nodes to the child node set, and re-execute the step of split detection on the child nodes in the child node set to obtain the detection results, until the detection results indicate that there are no splittable child nodes in the child node set, and execute the step of constructing a fuzzy decision tree for the dataset based on the root node set and the child node set.

[0070] Thus, by performing split detection on the child nodes in the child node set to determine whether there are any splittable child nodes, and then iteratively applying a second mapping reduction model to the splittable child nodes until no splittable child nodes remain in the child node set, a fuzzy decision tree for the dataset is constructed based on the root node set and the child node set. This method improves the efficiency of constructing the fuzzy decision tree by splitting nodes using a mapping reduction model.

[0071] In S201, split detection is performed on the child nodes in the child node set to obtain the detection results. The detection results are used to indicate whether there are splittable child nodes in the child node set. In one example, the conditions for determining that a child node is not splittable are: the depth of the decision tree reaches a first threshold, and the number of instances contained in the node or the fuzzy information entropy gain reaches a second threshold.

[0072] In S202, if the detection result indicates that there are splittable child nodes in the child node set, the splittable child nodes are moved to the node set to be split.

[0073] In S203, the fuzzy sub-partitions corresponding to each node in the set of nodes to be split are used as fuzzy partition inputs to the second mapping reduction model, resulting in multiple fuzzy sub-partitions output by the second mapping reduction model. In one example, the fuzzy sub-partitions corresponding to each node in the set of nodes to be split are used as inputs to the second mapping reduction model, and cyclic mapping reduction processing is performed to obtain fuzzy sub-partitions for further subdivision.

[0074] In S204, multiple fuzzy sub-partitions output by the second mapping reduction model are added as child nodes to the child node set, and the step of re-performing the split detection on the child nodes in the child node set is repeated until the detection result indicates that there are no splittable child nodes in the child node set. Then, the step of constructing a fuzzy decision tree for the dataset based on the root node set and the child node set is executed. In one example, multiple fuzzy sub-partitions obtained in S203 are added as new child nodes, along with the splittable child nodes moved to the splittable node set in S202, to the child node set. This continues until no splittable child nodes are found in the aforementioned child node set, at which point the fuzzy decision tree for the dataset is constructed based on the root node set and the child node set. In another example, when the splittable node set is empty, the loop stops, and the step of constructing a fuzzy decision tree for the dataset based on the root node set and the child node set is executed.

[0075] To improve the efficiency of constructing fuzzy decision trees based on distributed computing, such as... Figure 3 As shown, S101 may include:

[0076] S1011: Given the constructed dataset, the fuzzy entropy information corresponding to each attribute is calculated based on multiple attributes of the data in the constructed dataset using the first mapping reduction model.

[0077] S1012: Determine the fuzzy partition corresponding to each attribute based on the calculated fuzzy entropy information of each attribute.

[0078] Therefore, using the first mapping reduction model, based on multiple attributes of the data in the constructed dataset, the fuzzy entropy information corresponding to each attribute is calculated. Then, based on the calculated fuzzy entropy information corresponding to each attribute, the fuzzy partition corresponding to each attribute is determined. This enables the construction of fuzzy decision trees based on distributed computing, improving the efficiency of fuzzy decision tree construction.

[0079] In S1011, given the constructed dataset, the first mapping reduction model calculates the fuzzy entropy information corresponding to each attribute based on multiple attributes of the data in the constructed dataset. In one example, the training set (i.e. the constructed dataset) is divided into V blocks as input. Each block provides only one Map task. Q is the number of CUs available in the cluster. Up to Q tasks can be executed in parallel, including Map and Reduce.

[0080] Let TR k =[x 1，k , ..., x N，k ](x i，k (sorted in ascending order) is the training set TR along attribute X k The projection, For its corresponding fuzzy segmentation, I k For attribute X k The range of the projection, N k yes The number of fuzzy sets in the dataset is then determined by minimizing the weighted fuzzy entropy. (i.e., fuzzy entropy information), select the optimal candidate fuzzy partition (i.e. construct multiple fuzzy partitions in the dataset).

[0081] partition The weighted fuzzy entropy is calculated as shown in formula (1):

[0082]

[0083] in, It is set S k The cardinality;

[0084] Ent k (B k，j ) represents B k，j fuzzy entropy;

[0085] It is a fuzzy set B k，j The fuzzy cardinality is calculated as follows:

[0086] N k，j It is S k，j The number of points in the middle;

[0087] It is x i，k For fuzzy set B k，j The membership degree is calculated using a triangular membership function.

[0088] Understandable, I k Fuzzy segmentation within the value range In this context, 'i' is used generically, so the above... (i.e., weighted fuzzy entropy) and (i.e., minimizing the weighted fuzzy entropy) is the same. Among them, formula (1) involves The calculation involves the integration of i.

[0089] In S1012, based on the calculated fuzzy entropy information corresponding to each attribute, the fuzzy partition corresponding to each attribute is determined. In one example, in S1011, the weighted fuzzy entropy is minimized. (i.e., fuzzy entropy information) to select the optimal candidate fuzzy partition.

[0090] To improve the efficiency of constructing fuzzy decision trees based on distributed computing, such as... Figure 4 As shown, S102 may include:

[0091] S1021: Using the second mapping reduction model, based on multiple attributes of the data in each fuzzy partition in multiple fuzzy partitions, the data in each fuzzy partition in multiple fuzzy partitions is segmented to obtain multiple fuzzy sub-partitions of each fuzzy partition.

[0092] In this way, based on the multiple attributes of the data in each fuzzy partition, the data can be divided into multiple fuzzy sub-partitions, which can be used to construct fuzzy decision trees based on distributed computing, thereby improving the efficiency of fuzzy decision tree construction.

[0093] In S1021, the second mapping reduction model is used to segment the data of each fuzzy partition based on multiple attributes of the data in each fuzzy partition, resulting in multiple fuzzy sub-partitions for each fuzzy partition. In one example, the training set is divided into V blocks as input, and the initial split node set R contains only the root node. A repetitive iterative MapReduce stage is used, where for each iteration of the node set R... h Nodes to be classified in Mode k Calculate the extended attribute X of the selected node. k The process involves splitting child nodes and adding the splittable child nodes to the node set. If the node set is not empty, the process proceeds to the next iteration; otherwise, the final fuzzy decision tree is obtained. It can be understood that the extended attributes are part of the attributes of the data in the fuzzy partition.

[0094] In each iteration, each Map-Task divides the training set into V blocks of data and sets R of nodes to be split. h As input, a statistical vector D is computed for each node PN in the node set. v，k Each node PN v In the segmentation attribute Xk Generate T k There are 1 child node, each child node has CN j Only includes those belonging to fuzzy set A k，j Instances of fuzzy set D v Characterization, calculation formula is Where N j It is the number of instances in the set. It is instance x i The membership degree of the decision tree to its parent node PN. Specifically, for the root node of the decision tree, Then, the Reduce-Task is used to integrate the statistical data from the computation of V blocks, calculate the fuzzy information gain of the candidate attributes, and select the new extended attribute X. k The dataset is then mapped to k sub-fuzzy sets to form new split child nodes. k Add the child nodes that can be further divided to the set R of nodes to be split. h+1 .

[0095] The following specific embodiment illustrates the fuzzy decision tree construction process in the above technical solution. Figure 5 This is a schematic diagram of a specific embodiment, which is described in detail below:

[0096] 1. The first MapReduce process: Traverse the sample dataset and sort it according to the continuous attribute values ​​and their corresponding class distributions to obtain the fuzzy entropy information gain and candidate fuzzy partitions for each attribute.

[0097] 1.1 Map function design: Scan each data record and transform it into...<key,Value> Output in key-value pair format. Where key is the index of the k-th attribute, and Value = List(B v，k )={B(1) v，k , ..., B(L) v，k} is a list of fuzzy boundaries of the k-th attribute extracted from the v-th block.

[0098] 1.2 Reduce function design: The input of each Reduce-Task consists of V key-value pairs output by V Map functions. <k,List(B v，k Provided, and for the k-th attribute, the Reduce function takes B v，k By merging fuzzy entropy information and sorting it in ascending order, the minimum weighted fuzzy entropy on the candidate fuzzy partitions of each attribute table is obtained. And candidate fuzzy partitions

[0099] 2. The second MapReduce process: used to build a decision tree classifier in a distributed manner. The best fuzzy partitions obtained from the first MapReduce task are distributed on the CUs, and the second MapReduce step is executed iteratively.

[0100] 2.1 Map function design: Load the v-th block of V training sets, and then, for each iteration, if the node set R... h Nodes that need to be divided k Initialization vector D v，k Scanning each data record, for each attribute of each instance in the block, the Map-Task updates all D records using equations. v，k Vector, convert it to<key,Value> Output in key-value pair format. Here, key is the index of the k-th node to be partitioned, and Value = List(D v，k ), is the cardinality of the k-th attribute extracted from the v-th block.

[0101] 2.2 Reduce Function Design: The input to each Reduce-Task consists of V key-value pairs output by V Map functions. <y,List(D v，k Provided by List(D) v，k Vector D is created by adding all V vectors in the given vectors element-wise. k Storing data from the root node to the current node (Node). k The cardinality of each attribute value. Then, the Reduce-Task computes the expanded attribute X of the selected node. k Split the child nodes and add the splittable child nodes to the node set. If the node set is not empty, proceed to the next iteration; otherwise, obtain the final fuzzy decision tree.

[0102] 2.3 Iterative Execution: Repeatedly iterate through the MapReduce phases to identify the attribute X to be selected. k and their splitting points Node y In the initial iteration, the training set is divided into V blocks as input, and the split node set R contains only the root node. A MapReduce phase with repeated iterations is used. For each iteration, the node set R... h Nodes to be classified k Calculate the extended attribute Node of the selected node. k The algorithm splits the child nodes and adds the separable child nodes to the node set to be split. If the node set is not empty, it proceeds to the next iteration. The algorithm terminates when the node set R is empty, yielding the final fuzzy decision tree. The conditions for determining whether a node is indivisible are: the depth of the decision tree, the number of instances contained in the node, or the fuzzy information gain reaching a threshold.

[0103] Based on the above embodiments, this application has the following beneficial effects:

[0104] 1. Compared with existing distributed classification algorithms, this method combines the inherent interpretability of fuzzy set theory with the hierarchical reasoning and clear tree structure of decision trees to build a data-driven fuzzy decision tree classifier for datasets with high dimensionality and large data volume. It has interpretability, high accuracy and scalability.

[0105] 2. Based on the MapReduce paradigm, it is implemented in a distributed manner and can be easily deployed in multiple cloud computing environments, such as Hadoop and Spark. It overcomes the time and space limitations of traditional single-machine fuzzy decision tree construction on datasets with high dimensionality and large data volumes through parallel computing. It employs two MapReduce stages, which is simpler and more efficient than existing four-stage implementation methods.

[0106] The above describes a specific implementation of a fuzzy decision tree construction method provided in this application. Based on the fuzzy decision tree construction method provided in the above embodiments, this application also provides a specific implementation of a fuzzy decision tree construction apparatus, as described in the following embodiments.

[0107] like Figure 6 As shown in the embodiment of this application, a fuzzy decision tree construction device 600 is provided, which includes:

[0108] The first mapping and reduction module 601 is used to perform mapping and reduction processing on the data in the construction dataset through the first mapping and reduction model when the construction dataset is obtained, so as to obtain multiple fuzzy partitions in the construction dataset, each fuzzy partition including multiple data in the construction dataset.

[0109] The second mapping and reduction module 602 is used to perform mapping and reduction processing on the data of each fuzzy partition in multiple fuzzy partitions through the second mapping and reduction model to obtain multiple fuzzy sub-partitions of each fuzzy partition, and each fuzzy sub-partition includes at least one piece of data in the fuzzy partition.

[0110] The determination module 603 is used to take each fuzzy partition as the root node to obtain a root node set including multiple root nodes; and to take the fuzzy sub-partitions of each fuzzy partition as child nodes to obtain a child node set corresponding to the root node of the fuzzy partition, wherein the child node set includes multiple child nodes.

[0111] Module 604 is used to construct a fuzzy decision tree for the dataset based on the root node set and the child node set.

[0112] In the fuzzy decision tree construction apparatus 600 provided in this application embodiment, when a construction dataset is obtained, the first mapping and reduction module 601 performs mapping and reduction processing on the data in the construction dataset using a first mapping and reduction model to obtain multiple fuzzy partitions in the construction dataset. Then, the second mapping and reduction module 602 performs mapping and reduction processing on the data in each of the multiple fuzzy partitions to obtain multiple fuzzy sub-partitions of each fuzzy partition. The determination module 603 takes each fuzzy partition as a root node to obtain a root node set including multiple root nodes; and takes the fuzzy sub-partitions of each fuzzy partition as child nodes to obtain a child node set corresponding to the root node of the fuzzy partition. Finally, the construction module 604 constructs a fuzzy decision tree for the construction dataset based on the root node set and the child node set.

[0113] It can perform two mapping and reduction processes on the data in the constructed dataset through the first mapping and reduction model and the second mapping and reduction model respectively. It can build a fuzzy decision tree classifier in a distributed manner from big data based on distributed parallel computing, realize the data classification processing of datasets with high dimensionality and large data volume, and thus improve the efficiency of summarizing and classifying data with fuzzy correlation.

[0114] As another embodiment of this application, in order to improve the efficiency of constructing fuzzy decision trees, the fuzzy decision tree construction apparatus 600 may further include:

[0115] The detection module 605 is used to perform split detection on the child nodes in the child node set and obtain the detection result. The detection result is used to indicate whether there are splittable child nodes in the child node set.

[0116] The moving module 606 is used to move the splittable child node to the set of nodes to be split when the detection result indicates that there is a splittable child node in the child node set.

[0117] The first input module 607 is used to input the fuzzy sub-partitions corresponding to each node in the set of nodes to be split as fuzzy partitions into the second mapping reduction model, so as to obtain multiple fuzzy sub-partitions output by the second mapping reduction model.

[0118] The second input module 608 is used to add multiple fuzzy sub-partitions output by the second mapping reduction model as child nodes to the child node set, and re-execute the step of split detection on the child nodes in the child node set to obtain the detection result, until the detection result indicates that there are no splittable child nodes in the child node set, and then execute the step of constructing a fuzzy decision tree for the dataset based on the root node set and the child node set.

[0119] As another embodiment of this application, in order to improve the efficiency of constructing fuzzy decision trees based on distributed computing, the first mapping reduction module 601 may further include:

[0120] The calculation unit 6011 is used to calculate the fuzzy entropy information corresponding to each attribute based on multiple attributes of the data in the constructed dataset, using a first mapping reduction model, when the constructed dataset is obtained.

[0121] The determining unit 6022 is used to determine the fuzzy partition corresponding to each attribute based on the calculated fuzzy entropy information corresponding to each attribute.

[0122] As another embodiment of this application, in order to improve the efficiency of constructing fuzzy decision trees based on distributed computing, the aforementioned second mapping reduction module 602 can also be specifically used for

[0123] The second mapping reduction model is used to segment the data of each fuzzy partition in multiple fuzzy partitions based on multiple attributes of the data in each fuzzy partition, thereby obtaining multiple fuzzy sub-partitions of each fuzzy partition.

[0124] Based on the fuzzy decision tree construction method and apparatus provided in the above embodiments, this application also provides an electronic device 700, such as... Figure 7 As shown:

[0125] It includes a processor 701, a memory 702, and a computer program stored in the memory 702 and executable on the processor 701. When the computer program is executed by the processor 701, it implements the various processes of the above-described fuzzy decision tree construction method embodiment and achieves the same technical effect.

[0126] Specifically, the processor 701 may include a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits that can be configured to implement the embodiments of this application.

[0127] Memory 702 may include mass storage for data or instructions. For example, and not limitingly, memory 702 may include a hard disk drive (HDD), floppy disk drive, flash memory, optical disk, magneto-optical disk, magnetic tape, or Universal Serial Bus (USB) drive, or a combination of two or more of these. Where appropriate, memory 702 may include removable or non-removable (or fixed) media. Where appropriate, memory 702 may be internal or external to the integrated gateway disaster recovery device. In a particular embodiment, memory 702 is non-volatile solid-state memory.

[0128] In certain embodiments, the memory may include read-only memory (ROM), random access memory (RAM), disk storage media devices, optical storage media devices, flash memory devices, and electrical, optical, or other physical / tangible memory storage devices. Thus, typically, memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., memory devices) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described with reference to the method according to one aspect of this application.

[0129] The processor 701 reads and executes computer program instructions stored in the memory 702 to implement any of the fuzzy decision tree construction methods in the above embodiments.

[0130] In one example, the electronic device may also include a communication interface 703 and a bus 710. As an example, such as... Figure 7 As shown, the processor 701, memory 702, and communication interface 703 are connected through bus 710 and complete communication with each other.

[0131] The communication interface 703 is mainly used to realize communication between various modules, devices, units and / or equipment in the embodiments of this application.

[0132] Bus 710 includes hardware, software, or both, that couples components of an online data traffic metering device together. For example, and not limitingly, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an Infinite Bandwidth Interconnect, a Low Pin Count (LPC) bus, a memory bus, a Microchannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses, or combinations of two or more of these. Where appropriate, bus 710 may include one or more buses. Although specific buses are described and illustrated in embodiments of this application, any suitable bus or interconnect is contemplated herein.

[0133] This application also provides a computer-readable storage medium storing a computer program. When executed by a processor, the computer program implements the various processes of the above-described fuzzy decision tree construction method embodiments and achieves the same technical effects. To avoid repetition, it will not be described again here. The computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0134] It should be clarified that this application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of this application is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of this application.

[0135] The functional blocks shown in the above block diagram can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, they can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this application are programs or code segments used to perform the required tasks. Programs or code segments can be stored on a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried on a carrier wave. "Machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical disks, hard disks, fiber optic media, radio frequency (RF) links, etc. Code segments can be downloaded via computer networks such as the Internet, intranets, etc.

[0136] It should also be noted that the exemplary embodiments mentioned in this application describe methods or systems based on a series of steps or apparatus. However, this application is not limited to the order of the above steps; that is, the steps can be performed in the order mentioned in the embodiments, or in a different order, or several steps can be performed simultaneously.

[0137] The aspects of this application have been described above with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to embodiments of this application. It should be understood that each block in the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that these instructions, executable via the processor of the computer or other programmable data processing apparatus, enable the implementation of the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field-programmable logic circuit. It is also understood that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can also be implemented by dedicated hardware performing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

[0138] The above are merely specific embodiments of this application. Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here. It should be understood that the protection scope of this application is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in this application, and these modifications or substitutions should all be covered within the protection scope of this application.

Claims

1. A method for constructing a fuzzy decision tree based on multiple machines, characterized in that, include: Given a constructed dataset, the data in the constructed dataset is mapped and reduced using a first mapping and reduction model to obtain multiple fuzzy partitions in the constructed dataset. Each fuzzy partition includes multiple data in the constructed dataset, wherein the data in the constructed dataset includes multiple data blocks. The first mapping and reduction model performs mapping and reduction processing on the multiple data blocks in parallel on multiple machines to obtain the multiple fuzzy partitions. The data of each of the multiple fuzzy partitions is mapped and reduced by the second mapping and reduction model to obtain multiple fuzzy sub-partitions of each fuzzy partition. Each fuzzy sub-partition includes at least one piece of data from the fuzzy partition. The second mapping and reduction model is based on multiple machines performing mapping and reduction processing on each of the fuzzy partitions in parallel to obtain the multiple fuzzy sub-partitions. Each of the fuzzy partitions is taken as a root node to obtain a root node set including multiple root nodes; and each of the fuzzy sub-partitions of the fuzzy partitions is taken as a child node to obtain a child node set corresponding to the root node of the fuzzy partition, wherein the child node set includes multiple child nodes. Based on the root node set and the child node set, a fuzzy decision tree for the constructed dataset is constructed.

2. The method according to claim 1, characterized in that, The step of taking the fuzzy sub-partitions of each fuzzy partition as child nodes to obtain the set of child nodes of the root node corresponding to the fuzzy partition includes: Split detection is performed on the child nodes in the child node set to obtain detection results, which are used to indicate whether there are splittable child nodes in the child node set; If the detection result indicates that there are splittable child nodes in the set of child nodes, the splittable child nodes are moved to the set of nodes to be split. The fuzzy sub-partitions corresponding to each node in the set of nodes to be split are input as fuzzy partitions into the second mapping reduction model to obtain multiple fuzzy sub-partitions output by the second mapping reduction model. The multiple fuzzy sub-partitions output by the second mapping reduction model are added as child nodes to the child node set, and the step of re-performing the split detection of the child nodes in the child node set to obtain the detection result is repeated until the detection result indicates that there are no splittable child nodes in the child node set. Then, the step of constructing the fuzzy decision tree of the constructed dataset is performed based on the root node set and the child node set.

3. The method according to claim 1, characterized in that, Given a constructed dataset, the data in the constructed dataset is mapped and reduced using a first mapping and reduction model to obtain multiple fuzzy partitions in the constructed dataset, including: Given a constructed dataset, the fuzzy entropy information corresponding to each attribute is calculated using the first mapping reduction model based on multiple attributes of the data in the constructed dataset. Based on the calculated fuzzy entropy information corresponding to each attribute, the fuzzy partition corresponding to each attribute is determined.

4. The method according to claim 3, characterized in that, The process of performing mapping and reduction processing on the data of each of the plurality of fuzzy partitions through the second mapping and reduction model to obtain a plurality of fuzzy sub-partitions of each of the fuzzy partitions includes: Using the second mapping reduction model, the data of each fuzzy partition in the plurality of fuzzy partitions is segmented according to multiple attributes of the data in each fuzzy partition, to obtain multiple fuzzy sub-partitions of each fuzzy partition.

5. A fuzzy decision tree construction device based on multiple machines, characterized in that, include: The first mapping and reduction module is used to perform mapping and reduction processing on the data in the construction dataset through a first mapping and reduction model when the construction dataset is obtained, to obtain multiple fuzzy partitions in the construction dataset. Each fuzzy partition includes multiple data in the construction dataset, wherein the data in the construction dataset includes multiple data blocks. The first mapping and reduction model performs mapping and reduction processing on the multiple data blocks in parallel on multiple machines to obtain the multiple fuzzy partitions. The second mapping and reduction module is used to perform mapping and reduction processing on the data of each of the multiple fuzzy partitions through the second mapping and reduction model to obtain multiple fuzzy sub-partitions of each fuzzy partition. Each fuzzy sub-partition includes at least one piece of data in the fuzzy partition. The second mapping and reduction model performs mapping and reduction processing on each of the fuzzy partitions in parallel on multiple machines to obtain the multiple fuzzy sub-partitions. The determining module is configured to take each of the fuzzy partitions as root nodes to obtain a root node set including multiple root nodes; and to take the fuzzy sub-partitions of each of the fuzzy partitions as child nodes to obtain a child node set corresponding to the root node of the fuzzy partition, wherein the child node set includes multiple child nodes. A construction module is used to construct a fuzzy decision tree for the constructed dataset based on the root node set and the child node set.

6. An electronic device, characterized in that, The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, it implements the fuzzy decision tree construction method based on multiple machines as described in any one of claims 1-4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer program instructions, which, when executed by a processor, implement the fuzzy decision tree construction method based on multiple machines as described in any one of claims 1-4.

8. A computer program product, characterized in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device performs the fuzzy decision tree construction method based on multiple machines as described in any one of claims 1-4.

Images