Big data cluster-based data binning method, device and equipment and storage medium

Abstract

The present specification relates to the technical field of big data processing, and provides a data binning method and device based on a big data cluster, equipment and a storage medium. The method comprises the following steps: reading a data binning configuration file to obtain user-defined information; the user-defined information comprises a bin number and an evaluation function; determining a mapping number and a reduction number of a mapping reduction task according to a node cluster state and a data volume of a feature table; in the mapping stage, each Map is caused to separate each row of data in the feature table according to columns, and an index value of each column of data in the corresponding row is determined; in the shuffle stage, a partitioner is caused to place features of a same complete column in a same partition according to the index value; in the reduction stage, each Reduce reads a column of data from a corresponding partition and calls the evaluation function to process the column of data, and the processing results of each Reduce are reduced according to the bin number. The embodiments of the present specification can improve the processing efficiency of data binning processing and reduce resource consumption.

Description

Technical Field

[0001] This specification relates to the field of big data processing technology, and in particular to a data binning method, apparatus, device and storage medium based on big data clusters. Background Technology

[0002] In big data processing, data binning is a common process, which involves discretizing features using binning components (e.g., segmenting a continuous variable into multiple discrete intervals). Binning allows for a better understanding of data distribution and can improve the accuracy of data predictions to some extent. Therefore, data binning is of great significance in big data processing.

[0003] Currently, when performing binning operations based on big data tools, there are generally two methods:

[0004] Binning is performed using generic Hive SQL, such as equal-frequency binning and equal-distance binning, and the intermediate computation results are stored on the Hadoop Distributed File System (HDFS) cluster. Hive SQL is a tool provided by Hive that supports generic SQL syntax. It translates user-written SQL logic into multiple MapReduce tasks, which must be executed sequentially; that is, the second task can only start after the first MapReduce task has finished executing, and so on. This method generates multiple MapReduce computation stages depending on how the user writes the SQL and the table structure. If other logical computations are involved, the SQL logic also needs to be written manually, which will also generate additional MapReduce computation stages. Since each MapReduce computation saves intermediate data to the HDFS distributed file storage system, there is a significant I / O consumption and time cost associated with resource scheduling.

[0005] Another approach is to download the data to a local machine and process it using data processing tools such as pandas. However, with large datasets, the download speed is significantly affected by network bandwidth. Since this runs on a single machine, distributed computing capabilities cannot be utilized, and the limited performance of a personal computer restricts the amount of data that can be processed. Summary of the Invention

[0006] The purpose of the embodiments in this specification is to provide a data binning method, apparatus, device and storage medium based on big data clusters, so as to improve the processing efficiency of data binning and reduce the resource consumption of data binning.

[0007] To achieve the above objectives, in one aspect, embodiments of this specification provide a data binning method based on a big data cluster, including:

[0008] Read the data binning configuration file to obtain user-defined information; the user-defined information includes the number of bins and the evaluation function.

[0009] Based on the node cluster status and the amount of data in the feature table, determine the number of Map and Reduce tasks in the MapReduce task;

[0010] In the Map phase of the MapReduce task, each Map task splits each row of data in the feature table by column and determines the index value of each column of data in the corresponding row.

[0011] During the Shuffle phase of the MapReduce task, the partitioner places features of the same complete column in the same partition based on the index value.

[0012] In the Reduce phase of the MapReduce task, each Reducer reads a column of data from the corresponding partition and calls the evaluation function to process the column of data. The processing results of each Reducer are reduced according to the number of bins.

[0013] In the data binning method based on big data clusters in the embodiments of this specification, the number of Map tasks in a MapReduce task is determined according to the node cluster status and the amount of data in the feature table, including:

[0014] According to the formula Determine the number of Map tasks in a MapReduce task;

[0015] Wherein, min is the minimum value function, and its value in this formula represents the number of Maps; s is the amount of data in the feature table; t is the maximum amount of data processed by a single node in the node cluster in a single operation; and n is the number of available nodes in the node cluster.

[0016] In the data binning method based on big data clusters in the embodiments of this specification, the number of Reduce tasks in a MapReduce task is determined according to the node cluster status and the amount of data in the feature table, including:

[0017] The number of Reduce tasks in a MapReduce task is determined using the formula min′(c,n).

[0018] Wherein, min is the minimum value function, and its value in this formula represents the number of Reduces; c is the number of columns in the feature table that participate in binning; and n is the number of available nodes in the node cluster.

[0019] In the data binning method based on big data clusters in the embodiments of this specification, the Shuffle stage of the MapReduce task further includes:

[0020] When you need to distribute data from multiple complete columns in the same partition, write each complete column sequentially.

[0021] In the data binning method based on big data clusters in the embodiments of this specification, the user-defined information also includes binning method;

[0022] Correspondingly, the reduction of the processing results of each Reduce based on the number of bins includes:

[0023] The processing results of each Reduce are reduced based on the number of bins and the binning method.

[0024] In the data binning method based on big data clusters in the embodiments of this specification, each index value has the same length.

[0025] On the other hand, embodiments of this specification also provide a data binning device based on a big data cluster, comprising:

[0026] The information acquisition module is used to read the data bin configuration file to obtain user-defined information; the user-defined information includes the number of bins and the evaluation function.

[0027] The quantity determination module is used to determine the number of Map and Reduce tasks in a MapReduce task based on the node cluster status and the amount of data in the feature table.

[0028] The data mapping module is used in the Map phase of the MapReduce task to separate each row of data in the feature table by column and determine the index value of each column of data in the corresponding row for each Map phase.

[0029] The data shuffling module is used in the Shuffle phase of the MapReduce task to enable the partitioner to place features of the same complete column in the same partition according to the index value.

[0030] The data reduction module is used in the Reduce phase of the MapReduce task to enable each Reduce to read a column of data from the corresponding partition and call the evaluation function to process the column of data, and to reduce the processing results of each Reduce according to the number of bins.

[0031] On the other hand, embodiments of this specification also provide a computer device, including a memory, a processor, and a computer program stored in the memory, wherein the computer program, when run by the processor, executes instructions for the above-described method.

[0032] On the other hand, embodiments of this specification also provide a computer storage medium storing a computer program thereon, which, when run by the processor of a computer device, executes instructions for the above-described method.

[0033] On the other hand, embodiments of this specification also provide a computer program product, which includes a computer program that, when run by the processor of a computer device, executes instructions for the above-described method.

[0034] As can be seen from the technical solutions provided in the embodiments of this specification above, in these embodiments, users do not need to write SQL themselves; they only need to configure a data binning configuration file (including configuring user-defined information such as the number of bins, evaluation functions, etc.). During execution, the server node can obtain the user-defined information by reading the data binning configuration file, and then complete the binning operation and evaluation function calculation within a single MapReduce framework. The calculation is performed entirely in memory, without going through a distributed file system, thus avoiding the problem of automatically generating multiple MapReduce operations and consuming a large amount of I / O by writing SQL, thereby reducing the resource consumption of data binning. In addition, since the embodiments of this specification are also implemented through the MapReduce framework, there is no need to download large amounts of data, and multiple server nodes in a big data cluster can be used to calculate the binning operation. The efficiency improvement is particularly significant when there are many features, thereby improving the processing efficiency of data binning. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0036] Figure 1 This specification shows schematic diagrams of data binning systems based on big data clusters in some embodiments;

[0037] Figure 2 Flowcharts of data binning methods based on big data clusters in some embodiments of this specification are shown;

[0038] Figure 3 This specification shows a schematic diagram of the partitioning of MapReduce tasks in some embodiments;

[0039] Figure 4This specification shows a structural block diagram of a data binning device based on a big data cluster in some embodiments;

[0040] Figure 5 A structural block diagram of a computer device in some embodiments of this specification is shown.

[0041] [Explanation of Labels in the Attached Image]

[0042] 10. User terminal;

[0043] 20. Server-side node;

[0044] 30. Configuration Center;

[0045] 41. Information Acquisition Module;

[0046] 42. Quantity Determination Module;

[0047] 43. Data Mapping Module;

[0048] 44. Data shuffling module;

[0049] 45. Data Specification Module;

[0050] 502. Computer equipment;

[0051] 504, Processor;

[0052] 506. Memory;

[0053] 508. Drive mechanism;

[0054] 510. Input / output interface;

[0055] 512. Input devices;

[0056] 514. Output devices;

[0057] 516. Presentation equipment;

[0058] 518. Graphical User Interface;

[0059] 520. Network interface;

[0060] 522. Communication link;

[0061] 524. Communication bus. Detailed Implementation

[0062] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0063] The embodiments in this specification relate to data binning technology based on big data clusters and MapReduce. MapReduce is a programming model used for parallel computation of large-scale datasets. The core idea of ​​MapReduce is divide and conquer, that is, to break down a large or complex task into multiple subtasks (the subtasks can be broken down as long as they are independent of each other), process them in parallel, and finally reduce the results of each subtask. MapReduce mainly includes the Map (phase) and the Reduce (phase). The Map phase: splits the data, that is, decomposes the complex task into several subtasks to facilitate parallel processing; the Reduce phase: processes each subtask in parallel and performs a global reduction on the results.

[0064] In view of the problem that existing data binning technologies based on big data clusters and MapReduce cannot balance processing efficiency and resource consumption, the embodiments in this specification improve the data binning technology based on big data clusters and MapReduce.

[0065] Figure 1 The diagram illustrates a data binning system based on a big data cluster, as shown in some embodiments of this specification. This system is a data binning system based on a big data cluster and MapReduce, and may include a user terminal 10, a node cluster (including multiple server nodes 20), and a configuration center 30. Users can configure the schema of a feature table in the configuration center 30 through the user terminal 10. This schema may contain user-defined binning-related information (hereinafter referred to as user-defined information). The user-defined information may include the number of bins and evaluation functions, etc. The node cluster may be a distributed system node cluster.

[0066] The server node 20 in the node cluster can be configured to: read the data binning configuration file from the configuration center 30 to obtain user-defined information; determine the number of Map and Reduce tasks in the MapReduce task based on the node cluster status and the amount of data in the feature table; in the Map phase of the MapReduce task, each Map task splits each row of data in the feature table by column and determines the index value of each column of data in the corresponding row; in the Shuffle phase of the MapReduce task, the partitioner places features of the same complete column in the same partition according to the index value; in the Reduce phase of the MapReduce task, each Reduce task reads a column of data from the corresponding partition and calls the evaluation function to process the column of data, and reduces the processing results of each Reduce task according to the number of bins.

[0067] In some embodiments, the user terminal 10 can be a self-service terminal device, a mobile terminal (i.e., a smartphone), a display, a desktop computer, a tablet computer, a laptop computer, a digital assistant, or a smart wearable device, etc. Smart wearable devices can include smart bracelets, smartwatches, smart glasses, or smart helmets, etc. Of course, the user terminal 10 is not limited to the above-mentioned physical electronic devices; it can also be software running on the aforementioned electronic devices. The server node 20 can be an electronic device with computing and network interaction functions; it can also be software running on the electronic device that provides business logic for data processing and network interaction.

[0068] This specification provides a data binning method based on a big data cluster, which can be applied to the server-side node side described above. (Refer to...) Figure 2 As shown, in some embodiments, the data binning method based on big data clusters may include the following steps:

[0069] Step 201: Read the data binning configuration file to obtain user-defined information; the user-defined information includes the number of bins and the evaluation function;

[0070] Step 202: Determine the number of Map and Reduce tasks in the MapReduce task based on the node cluster status and the amount of data in the feature table;

[0071] Step 203: In the Map phase of the MapReduce task, each Map operation separates each row of data in the feature table by column and determines the index value of each column of data in the corresponding row.

[0072] Step 204: In the Shuffle phase of the MapReduce task, the partitioner places features of the same complete column in the same partition according to the index value.

[0073] Step 205: In the Reduce phase of the MapReduce task, each Reduce reads a column of data from the corresponding partition and calls the evaluation function to process the column of data, and reduces the processing results of each Reduce according to the number of bins.

[0074] In the embodiments described in this specification, users do not need to write SQL themselves; they only need to configure a data binning configuration file (including user-defined information such as the number of bins and evaluation functions). During execution, the server node obtains the user-defined information by reading the data binning configuration file. Then, the binning operation and evaluation function calculation can be completed within a single MapReduce framework. The calculation is performed entirely in memory, without going through a distributed file system, thus avoiding the problem of automatically generating multiple MapReduce operations and consuming a large amount of I / O by writing SQL, thereby reducing the resource consumption of data binning. In addition, since the embodiments in this specification are also implemented using the MapReduce framework, there is no need to download large amounts of data, and multiple server nodes in a big data cluster can be used for binning operations. The efficiency improvement is particularly significant when there are many features, thereby improving the processing efficiency of data binning.

[0075] Data binning configuration files can be created by the user in the configuration center. These files can include user-defined form information (i.e., user-defined information), such as the number of bins, binning methods, and evaluation functions. This configuration significantly improves the flexibility and scalability of data binning implementation. For example, for floating-point features, two basic binning methods can be provided: equal-frequency binning and equal-distance binning, along with an entry point for custom binning methods. If users need to customize binning, such as with existing third-party binning libraries, they only need to provide the address of the binning method when submitting the configuration, and the binning program will be automatically loaded based on the user's configuration. For string type features, custom binning logic can be used.

[0076] The binning number refers to how many intervals the data is divided into. For example, if the feature is age, and the binning number is 5, then the age range of 0 to 100 can be divided into 5 intervals. For example, with the binning method of equal interval binning, the age range of 0 to 100 can be divided into five intervals: [0, 20], [21, 40], [41, 60], [61, 80], [81, 100].

[0077] Binning method refers to the binning approach. Binning methods generally include unsupervised binning and supervised binning. Unsupervised binning can include, for example, equal-distance binning, equal-frequency binning, and binning based on k-means clustering. Supervised binning can include, for example, chi-square binning, Best-KS binning, and decision tree binning. Furthermore, third-party binning methods can be configured; simply provide the address of the binning method when submitting the configuration so that the binning program can be automatically loaded from this address later.

[0078] Evaluation functions are metrics used to measure the predictive power of features for a model. Typical evaluation functions include, but are not limited to, the Population Stability Index (PSI), Kolmogorov-Smirnov (KS), Information Value (IV), and Weight of Evidence (WOE). Of course, in other embodiments, any data processing tool (such as pandas, sparkmlib, etc.) can be configured to perform binning and subsequent evaluation calculations on the features, as needed.

[0079] By determining the number of Map and Reduce tasks in a MapReduce task based on the node cluster status and the amount of data in the feature table, the efficiency of data binning can be balanced with avoiding waste of system resources.

[0080] In some embodiments, determining the number of Map tasks in a MapReduce task based on the node cluster status and the amount of data in the feature table may include: using the formula Determine the number of Map tasks in the MapReduce task; where min is the minimum value function, and its value in this formula represents the number of Map tasks; s is the amount of data in the feature table (e.g., the number of data records in the feature table (i.e., the number of data rows)); t is the maximum amount of data processed by a single node in the node cluster; and n is the number of available nodes in the node cluster (i.e., the number of idle and available nodes).

[0081] In some embodiments, determining the number of Reduce tasks in a MapReduce task based on the node cluster status and the amount of data in the feature table may include: determining the number of Reduce tasks in a MapReduce task according to the formula min′(c,n); where min is a minimum value function, and its value in this formula represents the number of Reduce tasks; c is the number of columns in the feature table that participate in binning (which data participates in binning can be specified by the user); and n is the number of available nodes in the node cluster.

[0082] The feature table is the table where the raw data is written. It mainly contains: user identification data (e.g., user_id), feature data (e.g., age, gender, credit rating, etc.), and user evaluation results (e.g., users who meet the requirements are identified as 1, and users who do not meet the requirements are identified as 0). In an exemplary embodiment, the feature table can be a Hive table (Hive is a data warehouse tool based on Hadoop used for data extraction, transformation, and loading; it is a mechanism for storing, querying, and analyzing large-scale data stored in Hadoop). Features are stored column-wise, and there are two types: continuous features and discrete features. Continuous feature data is generally in floating-point format, while discrete features can be numeric or string types.

[0083] It should be noted that the node cluster status changes dynamically, and the data volume of different feature tables may also vary. Therefore, when facing different MapReduce tasks, it is necessary to determine the number of Map and Reduce tasks based on the corresponding node cluster status and the data volume of the feature tables.

[0084] In the traditional Map phase, data is read row by row, partitioned, and then processed by different Reduce programs. For example, rows 1-1000 are processed by Reduce1, rows 1001-2000 by Reduce2. This operation is encapsulated by the MapReduce framework and requires no user intervention. However, the embodiments in this specification modify the Map phase. Specifically, in the embodiments of this specification, each Map program reads a small portion of the data, partitions the data row by row according to the table's metadata, and calculates a key value for each column (the key value is the index value of the feature column order). This ensures that even if processed by different Map programs, the same column of data has the same key value. For example, if the column index is 1 and the feature column contains 2000 fields, the output key length should be at least 4 characters, and the key value range should be 0001-2000. After the Map output, the key values ​​can be sorted, so all keys should have the same length, i.e., each index value should have the same length; if the index value is not long enough, zeros are padded on the left to ensure correct sorting.

[0085] In a MapReduce task, there is a Shuffle operation (i.e., the Shuffle phase) between the Map and Reduce phases. This operation is also invisible to the user. The Shuffle phase actually controls which partition the data is written to and which Reduce phase processes it. In this phase, in the embodiments of this specification, a customized partitioning method is used, namely a customized Partitioner. The Partitioner can partition the data according to the index values ​​output by the Map phase, so that data of the same complete column is written to the same partition. When it is necessary to distribute data of multiple complete columns in the same partition, each complete column is written sequentially, that is, the writing of the second column of data will only begin after the first column of data has been completely written; this can help with load balancing in the subsequent Reduce phase.

[0086] For example, in such Figure 3 In the exemplary embodiment shown, the first line contains the table structure information of the feature table. Here, col1 to col4 represent features (i.e., feature data). y_tag represents the user evaluation result. The second line describes the actual operations performed in the Map phase (where map1 and map2 are two map processes), i.e., data processing according to columns. The third line represents the shuffle phase, where the operation is to place data with the same key value into the same partition based on the key output from the Map phase. For example, in the third line, all data with index 0000 is placed in the partition on the left side of the third line, and all data with index 0001 is placed in the partition on the right side of the third line. In the third line, field_name is the feature name, and value is the feature value.

[0087] In the traditional Reduce phase, users need to pre-write SQL (such as Hive SQL). Hive SQL automatically generates Reduce tasks that encapsulate common computational operations, such as sorting, counting, and averaging data. Hive SQL achieves user needs by combining different operations. However, in the embodiments described in this specification, users can pre-define binning numbers, evaluation logic (i.e., evaluation functions), etc., through configuration files. In the Reduce phase, each Reduce process reads a column of data from its corresponding partition and calls the pre-configured evaluation function to process that column (each Reduce process processes at least one complete column's characteristics to ensure that the same complete column is processed within the same Reduce). The processing results of each Reduce are then reduced (i.e., merged and summarized) according to the pre-configured binning number.

[0088] In the embodiments described in this specification, during the Reduce phase, the value information can be used to determine whether the current column is a numeric or string type. For numeric types, all values ​​can be converted to floating-point values. For string types, since strings are discrete variables, if binning is required, they need to be converted to floating-point values ​​according to certain rules.

[0089] Although the process described above includes multiple operations that occur in a specific order, it should be clearly understood that these processes may include more or fewer operations, which may be executed sequentially or in parallel (e.g., using parallel processors or a multithreaded environment).

[0090] Corresponding to the data binning method based on big data clusters described above, this specification also provides a data binning device based on big data clusters, see reference. Figure 4 As shown, in some embodiments, the data binning device based on a big data cluster may include:

[0091] Information acquisition module 41 is used to read the data bin configuration file to obtain user-defined information; the user-defined information includes the number of bins and the evaluation function;

[0092] The quantity determination module 42 is used to determine the number of Map and Reduce tasks in a MapReduce task based on the node cluster status and the amount of data in the feature table.

[0093] The data mapping module 43 is used in the Map phase of the MapReduce task to separate each row of data in the feature table by column and determine the index value of each column of data in the corresponding row for each Map phase.

[0094] The data shuffling module 44 is used to enable the partitioner to place features of the same complete column in the same partition according to the index value during the Shuffle phase of the MapReduce task.

[0095] The data reduction module 45 is used in the Reduce phase of the MapReduce task to enable each Reduce to read a column of data from the corresponding partition and call the evaluation function to process the column of data, and to reduce the processing results of each Reduce according to the number of bins.

[0096] In some embodiments of the data binning device based on big data clusters, the number of Map tasks in a MapReduce task is determined according to the node cluster status and the amount of data in the feature table, including:

[0097] According to the formula Determine the number of Map tasks in a MapReduce task;

[0098] Wherein, min is the minimum value function, and its value in this formula represents the number of Maps; s is the amount of data in the feature table; t is the maximum amount of data processed by a single node in the node cluster in a single operation; and n is the number of available nodes in the node cluster.

[0099] In some embodiments of the data binning device based on big data clusters, the number of Reduce tasks in a MapReduce task is determined according to the node cluster status and the amount of data in the feature table, including:

[0100] The number of Reduce tasks in a MapReduce task is determined using the formula min′(c,n).

[0101] Wherein, min is the minimum value function, and its value in this formula represents the number of Reduces; c is the number of columns in the feature table that participate in binning; and n is the number of available nodes in the node cluster.

[0102] In some embodiments of the data binning device based on big data clusters, during the Shuffle phase of the MapReduce task, the following further includes:

[0103] When you need to distribute data from multiple complete columns in the same partition, write each complete column sequentially.

[0104] In some embodiments of data binning devices based on big data clusters, the user-defined information further includes binning methods;

[0105] Correspondingly, the reduction of the processing results of each Reduce based on the number of bins includes:

[0106] The processing results of each Reduce are reduced based on the number of bins and the binning method.

[0107] In the data binning device based on big data clusters in the embodiments of this specification, each index value has the same length.

[0108] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0109] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the embodiments of this specification are all information and data authorized and agreed upon by the user and fully authorized by all parties.

[0110] Embodiments of this specification also provide a computer device. For example... Figure 5As shown, in some embodiments of this specification, the computer device 502 may include one or more processors 504, such as one or more central processing units (CPUs) or graphics processing units (GPUs), each of which may implement one or more hardware threads. The computer device 502 may also include any memory 506 for storing any kind of information such as code, settings, data, etc. In one specific embodiment, a computer program is stored on the memory 506 and can run on the processor 504. When the computer program is run by the processor 504, it can execute instructions of the data binning method based on big data clusters described in any of the above embodiments. Non-limitingly, for example, the memory 506 may include any type of RAM, any type of ROM, flash memory, hard disk, optical disk, etc. More generally, any memory can use any technology to store information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of the computer device 502. In one case, when the processor 504 executes associated instructions stored in any memory or combination of memories, the computer device 502 can perform any operation of the associated instructions. Computer device 502 also includes one or more drive mechanisms 508 for interacting with any memory, such as hard disk drive mechanism, optical disk drive mechanism, etc.

[0111] Computer device 502 may also include an input / output interface 510 (I / O) for receiving various inputs (via input device 512) and providing various outputs (via output device 514). A specific output mechanism may include a presentation device 516 and an associated graphical user interface 518 (GUI). In other embodiments, the input / output interface 510 (I / O), input device 512, and output device 514 may be omitted, and the device may function solely as a computer device within a network. Computer device 502 may also include one or more network interfaces 520 for exchanging data with other devices via one or more communication links 522. One or more communication buses 524 couple the components described above together.

[0112] Communication link 522 can be implemented in any way, such as via a local area network, a wide area network (e.g., the Internet), a point-to-point connection, or any combination thereof. Communication link 522 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., governed by any protocol or combination of protocols.

[0113] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), computer-readable storage media, and computer program products according to some embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processor to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processor, create a mechanism for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0114] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processor to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0115] These computer program instructions may also be loaded onto a computer or other programmable data processor, causing a series of operational steps to be performed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable device for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0116] In a typical configuration, a computer device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0117] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0118] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by computer equipment. As defined in this specification, computer-readable media does not include transient media, such as modulated data signals and carrier waves.

[0119] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, the embodiments of this specification can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, the embodiments of this specification can take the form of computer program products implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0120] The embodiments described in this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. The embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processors connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0121] It should also be understood that, in the embodiments of this specification, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0122] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0123] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the embodiments of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0124] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A data binning method based on a big data cluster, characterized in that, include: Read the data bin configuration file to obtain user-defined information; The user-defined information includes the number of bins and the evaluation function; Based on the node cluster status and the amount of data in the feature table, determine the number of Map and Reduce tasks in the MapReduce task; In the Map phase of the MapReduce task, each Map task divides each row of data in the feature table by column, determines the index value of each column of data in the corresponding row, and ensures that the index value of the same column of data is the same in different MapReduce tasks, and that the length of each index value is the same. During the Shuffle phase of the MapReduce task, the partitioner places features of the same complete column in the same partition based on the index value. In the Reduce phase of the MapReduce task, each Reducer reads a column of data from the corresponding partition and calls the evaluation function to process the column of data. The processing results of each Reducer are reduced according to the number of bins.

2. The data binning method based on big data clusters as described in claim 1, characterized in that, Based on the node cluster status and the amount of data in the feature table, determine the number of Map tasks in the MapReduce task, including: According to the formula Determine the number of Map tasks in a MapReduce task; Where min is the minimum value function, and its value in this formula represents the number of Maps; s The amount of data in the feature table; t This represents the maximum amount of data that a single node in the node cluster can process in a single operation. n This represents the number of available nodes in the node cluster.

3. The data binning method based on big data clusters as described in claim 1, characterized in that, Based on the node cluster status and the amount of data in the feature table, determine the number of Reduce tasks in the MapReduce task, including: According to the formula Determine the number of Reducers in a MapReduce task; Where min′ is the minimum value function, and its value in this formula represents the number of Reduces; c This refers to the number of columns in the feature table that participate in binning; n This represents the number of available nodes in the node cluster.

4. The data binning method based on big data clusters as described in claim 1, characterized in that, The Shuffle phase of the MapReduce task also includes: When you need to distribute data from multiple complete columns in the same partition, write each complete column sequentially.

5. The data binning method based on a big data cluster as described in claim 1, characterized in that, The user-defined information also includes the binning method; Correspondingly, the reduction of the processing results of each Reduce based on the number of bins includes: The processing results of each Reduce are reduced based on the number of bins and the binning method.

6. A data binning device based on a big data cluster, characterized in that, include: The information acquisition module is used to read the data bin configuration file to obtain user-defined information; The user-defined information includes the number of bins and the evaluation function; The quantity determination module is used to determine the number of Map and Reduce tasks in a MapReduce task based on the node cluster status and the amount of data in the feature table. The data mapping module is used in the Map phase of the MapReduce task to separate each row of data in the feature table by column, determine the index value of each column of data in the corresponding row, and make the index value of the same column of data the same in different MapReduce tasks, and the length of each index value is the same. The data shuffling module is used in the Shuffle phase of the MapReduce task to enable the partitioner to place features of the same complete column in the same partition according to the index value. The data reduction module is used in the Reduce phase of the MapReduce task to enable each Reduce to read a column of data from the corresponding partition and call the evaluation function to process the column of data, and to reduce the processing results of each Reduce according to the number of bins.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, When the computer program is run by the processor, it executes the instructions of the method according to any one of claims 1-5.

8. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is run by the processor of the computer device, it executes the instructions of the method according to any one of claims 1-5.

9. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, performs instructions according to any one of claims 1-5.

Images