A MapReduce memory overflow solution and device

Abstract

The present invention discloses a method and device for solving MapReduce memory overflow, including: pre-deduplication processing of the initial data files in the processing table based on a preset joint primary key to obtain at least one target data file; determining the number of tables to be divided, and creating a table division path based on the number of tables to be divided; copying the corresponding target data files to the corresponding table division path based on the number of tables to be divided and the preset table division logic; processing the target data files contained therein in parallel based on the MapReduce interface under each table division path to obtain at least one table division result; merging the table division results to obtain the target table division result. In the above process, the initial data files in the processing table are first deduplicated, and further, the obtained target data is processed in parallel under the table division path, which not only reduces the amount of data files processed by MapReduce, but also adopts a parallel processing method to improve the processing efficiency.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular to a MapReduce memory overflow solution and device. Background Art

[0002] As a computing model, framework and platform for parallel processing of big data, MapReduce has good scalability and is suitable for offline processing of massive data above PB level, playing an important role in data processing of large amounts of data. However, when the amount of data surges to three to four times the original amount of data and there is serious data skew, the amount of data allocated to one or several Reduces will be much higher than the average, resulting in very low processing efficiency and even serious Reduce memory overflow.

[0003] In actual scenarios, HQL statements can be directly called in HIVE to perform serial data processing to solve the above-mentioned memory overflow problem. However, compared with the MapReduce interface, the processing efficiency is extremely low, which affects the normal business process. Summary of the invention

[0004] In view of this, the present invention provides a MapReduce memory overflow solution and device, which is used to solve the above memory overflow situation. In actual scenarios, HQL statements can be directly called in HIVE to perform serial data processing to solve the above memory overflow situation, but compared with the MapReduce interface, there is a problem of extremely low processing efficiency. The specific solution is as follows:

[0005] A solution to MapReduce memory overflow includes:

[0006] Performing pre-deduplication processing on the initial data files in the to-be-processed table based on the preset joint primary key to obtain at least one target data file;

[0007] Determine the number of tables to be divided, and create a table division path based on the number of tables to be divided;

[0008] Based on the number of tables to be divided and the preset table division logic, the corresponding target data file is copied to the corresponding table division path;

[0009] Under each sharding path, the target data files contained therein are processed in parallel based on the MapReduce interface to obtain at least one sharding result;

[0010] The sub-table results are merged to obtain the target sub-table results.

[0011] The above method may optionally perform pre-deduplication processing on the initial data files in the to-be-processed table based on a preset joint primary key to obtain at least one target data file, including:

[0012] Parsing the preset joint primary key to determine each primary key included in the preset joint primary key;

[0013] In the Map phase, the initial data files corresponding to each primary key are determined, wherein each primary key corresponds to at least one initial data file;

[0014] In the Reduce phase, a target data file corresponding to each primary key is selected from each initial data file corresponding to each primary key.

[0015] The above method may optionally select a target data file corresponding to each primary key from each initial data file corresponding to each primary key in the Reduce phase, including:

[0016] Traversing each of the initial data files to obtain each corresponding item of each of the initial data files;

[0017] The initial data file containing the most complete data in each corresponding item is used as the corresponding target data file.

[0018] The above method may optionally include determining the number of tables to be divided, and creating a table division path based on the number of tables to be divided, including:

[0019] Acquire table partitioning parameters, and determine the number of tables to be partitioned based on the table partitioning parameters;

[0020] Create a table partition path in HDFS based on the number of tables to be partitioned.

[0021] The above method optionally processes the target data files contained therein in parallel based on the MapReduce interface in each sub-table path to obtain at least one sub-table result, including:

[0022] Assign a MapReduce interface to each partition path;

[0023] Based on the MapReduce interface, the target data files contained in each sharding path are processed in parallel to obtain at least one sharding result.

[0024] A MapReduce memory overflow solution device, comprising:

[0025] A deduplication module, used for pre-deduplication processing of the initial data files in the to-be-processed table based on a preset joint primary key, to obtain at least one target data file;

[0026] A determination and creation module, used to determine the number of tables to be divided, and create a table division path based on the number of tables to be divided;

[0027] A copy module, used for copying the corresponding target data file to the corresponding table partition path based on the number of tables to be partitioned and the preset table partition logic;

[0028] A processing module is used to process the target data files contained in each sub-table path in parallel based on the MapReduce interface to obtain at least one sub-table result;

[0029] The merging module is used to merge the sub-table results to obtain the target sub-table results.

[0030] In the above device, optionally, the deduplication module includes:

[0031] A first determining unit, configured to parse the preset joint primary key and determine each primary key included in the preset joint primary key;

[0032] A second determining unit is used to determine the initial data files corresponding to each primary key in the Map phase, wherein each primary key corresponds to at least one initial data file;

[0033] The selection unit is used to select a target data file corresponding to each primary key from each initial data file corresponding to each primary key in the Reduce phase.

[0034] In the above device, optionally, the selection unit includes:

[0035] An acquisition subunit, used for traversing each of the initial data files and acquiring each corresponding item of each of the initial data files;

[0036] The determination subunit is used to take the initial data file containing the most complete data in each corresponding item as the corresponding target data file.

[0037] In the above device, optionally, the determining and creating module includes:

[0038] A third determining unit, configured to obtain table partitioning parameters and determine the number of tables to be partitioned based on the table partitioning parameters;

[0039] The creation unit is used to create a table partition path in HDFS based on the number of tables to be partitioned.

[0040] In the above device, optionally, the processing module includes:

[0041] The allocation unit is used to allocate a MapReduce interface to each partition path;

[0042] The processing unit is used to process the target data files contained in each sharding path in parallel based on the MapReduce interface to obtain at least one sharding result.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] The present invention discloses a MapReduce memory overflow solution and device, including: pre-deduplication processing of the initial data files in the processing table based on a preset joint primary key to obtain at least one target data file; determining the number of tables to be divided, and creating a table division path based on the number of tables to be divided; copying the corresponding target data files to the corresponding table division path based on the number of tables to be divided and the preset table division logic; processing the target data files contained therein in parallel based on the MapReduce interface under each table division path to obtain at least one table division result; merging the table division results to obtain the target table division result. In the above process, the initial data files in the processing table are first deduplicated, and further, the obtained target data is processed in parallel under the table division path. Compared with the prior art of calling HQL statements in HIVE to perform serial data processing to solve memory overflow, it not only reduces the amount of data files processed by MapReduce, but also adopts a parallel processing method to improve processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0046] Figure 1 A flowchart of a MapReduce memory overflow solution disclosed in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a processing flow of a MapReduce memory overflow solution disclosed in an embodiment of the present application;

[0048] Figure 3 This is a structural block diagram of a MapReduce memory overflow solution device disclosed in an embodiment of the present application. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

[0051] The present invention discloses a method and device for solving MapReduce memory overflow, which is applied in the process of preventing MapReduce memory overflow, wherein MapReduce is a distributed computing model, which is mainly used to solve the offline parallel computing problem of massive data. In the prior art, it is very common to query, process and process data based on MapReduce, which has the characteristics of large amount of data that can be processed and strong scalability. However, the surge in data volume is very common in practical applications, and the resulting data tilt problem is inevitable in the execution process of MapReduce, which is easy to cause serious memory overflow of Reduce and cannot run normally. If it is not optimized and solved in time, it will greatly affect the orderly operation of the system.

[0052] To solve the above-mentioned serious reduce memory overflow caused by data skew, there are currently several solutions in the industry:

[0053] Increase the running memory of the reduce program. However, this method has limitations because the overall memory is limited and may have unstable and uncertain effects on the running of other tasks in the cluster.

[0054] Increasing the number of reduce operations can effectively solve the data skew problem, but it comes at the cost of huge hardware investment.

[0055] Although converting the MapReduce interface program into HQL statements can complete data processing normally, the efficiency is extremely low and it is difficult to meet the enterprise's internal requirements for data processing timeliness.

[0056] Data memory overflow caused by data skew is very common. Data skew means that the amount of data allocated to one or several reducers is much higher than the average due to problems such as uneven key value distribution and business data characteristics, which greatly affects data processing efficiency.

[0057] Most existing methods rely on brute force solutions such as increasing program resources. However, as the amount of data accumulates, the above methods will inevitably have problems again, and are not scalable and applicable. Therefore, it is an urgent problem to propose a data skew solution that does not rely on a large amount of hardware investment, does not reduce the timeliness of data processing, and still has good performance as the amount of data accumulates. Therefore, the present invention provides a MapReduce memory overflow solution to solve the above problem. The execution process of the method is as follows: Figure 1 As shown, the steps include:

[0058] S101, performing pre-deduplication processing on the initial data files in the to-be-processed table based on a preset joint primary key to obtain at least one target data file;

[0059] In the embodiment of the present invention, the preset joint primary key is determined based on experience or specific application scenarios, and is not specifically limited in the embodiment of the present invention. Preferably, the preset joint primary key can be a multidimensional field, and the preset joint primary key is used for deduplication processing of the initial data file in the table to be processed, wherein the preset joint primary key contains at least one primary key, and preferably, the primary key is a specific field in the multidimensional field. Furthermore, because the preset joint primary key has the characteristics of low duplication and uniform distribution, as long as the preset joint primary key is selected properly, the key value (key value) in the Map stage during pre-deduplicate can be evenly distributed, thereby avoiding the occurrence of data skew and making the target data file after deduplication evenly distributed.

[0060] The table to be processed contains multiple initial data files, each of which has a primary key corresponding to it, and one primary key can correspond to multiple initial data files. The pre-deduplication processing process for MapReduce mainly includes the Map stage and the Reduce stage. In the Map stage, the initial data file corresponding to each primary key in the table to be processed is determined. The specific determination process can be traversed through the table to be processed, and the initial data file containing the corresponding primary key is associated with the primary key. Since there may be a situation in the Map stage where one primary key corresponds to multiple initial data files, in the Reduce stage, the situation in the Map stage where one primary key corresponds to multiple initial data files is deduplicated. The specific deduplication process is as follows: traverse the initial data files to obtain the corresponding items of the initial data files; and use the initial data file with the most complete data in the corresponding items as the target data file corresponding to it. Further, if there is only one initial data file corresponding to the current primary key, then the initial data file is the target data file corresponding to the current primary key.

[0061] S102, determining the number of tables to be divided, and creating a table division path based on the number of tables to be divided;

[0062] In the embodiment of the present invention, since the data volume and file size of the target data files output in the Reduce stage are basically the same, after determining the target data files corresponding to each primary key, the table partitioning parameters are obtained, wherein the table partitioning parameters are based on the underlying space parameters of the server. In the embodiment of the present invention, the specific values ​​of the table partitioning parameters are not limited. Furthermore, the number of target data files generated on the HDFS is automatically identified in units of file dimensions, the data volume of the target data files is obtained, and the ratio of the data volume to the table partitioning parameters is used as the number of tables to be partitioned, and then the corresponding number of table partitioning paths are created in sequence.

[0063] S103, based on the number of tables to be divided and the preset table division logic, copy the corresponding target data file to the corresponding table division path;

[0064] In the embodiment of the present invention, based on the number of tables to be divided and the preset table division logic, the corresponding target data files are copied to the table division path corresponding to each table, that is, the HDFS path, wherein the preset table division logic is limited based on the specific application scenario, and is not specifically limited in the embodiment of the present invention, that is, the adaptive table division logic design of automatic calculation of the number of tables and automatic copying of table division data is completed, and the user only needs to specify the table file size under each table division path. The above process has the advantages of strong applicability, application interface, and easy expansion.

[0065] S104, processing the target data files contained in each sub-table path in parallel based on the MapReduce interface to obtain at least one sub-table result;

[0066] In an embodiment of the present invention, the target data files under the corresponding sub-table paths are processed in parallel under the sub-table paths based on the MapReduce interface, wherein the specific processing method is related to the corresponding processing program in the MapReduce interface, and the processing program varies according to different specific application scenarios. In an embodiment of the present invention, the processing program is not limited. After the processing is completed, there is at least one sub-table result for each sub-table path, wherein the number of the sub-table results is related to the number of target data files stored under the sub-table path.

[0067] S105: Merge the sub-table results to obtain target sub-table results.

[0068] In an embodiment of the present invention, it is necessary to merge the results of each sub-table. Preferably, in order to avoid the sub-table results being overwritten due to the same name, before merging the sub-table results, it is necessary to rename the sub-table results to ensure that the name of the sub-table result is unique, copy all the renamed sub-table results to a specified location, complete the merging of the sub-table results, and obtain the target sub-table result, wherein the specified location can be specified based on experience or a specific application scenario. In an embodiment of the present invention, the specified location is not specifically limited.

[0069] The present invention discloses a solution to MapReduce memory overflow, including: pre-deduplication processing of the initial data files in the table to be processed based on a preset joint primary key to obtain at least one target data file; determining the number of tables to be divided, and creating a table division path based on the number of tables to be divided; copying the corresponding target data files to the corresponding table division path based on the number of tables to be divided and the preset table division logic; processing the target data files contained therein in parallel based on the MapReduce interface under each table division path to obtain at least one table division result; merging the table division results to obtain the target table division result. In the above process, the initial data files in the table to be processed are first deduplicated, and further, the obtained target data is processed in parallel under the table division path. Compared with the prior art of calling HQL statements in HIVE to perform serial data processing to solve memory overflow, it not only reduces the amount of data files processed by MapReduce, but also adopts a parallel processing method to improve processing efficiency.

[0070] In the embodiment of the present invention, the overall processing flow diagram for the above process is as follows: Figure 2As shown in the figure, the data to be processed must go through the following steps before processing: pre-deduplication, adaptive automatic table partitioning, MapReduce interface, and table partitioning merging interface. Through the pre-deduplication interface, the data after deduplication can be basically evenly distributed according to the set multi-dimensional deduplication primary key. Then, the number of tables is automatically determined according to the data volume based on the file dimension to complete the table partitioning. Then, table partition 1, table partition 2...table partition n are sent to the MapReduce interface in turn for data processing. After the processing is completed, the table partitioning results of each table are merged to complete the core processing logic.

[0071] In practical applications, data is always gradually accumulated over time. Whether adding cluster nodes to increase memory or converting MapReduce interfaces to HQL statements, there are problems of instability, uncertainty and low data processing efficiency. The present invention can automatically adjust the number of sub-tables according to the increase in data volume, overcome the defects of traditional methods, have more robustness and scalability, realize the decoupling of MapReduce memory requirements and data volume, analyze and solve the problem of data skew from the essence, solve the problem of MapReduce memory overflow caused by the surge in data volume without reducing processing efficiency and without increasing hardware resources, and has internal scalability.

[0072] Compared with other existing technologies, the advantages of the present invention are embodied in the following aspects:

[0073] Strong adaptability. The technical method proposed in the present invention allows users to set the table size parameter before the system is running. The parameter can be adjusted adaptively according to the memory size occupied by the table file. The larger the memory occupied, the smaller the table size, thereby realizing automatic table partitioning, which can ensure that each small table after partitioning can be processed normally and smoothly through the MapReduce core interface, and realize the decoupling of the data volume of the table to be processed and the memory required by the MapReduce interface program.

[0074] Good promotional properties. The present invention does not need to increase the number of Hadoop cluster nodes or the number of hardware resources to expand memory, and has the possibility of application to systems with limited hardware resources; even if the amount of data continues to accumulate, data processing can still be completed without adding any hardware resources, and has good applicability to small clusters.

[0075] Strong robustness. The traditional method of achieving load balancing by expanding memory or adding complex algorithms can successfully complete MapReduce processing in the short term, but the processing efficiency will continue to be affected and it does not have long-term stability. The present invention can achieve long-term and effective operation of the system by adding any complex algorithms without losing processing efficiency.

[0076] The present invention solves the contradiction between the low efficiency of HQL statements and the easy memory overflow of MapReduce interface, analyzes the essence of data tilt from the essence, black-boxes the MapReduce interface, solves the data tilt based on the pre-deduplication of the table to be processed and the adaptive automatic table partitioning operation, and each sub-table is spliced ​​in turn through the MapReduce interface black box, so as to solve the problem of memory overflow. The present invention does not need to add any hardware resources, and the number of sub-tables can be automatically adjusted based on the size of the data volume to ensure that each sub-table passing through the MapReduce interface black box can complete the processing normally, has extremely strong robustness and adaptability, can be applied to Hadoop clusters of various sizes, and has good scalability. The present invention has very good adaptability even when data is continuously accumulated, can automatically adjust the number of sub-tables according to the size of the data magnitude, and basically ensures that there will be no memory overflow under the premise of ensuring data processing efficiency, and has good universality.

[0077] Based on the above-mentioned MapReduce memory overflow solution, an embodiment of the present invention further provides a MapReduce memory overflow solution device. The structural block diagram of the device is as follows: Figure 3 As shown, including:

[0078] Deduplication module 201 , determination and creation module 202 , copy module 203 , processing module 204 and merging module 205 .

[0079] in,

[0080] The deduplication module 201 is used to perform pre-deduplication processing 202 on the initial data files in the to-be-processed table based on a preset joint primary key to obtain at least one target data file;

[0081] The determination and creation module is used to determine the number of tables to be divided and create a table division path based on the number of tables to be divided;

[0082] The copy module 203 is used to copy the corresponding target data file to the corresponding table partition path based on the number of tables to be partitioned and the preset table partition logic;

[0083] The processing module 204 is used to process the target data files contained in each table partition path in parallel based on the MapReduce interface to obtain at least one table partition result;

[0084] The merging module 205 is used to merge the sub-table results to obtain target sub-table results.

[0085] The present invention discloses a MapReduce memory overflow solution device, including: pre-deduplication processing of the initial data files in the processing table based on a preset joint primary key to obtain at least one target data file; determining the number of tables to be divided, and creating a table division path based on the number of tables to be divided; copying the corresponding target data files to the corresponding table division path based on the number of tables to be divided and the preset table division logic; processing the target data files contained therein in parallel based on the MapReduce interface under each table division path to obtain at least one table division result; merging the table division results to obtain the target table division result. In the above process, the initial data files in the processing table are first deduplicated, and further, the obtained target data is processed in parallel under the table division path. Compared with the prior art of calling HQL statements in HIVE to perform serial data processing to solve memory overflow, it not only reduces the amount of data files processed by MapReduce, but also adopts a parallel processing method to improve processing efficiency.

[0086] In the embodiment of the present invention, the deduplication module 201 includes:

[0087] A first determining unit 206 , a second determining unit 207 and a selecting unit 208 .

[0088] in,

[0089] The first determining unit 206 is used to parse the preset joint primary key and determine each primary key included in the preset joint primary key;

[0090] The second determining unit 207 is used to determine the initial data files corresponding to each primary key in the Map phase, wherein each primary key corresponds to at least one initial data file;

[0091] The selection unit 208 is used to select a target data file corresponding to each primary key from each initial data file corresponding to each primary key in the Reduce phase.

[0092] In the embodiment of the present invention, the selection unit 208 includes:

[0093] an acquisition subunit 209 and a determination subunit 210 .

[0094] in,

[0095] The acquisition subunit 209 is used to traverse the initial data files and acquire the corresponding items of the initial data files;

[0096] The determining subunit 210 is used to take the initial data file containing the most complete data in each corresponding item as the corresponding target data file.

[0097] In the embodiment of the present invention, the determining and creating module 202 includes:

[0098] The third determining unit 211 and the creating unit 212 .

[0099] in,

[0100] The third determining unit 211 is used to obtain table partitioning parameters and determine the number of tables to be partitioned based on the table partitioning parameters;

[0101] The creating unit 212 is used to create a table partitioning path in HDFS based on the number of tables to be partitioned.

[0102] In the embodiment of the present invention, the processing module 204 includes:

[0103] Allocating unit 213 and processing unit 214 .

[0104] in,

[0105] The allocation unit 213 is used to allocate a MapReduce interface to each partition path;

[0106] The processing unit 214 is used to process the target data files included in each table partition path in parallel based on the MapReduce interface to obtain at least one table partition result.

[0107] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0108] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0109] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0110] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0111] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0112] The memory may include non-permanent memory in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0113] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0114] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0115] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0116] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A solution to MapReduce memory overflow, characterized in that: include: Performing pre-deduplication processing on the initial data files in the processing table based on a preset joint primary key to obtain at least one target data file, wherein the preset joint primary key is a multidimensional field, and the preset joint primary key has the characteristics of low duplication and uniform distribution. The preset joint primary key can be properly selected to make the key value distribution of the Map stage during pre-deduplication uniform, so as to avoid the occurrence of data skew, and make the target data files after deduplication uniformly distributed; Determine the number of tables to be divided, and create a table division path based on the number of tables to be divided; Based on the number of tables to be divided and the preset table division logic, the corresponding target data file is copied to the corresponding table division path; Under each sharding path, the target data files contained therein are processed in parallel based on the MapReduce interface to obtain at least one sharding result; The sub-table results are merged to obtain the target sub-table results.

2. The method according to claim 1, characterized in that Performing pre-deduplication processing on the initial data files in the to-be-processed table based on the preset joint primary key to obtain at least one target data file, including: Parsing the preset joint primary key to determine each primary key included in the preset joint primary key; In the Map phase, the initial data files corresponding to each primary key are determined, wherein each primary key corresponds to at least one initial data file; In the Reduce phase, a target data file corresponding to each primary key is selected from each initial data file corresponding to each primary key.

3. The method according to claim 2, characterized in that In the Reduce phase, the target data files corresponding to each primary key are selected from the initial data files corresponding to each primary key, including: Traversing each of the initial data files to obtain each corresponding item of each of the initial data files; The initial data file containing the most complete data in each corresponding item is used as the corresponding target data file.

4. The method according to claim 1, characterized in that Determining the number of tables to be divided, and creating a table division path based on the number of tables to be divided, including: Acquire table partitioning parameters, and determine the number of tables to be partitioned based on the table partitioning parameters; Create a table partition path in HDFS based on the number of tables to be partitioned.

5. The method according to claim 1, characterized in that In each partition path, the target data files contained therein are processed in parallel based on the MapReduce interface to obtain at least one partition result, including: Assign a MapReduce interface to each partition path; Based on the MapReduce interface, the target data files contained in each sharding path are processed in parallel to obtain at least one sharding result.

6. A MapReduce memory overflow solution, characterized in that: include: A deduplication module is used to perform pre-deduplication processing on the initial data files in the processing table based on a preset joint primary key to obtain at least one target data file. The preset joint primary key is a multidimensional field. The preset joint primary key has the characteristics of low duplication and uniform distribution. The preset joint primary key is selected appropriately to make the key value distribution of the Map stage during pre-deduplication uniform, so as to avoid the occurrence of data skew, and make the target data files after deduplication uniformly distributed; A determination and creation module, used to determine the number of tables to be divided, and create a table division path based on the number of tables to be divided; A copy module, used for copying the corresponding target data file to the corresponding table partition path based on the number of tables to be partitioned and the preset table partition logic; A processing module is used to process the target data files contained in each sub-table path in parallel based on the MapReduce interface to obtain at least one sub-table result; The merging module is used to merge the sub-table results to obtain the target sub-table results.

7. The device according to claim 6, characterized in that The deduplication module comprises: A first determining unit, configured to parse the preset joint primary key and determine each primary key included in the preset joint primary key; A second determining unit is used to determine the initial data files corresponding to each primary key in the Map phase, wherein each primary key corresponds to at least one initial data file; The selection unit is used to select a target data file corresponding to each primary key from each initial data file corresponding to each primary key in the Reduce phase.

8. The device according to claim 7, characterized in that The selection unit comprises: An acquisition subunit, used for traversing each of the initial data files and acquiring each corresponding item of each of the initial data files; The determination subunit is used to take the initial data file containing the most complete data in each corresponding item as the corresponding target data file.

9. The device according to claim 6, characterized in that The determination and creation module includes: A third determining unit, configured to obtain table partitioning parameters and determine the number of tables to be partitioned based on the table partitioning parameters; The creation unit is used to create a table partition path in HDFS based on the number of tables to be partitioned.

10. The device according to claim 6, characterized in that The processing module comprises: The allocation unit is used to allocate a MapReduce interface to each partition path; The processing unit is used to process the target data files contained in each sharding path in parallel based on the MapReduce interface to obtain at least one sharding result.

Images