MapReduce task execution method and device, electronic equipment and storage medium

Abstract

The invention provides a MapReduce task execution method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining the current load information corresponding to each current partition, and the current load information comprises the current data size and the current load condition; under the condition that it is determined that multiple target low-load partitions exist according to the current load information corresponding to each current partition, the multiple target low-load partitions are combined, and a combined partition is obtained; and executing processing of the data in the merged partition through a target task to obtain a processed result. According to the method and the device, a plurality of target low-load partitions are combined into the combined partition, and processing of the data in the combined partition is executed through one target task, so that the purpose of improving the load of a single task can be achieved, and the technical effect of improving the resource utilization efficiency is achieved; therefore, the problems of poor static partition performance and low resource utilization efficiency of the MapReduce framework in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the technical field of distributed data processing, and particularly to a method and apparatus for task execution of MapReduce, an electronic device, and a storage medium. Background Art

[0002] In the field of big data processing, MapReduce, as an important distributed computing framework, is widely used in scenarios such as data analysis and machine learning. The MapReduce framework accelerates task processing by allocating data partitions to multiple computing nodes. A partition is the basic unit of task parallelization in distributed computing. By dividing a large-scale data set into several subsets, each subset is called a partition. However, with the continuous growth of data scale, the MapReduce framework has encountered bottlenecks in terms of partition and resource utilization efficiency. The partition method of existing MapReduce solutions is set before task execution, so the data processing speed is easily limited by the partition structure of fixed static partitions.

[0003] Static partition configuration cannot comprehensively predict the data distribution before execution. In scenarios where the data scale and distribution characteristics are uncertain, its performance is often poor. In addition, during execution, some partitions may have lighter tasks, while some partitions have heavier tasks, resulting in low resource allocation and utilization efficiency.

[0004] It can be seen that the MapReduce framework in related technologies has problems of poor static partition performance and low resource utilization efficiency. Summary of the Invention

[0005] The present application provides a method and apparatus for task execution of MapReduce, an electronic device, and a storage medium, so as to at least solve the problems of poor static partition performance and low resource utilization efficiency in the MapReduce framework in related technologies.

[0006] According to one aspect of the embodiments of the present application, a method for task execution of MapReduce is provided, including:

[0007] Obtaining current load information corresponding to each current partition, where the current load information includes: current data volume and current load condition;

[0008] When it is determined, according to the current load information corresponding to each current partition, that there are multiple target low-load partitions, merging the multiple target low-load partitions to obtain a merged partition;

[0009] Processing the data in the merged partition through a target task to obtain a processed result.

[0010] Optionally, as in the foregoing method, determining that there are multiple target low-load partitions according to the current load information corresponding to each current partition includes:

[0011] Determining candidate low-load partitions among all current partitions according to the current load information corresponding to each current partition, where the load of the candidate low-load partitions is less than a preset ratio of the other current partitions except the candidate low-load partitions among all current partitions;

[0012] Determining the multiple target low-load partitions among the candidate low-load partitions.

[0013] Optionally, as in the foregoing method, determining the multiple target low-load partitions among the candidate low-load partitions includes:

[0014] Determining the multiple target low-load partitions among the multiple candidate low-load partitions according to the data volume of each candidate low-load partition and the physical location relationship among the candidate low-load partitions.

[0015] Optionally, as in the foregoing method, before processing the data in the merged partition by one target task, the method further includes:

[0016] Merging the data in the merged partition to obtain merged data;

[0017] Determining a partition number corresponding to each latest partition according to the number of all latest partitions after generating the merged partition, where all the latest partitions include the merged partition and the partitions in all current partitions that have not been merged.

[0018] Optionally, as in the foregoing method, the method further includes:

[0019] Obtaining real-time load information corresponding to each processed partition, where each processed result is one of the processed partitions;

[0020] In the case of determining that there are multiple target low-load processed partitions according to the real-time load information corresponding to each processed partition, merging the multiple target low-load processed partitions to obtain a merged processed partition;

[0021] Processing the data in the merged processed partition through a specified task to obtain a latest processed result.

[0022] Optionally, as in the foregoing method, the method further includes:

[0023] When it is determined, according to the current load information corresponding to each current partition, that there is a target high-load partition, at least two split partitions are obtained after splitting the target high-load partition;

[0024] The data in each split partition is processed by a specified task corresponding to each split partition to obtain a split processing result.

[0025] Optionally, as in the foregoing method, the determining that there is a target high-load partition according to the current load information corresponding to each current partition includes:

[0026] According to the current load information corresponding to each current partition, it is judged whether there is a candidate high-load partition in all the current partitions whose load is higher than a target ratio of other current partitions except itself;

[0027] If there is the candidate high-load partition, the candidate high-load partition is determined as the target high-load partition.

[0028] According to another aspect of the embodiments of the present application, there is also provided a task execution device for MapReduce, including:

[0029] An acquisition module, configured to acquire current load information corresponding to each current partition, where the current load information includes: current data volume and current load condition;

[0030] A merging module, configured to merge the multiple target low-load partitions to obtain a merged partition when it is determined, according to the current load information corresponding to each current partition, that there are multiple target low-load partitions;

[0031] A task execution module, configured to process the data in the merged partition by a target task to obtain a processed result.

[0032] According to still another aspect of the embodiments of the present application, there is also provided an electronic device, including a processor, a communication interface, a memory, and a communication bus, where the processor, the communication interface, and the memory complete communication with each other through the communication bus; where the memory is used to store a computer program; the processor is configured to execute the method steps in any of the foregoing embodiments by running the computer program stored on the memory.

[0033] According to still another aspect of the embodiments of the present application, there is also provided a computer-readable storage medium, where a computer program is stored in the storage medium, and the computer program is set to execute the method steps in any of the foregoing embodiments when running.

[0034] In the embodiments of the present application, a method of merging partitions with low load is adopted. By obtaining the current load information corresponding to each current partition, where the current load information includes: the current data volume and the current load condition; when it is determined, according to the current load information corresponding to each current partition, that there are multiple target low-load partitions, the multiple target low-load partitions are merged to obtain a merged partition; and a target task is used to process the data in the merged partition to obtain a processed result. Since multiple target low-load partitions are merged to obtain a merged partition, and a target task is used to process the data in the merged partition, the purpose of increasing the load of a single task can be achieved, and the technical effect of improving the resource utilization efficiency is achieved. Furthermore, the problems in the related art that the MapReduce framework has poor performance of static partitions and low resource utilization efficiency are solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, showing embodiments consistent with the present application, and are used together with the specification to explain the principles of the present application.

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0037] Figure 1 is a schematic diagram of the hardware environment of an optional MapReduce task execution method according to an embodiment of the present application;

[0038] Figure 2 is a schematic flowchart of an optional MapReduce task execution method according to an embodiment of the present application;

[0039] Figure 3 is a schematic flowchart of an optional MapReduce task execution method according to another embodiment of the present application;

[0040] Figure 4 is a schematic flowchart of an optional MapReduce task execution method according to another embodiment of the present application;

[0041] Figure 5 is a schematic diagram before and after partition merging in an optional MapReduce task execution method according to another embodiment of the present application;

[0042] Figure 6It is a schematic flowchart of an optional MapReduce task execution method according to another embodiment of the present application;

[0043] Figure 7 It is a schematic flowchart of an optional MapReduce task execution method according to an application example of the present application;

[0044] Figure 8 It is a structural block diagram of an optional MapReduce task execution method device according to an embodiment of the present application;

[0045] Figure 9 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. Detailed implementation manners

[0046] In order to enable those skilled in the art to better understand the solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0047] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to be used to describe a specific order or sequence. It should be understood that such used data may be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily have to be limited to those clearly listed steps or units, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0048] First, some nouns or terms that appear during the description of the embodiments of the present application are applicable to the following explanations:

[0049] 1. MapReduce (MR task): A distributed computing framework that decomposes big data processing tasks into a Map stage and a Reduce stage to achieve efficient parallel processing of large-scale data.

[0050] 2. Partition: Divide a large-scale data set into several subsets, and each subset is called a partition. A partition is the basic unit of task parallelization in distributed computing.

[0051] According to one aspect of the embodiments of the present application, a method for executing MapReduce tasks is provided. Optionally, in this embodiment, the above-mentioned method for executing MapReduce tasks may be applied to, for example, Figure 1 the hardware environment composed of the terminal 1402 and the server 1404 as shown. As Figure 1 shown, the server 1404 is connected to the terminal 1402 through a network, and can be used to provide services (such as game services, application services, etc.) for the terminal or the client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 1404.

[0052] The above network may include, but is not limited to, at least one of the following: wired network, wireless network. The above wired network may include, but is not limited to, at least one of the following: wide area network, metropolitan area network, local area network. The above wireless network may include, but is not limited to, at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal is not limited to being a PC, mobile phone, tablet computer, etc.

[0053] The method for executing MapReduce tasks in the embodiments of the present application may be executed by the server, or may be executed by the terminal, or may be jointly executed by the server and the terminal. Among them, when the terminal executes the method for executing MapReduce tasks in the embodiments of the present application, it may also be executed by the client installed on it.

[0054] Taking the method for executing MapReduce tasks in this embodiment executed by the server as an example, Figure 2 A method for executing MapReduce tasks provided by the embodiments of the present application includes the following steps:

[0055] Step S202, obtain the current load information corresponding to each current partition, where the current load information includes: current data volume and current load condition.

[0056] The method for executing MapReduce tasks in this embodiment may be applied to scenarios of parallel processing of large-scale data under the MapReduce framework during distributed data processing, such as: scenarios of data analysis under MapReduce, scenarios of machine learning under MapReduce, etc., or other data processing scenarios under MapReduce.

[0057] Specifically, during the execution of a MapReduce job, real-time load information of partitions is obtained by monitoring the current data volume and current load conditions (such as computing time, memory usage, etc.) of each current partition in real time. Further, the current load information can be obtained through a preset monitoring framework and tools. For example, log analysis of monitoring tools in the Hadoop ecosystem (such as Apache Ambari, Ganglia, etc.). The current load information can also be obtained by implementing data collection, such as through JMX (Java Management Extensions), operating system-level monitoring, etc.

[0058] Step S204, in the case where it is determined, based on the current load information corresponding to each current partition, that there are multiple target low-load partitions, the multiple target low-load partitions are merged to obtain a merged partition.

[0059] Specifically, after determining the current load information corresponding to each current partition, the target low-load partitions can be determined among all the current partitions according to all the current load information. The multiple target low-load partitions can be the N partitions with the lowest load among all the current partitions and meeting the preset merging requirements. After determining the multiple target low-load partitions, the multiple target low-load partitions can be merged to obtain a merged partition. Further, after executing this step S204 once, it can also be determined, based on the obtained merged partition, whether there are partitions that can be further merged into another merged partition. If so, they are merged, and this step S204 is continuously looped until there are no partitions that can be further merged.

[0060] Step S206, a target task is used to process the data in the merged partition to obtain a processed result.

[0061] Specifically, after obtaining the merged partition, a target task can be used to process the data in the merged partition and obtain a processed result.

[0062] In the embodiments of the present application, a method of merging partitions with lower loads is adopted. By obtaining the current load information corresponding to each current partition, where the current load information includes: the current data volume and the current load situation; when it is determined, based on the current load information corresponding to each current partition, that there are multiple target low-load partitions, the multiple target low-load partitions are merged to obtain a merged partition; and a target task is used to process the data in the merged partition to obtain a processed result. Since the multiple target low-load partitions are merged to obtain a merged partition, and a target task is used to process the data in the merged partition, the purpose of increasing the load of a single task can be achieved, and the technical effect of improving the resource utilization efficiency is achieved. Furthermore, the problems in the related art, such as the poor performance of static partitions and low resource utilization efficiency in the MapReduce framework, are solved.

[0063] As Figure 3 shown, as an optional embodiment, for the foregoing method, the foregoing step S204 of determining that there are multiple target low-load partitions based on the current load information corresponding to each current partition may be implemented through the following step S302 and step S304:

[0064] Step S302: Based on the current load information corresponding to each current partition, candidate low-load partitions are determined among all the current partitions, where the load of the candidate low-load partitions is less than a preset ratio of the loads of the other current partitions among all the current partitions except the candidate low-load partitions.

[0065] That is to say, after the current load information corresponding to each current partition is determined, candidate low-load partitions can be determined among all the current partitions according to all the current load information. For example, when it is determined based on all the current load information that the loads corresponding to some partitions are lower than a preset ratio (for example, 10%) of the loads of other current partitions, the dynamic partition merging mechanism is triggered, and these partitions are determined as candidate low-load partitions.

[0066] Step S304: Multiple target low-load partitions are determined among the candidate low-load partitions.

[0067] That is to say, after the candidate low-load partitions are determined, further screening is required among the candidate low-load partitions to determine multiple target low-load partitions that can be merged into one partition. Optionally, the multiple target low-load partitions may meet at least one of the following conditions: the total load of the multiple target low-load partitions is approximately the same as the load of any one of the other current partitions (for example, the deviation is within a preset range, such as 1%, 2%, etc.), and the total load of the multiple target low-load partitions does not exceed a preset load upper limit, etc.

[0068] As an alternative embodiment, for the method as described above, the multiple target low-load partitions in step S304 can be determined in the candidate low-load partitions through the following steps: Determine the multiple target low-load partitions from the multiple candidate low-load partitions according to the data volume of each candidate low-load partition and the physical location relationship between the candidate low-load partitions.

[0069] That is to say, after obtaining the candidate low-load partitions, based on the data volume of each candidate low-load partition and the physical location relationship between the candidate low-load partitions, multiple target low-load partitions can be determined from the multiple candidate low-load partitions. For example, the total data volume of the multiple target low-load partitions needs to be approximately the same as the data volume of any one of the other current partitions (for example, the deviation is within a preset range, such as 1%, 2%, etc.), and the physical location relationship between the multiple target low-load partitions needs to satisfy that the physical locations are within a preset distance range, such as the same physical machine, the same region, etc. To reduce the network transmission overhead caused by merging.

[0070] As Figure 4 shown, as an alternative embodiment, for the method as described above, before processing the data in the merged partition through a target task, the method may further include the following steps S402 and S404:

[0071] Step S402: Merge the data in the merged partition to obtain merged data.

[0072] Specifically, the data in the file corresponding to each partition can be read and then written into the same file, so that the data in the same file is used as the merged data.

[0073] Step S404: Determine the partition number corresponding to each of the latest partitions according to the number of all the latest partitions after generating the merged partition, where all the latest partitions include the merged partition and the unmerged partitions among all the current partitions.

[0074] After obtaining the merged partition, the re-partition numbering can be performed on all the latest partitions after the merge. For example, as Figure 5 shown, when there are 6 current partitions with partition numbers Reduce1, Reduce2, Reduce3, Reduce4, Reduce5, and Reduce6 respectively, if the three current partitions with partition numbers Reduce3, Reduce4, and Reduce5 are merged, the partition number of the merged partition is Reduce3, the partition number corresponding to the original partition with the partition number Reduce6 is modified to Reduce4, and the partition numbers of the partitions with partition numbers Reduce1 and Reduce2 remain unchanged.

[0075] For example, the partition metadata of the task (i.e., partition information (i.e., data in the latest partition), partition number) will be updated to ensure that the Reduce stage of MapReduce can correctly identify the new partition. The specific operations include updating partition information, reassigning partition numbers, etc.

[0076] As an alternative embodiment, as the method described above, the method further includes the following steps: obtaining real-time load information corresponding to each processed partition, where each processed result is a processed partition; in the case where it is determined, according to the real-time load information corresponding to each processed partition, that there are multiple target low-load processed partitions, merging the multiple target low-load processed partitions to obtain a merged processed partition; processing the data in the merged processed partition through a specified task to obtain the latest processed result. That is to say, when a partition merge is performed once and the processed result corresponding to each latest partition is obtained, the processed result is used as a processed partition, and it is continued to determine whether there are multiple target low-load processed partitions in the processed partition. If there are multiple target low-load processed partitions, the multiple target low-load processed partitions can also be merged to obtain a merged processed partition; processing the data in the merged processed partition through a specified task to obtain the latest processed result. And, the stage for performing partition merge can be the Map stage or the Reduce stage. Thus, partition merge can be performed each time data is processed to ensure that the load is balanced among all partitions in different execution stages, ultimately achieving the purpose of improving execution efficiency and resource utilization.

[0077] As Figure 6 shown, as an alternative embodiment, as the method described above, the method further includes the following steps:

[0078] Step S602, in the case where it is determined, according to the current load information corresponding to each current partition, that there is a target high-load partition, obtaining at least two split partitions after splitting the target high-load partition.

[0079] Specifically, after determining the current load information corresponding to each current partition, it can be determined whether there is a target high-load partition among all current partitions according to all the current load information. The target high-load partition can be the partition with the highest load among all current partitions and meeting the preset splitting requirements (for example, the load is higher than the preset upper limit). After determining the target high-load partition, the target high-load partition can be split to obtain at least two split partitions.

[0080] Step S604: Process the data in each split partition through a specified task corresponding to each split partition to obtain a split processing result.

[0081] Specifically, after obtaining the split partitions, determine a specified task corresponding to each split partition, and then process the data in all split partitions through each specified task to obtain a split processing result corresponding to each split partition.

[0082] Through the method of this embodiment, by splitting high-load partitions, the execution efficiency can be improved, and the problem of low processing efficiency caused by excessive load in a single partition can be avoided.

[0083] As an optional embodiment, for the method as described above, step S602 of determining the existence of target high-load partitions according to the current load information corresponding to each current partition can be implemented through the following steps: According to the current load information corresponding to each current partition, determine whether there is a candidate high-load partition in all current partitions whose load is higher than a target ratio of other current partitions except itself; if there is a candidate high-load partition, determine the candidate high-load partition as the target high-load partition. That is to say, after determining the current load information corresponding to each current partition, according to all the current load information, determine the candidate high-load partitions among all current partitions. For example, when it is determined based on all the current load information that the load of some partitions is higher than a preset target ratio (e.g., 10%) of other current partitions, the dynamic partition splitting mechanism is triggered, and these partitions are determined as candidate high-load partitions. The candidate high-load partitions can be directly determined as the target high-load partitions, or those with a load exceeding the preset upper limit among the candidate high-load partitions can be determined as the target high-load partitions.

[0084] As Figure 7 shown, provide an application example applying any of the foregoing embodiments:

[0085] 1. Task execution monitoring: During the execution of a MapReduce job, obtain the current load information of partitions by monitoring the data volume and load conditions (including computing time, memory usage, etc.) of each partition in real time.

[0086] 2. Trigger condition judgment: Dynamically judge whether all current partitions need to be merged according to the current load information obtained by monitoring. When it is detected that the load of some current partitions exceeds the set threshold (such as the data volume is 10% larger than other partitions), it is determined that the load of other partitions may be too low, and the dynamic partition merging mechanism will be automatically triggered.

[0087] 3. Partition merging strategy: Based on the partition load situation, select several target low-load partitions with lower loads for merging to form a new merged partition. During the merging process, prioritize low-load partitions with smaller data volumes and physical proximity to reduce the network transmission overhead caused by merging.

[0088] 4. Partition metadata update: After the system merges partitions, it updates the partition metadata of the task (i.e., partition information (i.e., data in the latest partition), partition number) to ensure that the Reduce stage of MapReduce can correctly identify the new partition. Specific operations include updating partition information, reassigning partition numbers, etc.

[0089] 5. Repeated execution: During different stages of task execution (e.g., different Map stages or different Reduce stages), dynamic partition merging operations can be performed multiple times to ensure that the loads of each partition in different execution stages remain balanced in different execution stages, ultimately achieving the purpose of improving execution efficiency and resource utilization.

[0090] It should be noted that for the foregoing method embodiments, for the sake of simple description, they are all expressed as a series of action combinations. However, those skilled in the art should know that this application is not limited by the described action sequence because according to this application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0091] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases, the former is a better implementation method. Based on such an understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM (Read-Only Memory), RAM (Random Access Memory), magnetic disk, optical disc), and includes several instructions to enable a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in various embodiments of this application.

[0092] According to another aspect of the embodiments of this application, there is also provided a MapReduce task execution device for implementing the above MapReduce task execution method. Figure 8 It is a structural block diagram of an optional MapReduce task execution device according to the embodiments of this application, asFigure 8 As shown, the device may include:

[0093] An acquisition module 81, configured to acquire current load information corresponding to each current partition, where the current load information includes: current data volume and current load condition;

[0094] A merging module 82, configured to merge multiple target low-load partitions to obtain a merged partition when it is determined, according to the current load information corresponding to each current partition, that there are multiple target low-load partitions;

[0095] A task execution module 83, configured to process the data in the merged partition through a target task to obtain a processed result.

[0096] It should be noted that the acquisition module 81 in this embodiment may be used to execute the above-mentioned step S202, the merging module 82 in this embodiment may be used to execute the above-mentioned step S204, and the task execution module 83 in this embodiment may be used to execute the above-mentioned step S206.

[0097] Through the above modules, by adopting the method of merging partitions with lower loads, by acquiring the current load information corresponding to each current partition, where the current load information includes: current data volume and current load condition; when it is determined, according to the current load information corresponding to each current partition, that there are multiple target low-load partitions, merging the multiple target low-load partitions to obtain a merged partition; processing the data in the merged partition through a target task to obtain a processed result. Since multiple target low-load partitions are merged to obtain a merged partition, and the data in the merged partition is processed through a target task, the purpose of improving the load of a single task can be achieved, and the technical effect of improving resource utilization efficiency is achieved, thereby solving the problems of poor performance of static partitions and low resource utilization efficiency existing in the related MapReduce framework.

[0098] The device in this embodiment, in addition to including the above modules, may further include modules for executing any method in the embodiments of the task execution method of MapReduce as described above.

[0099] It should be noted here that the examples and application scenarios implemented by the above modules and the corresponding steps are the same, but are not limited to the content disclosed in the above embodiments. It should be noted that the above modules, as part of the device, can run in a hardware environment as shown in Figure 1 and can be implemented by software or by hardware, where the hardware environment includes a network environment.

[0100] According to another aspect of the embodiments of the present application, there is also provided an electronic device for implementing the above-mentioned task execution method of MapReduce. The electronic device may be a server, a terminal, or a combination thereof.

[0101] According to another embodiment of the present application, there is also provided an electronic device, including: as Figure 9 shown, the electronic device may include: a processor 1501, a communication interface 1502, a memory 1503, and a communication bus 1504. Among them, the processor 1501, the communication interface 1502, and the memory 1503 communicate with each other through the communication bus 1504.

[0102] The memory 1503 is used to store a computer program;

[0103] The processor 1501, when executing the program stored on the memory 1503, realizes the following steps:

[0104] Step S202, obtaining current load information corresponding to each current partition. Among them, the current load information includes: current data volume and current load situation.

[0105] Step S204, when it is determined that there are multiple target low-load partitions according to the current load information corresponding to each current partition, merging the multiple target low-load partitions to obtain a merged partition.

[0106] Step S206, processing the data in the merged partition through a target task to obtain a processed result.

[0107] Optionally, in this embodiment, the above-mentioned communication bus may be a PCI (Peripheral Component Interconnect) bus, an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of simplicity, only a thick line is shown in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above-mentioned electronic device and other devices.

[0108] The memory may include a random access memory (RAM), and may also include a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located far from the aforementioned processor.

[0109] As an example, the above-mentioned memory 1503 may include, but is not limited to, the acquisition module 81, the merging module 92, and the task execution module 83 in the task execution device of the above-mentioned MapReduce. In addition, it may also include, but is not limited to, other module units in the task execution device of the above-mentioned MapReduce, which will not be elaborated in this example.

[0110] The above-mentioned processor may be a general-purpose processor, which may include, but is not limited to: CPU (Central Processing Unit, central processor), NP (Network Processor, network processor), etc.; it may also be a DSP (Digital Signal Processor, digital signal processor), ASIC (Application Specific Integrated Circuit, application-specific integrated circuit), FPGA (Field-Programmable Gate Array, field programmable gate array), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0111] The embodiment of the present application also provides a computer-readable storage medium, and the storage medium includes a stored program, wherein when the program runs, it executes the method steps of the above-mentioned method embodiment.

[0112] Optionally, in this embodiment, the above-mentioned storage medium may include, but is not limited to: various media such as USB flash drives, ROMs, RAMs, mobile hard disks, magnetic disks, or optical discs that can store program codes.

[0113] The serial numbers of the above-mentioned embodiments of the present application are only for description and do not represent the superiority or inferiority of the embodiments.

[0114] If the integrated unit in the above-mentioned embodiment is implemented in the form of a software functional unit and sold or used as an independent product, it may be stored in the above-mentioned computer-readable storage medium. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, may be embodied in the form of a software product. The computer software product is stored in the storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in various embodiments of the present application.

[0115] In the above-mentioned embodiments of the present application, the descriptions of each embodiment have their own emphases. For the parts not detailed in a certain embodiment, reference may be made to the relevant descriptions of other embodiments.

[0116] In several embodiments provided by the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are merely illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed coupling, direct coupling, or communication connection to each other can be through some interfaces. The indirect coupling or communication connection of units or modules can be in electrical or other forms.

[0117] The units described as separate components may or may not be physically separated. The components displayed as units may or may not be physical units, that is, they can be located in one place or distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0118] In addition, each functional unit in various embodiments of the present application can be integrated in a processing unit, or each unit can exist physically alone, or two or more units can be integrated in one unit. The above-mentioned integrated units can be implemented in the form of hardware or in the form of software functional units.

[0119] The above is only the preferred embodiment of the present application. It should be noted that for those of ordinary skill in the art, without departing from the principle of the present application, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present application.

Claims

1. A MapReduce task execution method, characterized in that: include: Obtaining current load information corresponding to each current partition, wherein the current load information includes: current data volume and current load condition; When it is determined that there are multiple target low-load partitions according to the current load information corresponding to each current partition, merging the multiple target low-load partitions to obtain a merged partition; The data in the merged partition is processed by a target task to obtain a processed result.

2. The method according to claim 1, characterized in that: The determining, based on the current load information corresponding to each current partition, that there are multiple target low-load partitions includes: Determine a candidate low-load partition among all current partitions according to the current load information corresponding to each current partition, wherein the load of the candidate low-load partition is less than a preset proportion of other current partitions among all current partitions except the candidate low-load partition; The plurality of target low-load partitions are determined from among the candidate low-load partitions.

3. The method according to claim 2, characterized in that The step of determining the plurality of target low-load partitions from the candidate low-load partitions comprises: The plurality of target low-load partitions are determined from the plurality of candidate low-load partitions according to the data volume of each candidate low-load partition and the physical position relationship between the candidate low-load partitions.

4. The method according to claim 1, characterized in that: Before executing the processing of the data in the merged partition by a target task, the method further includes: Merging the data in the merged partitions to obtain merged data; According to the number of all latest partitions after the merged partition is generated, a partition number corresponding to each latest partition is determined, wherein all the latest partitions include the merged partition and all partitions in the current partitions that have not been merged.

5. The method according to claim 1, characterized in that The method further comprises: Acquire real-time load information corresponding to each processed partition, wherein each processed result is a processed partition; When it is determined that there are multiple target low-load processed partitions according to the real-time load information corresponding to each processed partition, the multiple target low-load processed partitions are merged to obtain a merged processed partition; The data in the merged partition is processed by a designated task to obtain the latest processed result.

6. The method according to claim 1, characterized in that The method further comprises: When it is determined that there is a target high-load partition according to the current load information corresponding to each current partition, obtaining at least two split partitions after splitting the target high-load partition; The data in each split partition is processed by a designated task corresponding to each split partition to obtain a split processing result.

7. The method according to claim 6, characterized in that The determining, based on the current load information corresponding to each current partition, that there is a target high-load partition includes: According to the current load information corresponding to each current partition, determining whether there is a candidate high-load partition among all current partitions whose load is higher than the target ratio of other current partitions except the current partition itself; If the candidate high-load partition exists, the candidate high-load partition is determined as the target high-load partition.

8. A MapReduce task execution device, characterized in that: include: An acquisition module, used to acquire current load information corresponding to each current partition, wherein the current load information includes: current data volume and current load condition; a merging module, configured to merge the multiple target low-load partitions to obtain a merged partition when it is determined that there are multiple target low-load partitions according to the current load information corresponding to each current partition; The task execution module is used to execute the processing of the data in the merged partition through a target task to obtain the processed result.

9. An electronic device comprising a processor, a communication interface, a memory and a communication bus, wherein: The processor, the communication interface and the memory communicate with each other via the communication bus, wherein: The memory is used to store computer programs; The processor is configured to execute the method according to any one of claims 1 to 7 by running the computer program stored in the memory.

10. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 7 when executed.

Images