Performance optimization method and device of mapping protocol framework, equipment and medium

Abstract

The invention discloses a performance optimization method and device for a mapping protocol framework, equipment and a medium. The method comprises the following steps: carrying out equalization processing based on a pre-configured node parameter, and concurrently executing a mapping / specification task according to an equalization processing result; if the task is executed to the tail end of the mapping stage or the tail end of the data shuffling and sorting stage, dynamic balance processing is carried out on the disk input and output resource weight of the process where the task being executed is located through a completely fair queue; and if the task is executed to the tail end of the protocol stage, calculating and executing the corresponding distributed file system client write-in blocking delay for the process where the executed task is located. According to the technical scheme provided by the embodiment of the invention, complete fair queue weight dynamic balance and client write-in blocking are realized by using a cgroup technology, dynamic isolation of disk input and output resources of the mapping / protocol task is realized, and the resource utilization rate is improved.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a performance optimization method, device, equipment and medium for a mapping specification framework. Background Art

[0002] The mapping specification framework is one of the most mature big data computing frameworks. Currently, resource management and task scheduling services are commonly used to schedule and isolate resources for the mapping specification framework. However, resource management and task scheduling services only support coarse-grained isolation of the CPU and memory, and cannot effectively isolate resources for input and output devices. As a result, when the input and output resources are saturated, there will inevitably be large-scale data skew caused by input and output resource preemption. Summary of the invention

[0003] The present invention provides a performance optimization method, device, equipment and medium for a mapping / reduce framework, which realizes completely fair dynamic balancing of queue weights and client write blocking by using cgroup technology, realizes dynamic isolation of disk input and output resources of mapping / reduce tasks, and improves resource utilization.

[0004] According to one aspect of the present invention, a method for optimizing the performance of a mapping-reduce framework is provided, comprising:

[0005] Perform balancing based on pre-configured node parameters and concurrently execute mapping / reduction tasks based on the balancing results;

[0006] If the mapping / reducing task is executed to the end of the mapping phase or the end of the data shuffling and sorting phase, the disk input and output resource weights of the process where the mapping / reducing task is being executed are dynamically balanced through a completely fair queue;

[0007] If the mapping / reducing task is executed to the end of the reduction phase, the corresponding distributed file system client write blocking delay is calculated and executed for the process where the mapping / reducing task is being executed.

[0008] Optionally, before performing balancing processing based on pre-configured node parameters and concurrently executing mapping / reducing tasks according to the balancing processing result, the following is further included:

[0009] Set the block scheduling algorithm involving disks in the local directory parameters of the computing nodes in the resource management and task scheduling service to completely fair queues;

[0010] Configure the number of CPU cores for mapping tasks and reduction tasks;

[0011] Configure the number of CPU cores that can be allocated to computing nodes in the resource management and task scheduling services.

[0012] In order to achieve nearly simultaneous completion of all mapping / reduction tasks by adopting the above technical solution, it is necessary to configure the relevant parameters of the input and output resource scheduling of the device in advance.

[0013] Optionally, the performing balancing processing based on pre-configured node parameters includes:

[0014] Perform map / reduce task balance detection based on pre-configured node parameters;

[0015] If it is detected that the disk input and output resources are saturated and the CPU resources are idle, the number of concurrent tasks of the node is adjusted downward;

[0016] If it is detected that the disk I / O resources are idle and the CPU resources are saturated, the number of CPU cores for each task is adjusted upward;

[0017] If it is detected that the disk input and output resources are idle and the CPU resources are idle, the number of concurrent tasks of the node is adjusted upward.

[0018] By adopting the above technical solution, the pre-configured node parameters can be balanced and optimized according to the usage saturation of the central processing unit resources and disk input and output resources, thereby further improving the overall performance.

[0019] Optionally, the dynamically balancing the disk input and output resource weights of the process where the mapping / reducing task being executed is located by using a completely fair queue includes:

[0020] Read the process ID of the distributed file system data node process where the mapping / reduction task being executed in this node is located;

[0021] Initialize the disk input and output resource weight of the distributed file system data node process corresponding to the process number through a completely fair queue;

[0022] Start the monitoring service, obtain the execution progress of each of the mapping / reduction tasks being executed, and calculate the average execution progress and the total time consumed;

[0023] According to the first formula, the disk input / output resource weight of the process where each of the executing mapping / reducing tasks is located is calculated and modified.

[0024] By adopting the above technical solution, by using the completely fair queue technology in cgroup, the disk input and output resource weights of the processes where the mapping / reduction tasks are located that are executed to the end of the mapping phase or the end of the data shuffling and sorting phase are dynamically adjusted to improve resource utilization, avoid waste of input and output resources, and improve test performance, so that the input and output resource weights occupied by all tasks in the same node are the same.

[0025] Optionally, the calculating and modifying, according to the first formula, the disk input / output resource weight of the process where each of the executing mapping / reducing tasks is located includes:

[0026] According to the execution progress of the mapping / reduction task being executed and the corresponding average execution progress and total time, use the first formula

[0027] Math.max(L,Math.min(H,100 / Math.pow(2,d / (ma / t+1)))), calculates the disk input and output resource weight of the process where the mapping / reducing task is located;

[0028] According to the process number corresponding to the mapping / reduction task, determine the process where the mapping / reduction task is located, and reassign the calculated disk input and output resource weight to the process;

[0029] Among them, L in the first formula is the lower limit of the weight, H is the upper limit of the weight, d is the difference between the execution progress of the mapping / reduction task and the average execution progress ma, and t is the total time consumption.

[0030] By adopting the above technical solution, the first formula can be used to adaptively adjust the disk input and output resource weights of the process where the task is located in multiple dimensions such as time and task progress difference, optimize the use of input and output resources, and reduce data skew.

[0031] Optionally, the step of calculating and executing a corresponding distributed file system client write blocking delay for the process where the mapping / reducing task being executed resides includes:

[0032] Added a write blocking delay parameter to the write context method of the mapping specification framework;

[0033] By accessing the monitoring service, the execution progress of each of the mapping / reduction tasks being executed, as well as the corresponding average execution progress and total time consumption, the CPU load of this node and the input and output throughput of each disk are obtained;

[0034] For the mapping / reduce task being executed, if the execution progress of the mapping / reduce task is greater than the execution progress average, and the input and output throughput of the disk corresponding to the mapping / reduce task reaches a first threshold, or the central processing unit load reaches a second threshold, the distributed file system client write blocking delay is calculated according to the second formula and executed on the process where the mapping / reduce task is located.

[0035] By adopting the above technical solution and using cgroup technology, the input and output blocking delay of the process where the mapping / reduction task is located at the end of the specification phase is dynamically adjusted to improve the overall resource utilization, balance the task progress, and improve the test performance, so that all tasks of the node can be completed almost at the same time, avoiding the overall progress being blocked by the last task.

[0036] Optionally, the step of calculating the distributed file system client write blocking delay according to the second formula and executing the calculation on the process where the mapping / reducing task is located includes:

[0037] According to the execution progress of the mapping / reduce task being executed and the corresponding average execution progress and total time consumption, the distributed file system client write blocking delay is calculated using the second formula 100*d / (ma / t+1), and is executed on the process where the mapping / reduce task is located;

[0038] Wherein, d in the second formula is the difference between the execution progress of the mapping / reduction task and the average execution progress ma, and t is the total time consumption.

[0039] By adopting the above technical solution, the second formula can be used to adaptively adjust the distributed file system client write blocking delay according to the actual progress and overall time consumption of the task, so as to ensure that different mapping / reduction tasks can be completed in a similar time and balance the overall resource usage.

[0040] According to another aspect of the present invention, a performance optimization device for a mapping reduction framework is provided, comprising:

[0041] The parameter balancing processing module is used to perform balancing processing based on pre-configured node parameters and concurrently execute mapping / reduction tasks according to the balancing processing results;

[0042] A weight adjustment module, used for dynamically balancing the disk input and output resource weights of the process where the mapping / reduction task is being executed through a completely fair queue if the mapping / reduction task is executed to the end of the mapping phase or the end of the data shuffling and sorting phase;

[0043] The dynamic delay module is used to calculate and execute the corresponding distributed file system client write blocking delay for the process where the mapping / reducing task is being executed if the mapping / reducing task is executed to the end of the reduction stage.

[0044] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0045] at least one processor; and

[0046] a memory communicatively connected to the at least one processor; wherein,

[0047] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the performance optimization method of the mapping reduction framework described in any embodiment of the present invention.

[0048] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the performance optimization method of the mapping reduction framework described in any embodiment of the present invention when executed.

[0049] The technical solution of the embodiment of the present invention performs balancing processing based on pre-configured node parameters, and concurrently executes mapping / reduction tasks according to the balancing processing results; if the mapping / reduction task is executed to the end of the mapping stage or the end of the data shuffling and sorting stage, the disk input and output resource weights of the process where the mapping / reduction task is being executed are dynamically balanced through a completely fair queue; if the mapping / reduction task is executed to the end of the reduction stage, the corresponding distributed file system client write blocking delay is calculated and executed for the process where the mapping / reduction task is being executed, thereby solving the data skew problem caused by I / O bottlenecks under high concurrency, and realizing dynamic balancing of completely fair queue weights and dynamic write blocking of clients by using cgroup technology, thereby realizing dynamic isolation of disk input and output resources of mapping / reduction tasks and improving resource utilization.

[0050] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0052] Figure 1 This is a flowchart of a method for optimizing the performance of a mapping-reduce framework provided in accordance with Embodiment 1 of the present invention;

[0053] Figure 2 is a flowchart of another method for optimizing the performance of a mapping-reduce framework provided in accordance with Embodiment 2 of the present invention;

[0054] Figure 32 is a schematic diagram of the structure of a performance optimization device for a mapping reduction framework provided according to Embodiment 3 of the present invention;

[0055] Figure 4 It is a structural schematic diagram of an electronic device for implementing the performance optimization method of the mapping reduction framework of an embodiment of the present invention. DETAILED DESCRIPTION

[0056] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0057] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0058] Embodiment 1

[0059] Figure 1 This is a flowchart of a performance optimization method for a mapping specification framework provided according to the first embodiment of the present invention. This embodiment can be applied to the case where the cgroup technology is used to dynamically isolate the disk input and output of the mapping / reduction tasks in the node. The method can be executed by a performance optimization device for the mapping specification framework. The device can be implemented in the form of hardware and / or software, and the device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0060] S110, performing balancing processing based on pre-configured node parameters, and concurrently executing mapping / reduction tasks according to the balancing processing result.

[0061] The mapping-reduce framework in this embodiment refers to the MapReduce framework of Hadoop2 or above based on the resource management and task scheduling service YARN. Node parameters refer to parameters related to resource allocation and task scheduling. In this embodiment, in order to achieve nearly simultaneous completion of all mapping tasks (i.e., Map tasks) / reduce tasks (i.e., Reduce tasks), it is necessary to configure the relevant parameters of the input and output resource scheduling of the device in advance, and then balance and optimize the configured parameters based on the saturation of the CPU resources and disk input and output resources of this node, and execute the mapping / reduce tasks concurrently according to the parameters such as the concurrent number of node tasks and the number of CPU cores of each task determined after the balancing process.

[0062] In an optional embodiment, before performing balancing processing based on pre-configured node parameters and concurrently executing mapping / reduction tasks according to the balancing processing results, it also includes: setting the block scheduling algorithm involving the disk of the computing node local directory parameters in the resource management and task scheduling service to a completely fair queue; configuring the number of central processing unit cores for mapping tasks and reduction tasks; configuring the number of central processing unit cores that can be allocated to the computing node service in the resource management and task scheduling service.

[0063] In this embodiment, the computing node local directory parameter yarn.nodemanager.local-dirs in the resource management and task scheduling service can specify the local directory for storing container logs and other data in the computing node service (i.e., NodeManager in YARN) in the resource management and task scheduling service. By setting the block scheduling algorithm of the computing node local directory parameter in the resource management and task scheduling service to a completely fair queue, the completely fair queue technology in the cgroup can be used to dynamically limit the disk input and output resources. The parameter mapreduce.map / reduce.cpu.vcores is used to specify the number of virtual CPU cores required for the mapping and reduction tasks, respectively. In order to better verify the effectiveness of the scheme of this embodiment and the involved scenarios, the parameter can be set to 1. The parameter yarn.nodemanager.resource.cpu-vcores is used to set the number of logical CPU cores that can be allocated for the computing node service in the resource management and task scheduling service. Exemplarily, this parameter can be set to 32 logical cores. These parameters play a key role in the configuration of Hadoop and resource management and task scheduling services, and have a direct impact on resource allocation, task scheduling, and overall cluster performance.

[0064] S120: If the mapping / reducing task is executed to the end of the mapping phase or the end of the data shuffling and sorting phase, the disk input and output resource weights of the process where the mapping / reducing task is being executed are dynamically balanced through a completely fair queue.

[0065] In this embodiment, the performance bottleneck caused by the imbalance of disk input and output is usually concentrated at the end of the mapping stage, the end of the data shuffling and sorting stage, and the end of the reduction stage. In order to dynamically limit the input and output resources of the device and avoid the waste of input and output resources, when the mapping / reduction task is executed to the end of the mapping stage or the end of the data shuffling and sorting stage, the monitoring service can be enabled to monitor the progress of the task execution, and the completely fair queue is used to dynamically balance the disk input and output resource weights of the process where the mapping / reduction task is being executed, so that the input and output resource weights occupied by each task in the same node are the same.

[0066] Optionally, the disk input and output resource weights of the process where the mapping / reduce task being executed is located are dynamically balanced through a completely fair queue, including: reading the process number of the distributed file system data node process where the mapping / reduce task being executed in this node is located; initializing the disk input and output resource weights of the distributed file system data node process corresponding to the process number through a completely fair queue; starting a listening service to obtain the execution progress of each mapping / reduce task being executed, and calculating the average execution progress and the total time consumed; according to the first formula, calculating and modifying the disk input and output resource weights of the process where each mapping / reduce task being executed is located.

[0067] In this embodiment, when the service process of the resource management and task scheduling service computing node is running in this node, if the mapping / reduction task is executed to the end of the mapping phase or the end of the data shuffling and sorting phase, an independent service process can be started to calculate and dynamically modify the disk input and output resource weight corresponding to the mapping / reduction task. Specifically, the process number of the distributed file system data node (i.e., HDFSDataNode) process where the mapping / reduction task is being executed in this node, as well as the central processing unit load of this node and the input and output throughput of each disk can be read. Then, for the distributed file system data node process corresponding to each process number, the disk input and output resource weight value of the process is adjusted to be much greater than the default weight using the completely fair queue technology. For example, if the default weight value is 100, the disk input and output resource weight value of the process can be increased to 1000 to give priority to copying and avoid calculation imbalance during dynamic tuning. Turn on monitoring, and monitor the execution progress of each mapping / reduction task currently in execution every three seconds. Among them, the execution progress can be expressed as a percentage. Calculate the average execution progress and overall time consumption of all tasks being executed in this node. According to the pre-designed first formula, the disk input and output resource weights of the processes where each map / reduce task is being executed are coarsely calculated, and the corresponding weights are adjusted. Among them, the first formula dynamically adjusts the weight according to the difference between the execution progress of the task and the average execution progress.

[0068] By adopting the above technical solution, by using the completely fair queue technology in cgroup, the disk input and output resource weights of the processes where the mapping / reduction tasks are located that are executed to the end of the mapping phase or the end of the data shuffling and sorting phase are dynamically adjusted to improve resource utilization, avoid waste of input and output resources, and improve test performance, so that the input and output resource weights occupied by all tasks in the same node are the same.

[0069] S130: If the mapping / reducing task is executed to the end of the reduction phase, a corresponding distributed file system client write blocking delay is calculated and executed for the process where the mapping / reducing task is being executed.

[0070] In this embodiment, in order to balance the execution progress of each task, when the mapping / reduce task is executed to the end of the protocol stage, the distributed file system client write blocking delay can be calculated for the process where the mapping / reduce task is being executed, and the cgroup control process can be used to suspend and resume the control process. After the control process is suspended for a corresponding period of time, the write operation can be continued.

[0071] Optionally, for the process where the mapping / reduce task being executed is located, the corresponding distributed file system client write blocking delay is calculated and executed, including: adding a write blocking delay parameter in the write context method of the mapping / reduce framework; obtaining the execution progress of each mapping / reduce task being executed, as well as the corresponding execution progress average and total time consumed by accessing the listening service, and obtaining the central processing unit load of this node and the input and output throughput of each disk; for the mapping / reduce task being executed, if the execution progress of the mapping / reduce task is greater than the execution progress average, and the input and output throughput of the disk corresponding to the mapping / reduce task reaches a first threshold, or the central processing unit load reaches a second threshold, then the distributed file system client write blocking delay is calculated according to the second formula and executed on the process where the mapping / reduce task is located.

[0072] In this embodiment, the writing of the mapping / reduction task depends on the Hadoop distributed file system, so it is necessary to add a write blocking delay parameter in the write context method (i.e., context.write method) of the mapping / reduction framework to ensure that the task execution is balanced. After modifying the mapping / reduction framework, before the native write context method writes DFSOutputStream, by accessing the monitoring service, the execution progress of each mapping / reduction task being executed, the average execution progress ma of all tasks, the total time t, the difference d between the execution progress of each mapping / reduction task and the average execution progress ma, and the CPU load of this node and the input and output throughput of each disk. Then for each mapping / reduction task being executed, if the execution progress of the task is greater than the average execution progress, and the input and output throughput of the disk corresponding to the task reaches the first threshold, such as 99%, or the CPU load reaches the second threshold, such as 100%, then the distributed file system client write blocking delay is calculated coarsely according to the second formula, and the corresponding time of the process where the mapping / reduction task is located is controlled to suspend based on the cgroup technology, and then the write operation is continued.

[0073] By adopting the above technical solution and using cgroup technology, the input and output blocking delay of the process where the mapping / reduction task is located at the end of the specification phase is dynamically adjusted to improve the overall resource utilization, balance the task progress, and improve the test performance, so that all tasks of the node can be completed almost at the same time, avoiding the overall progress being blocked by the last task.

[0074] The technical solution of the embodiment of the present invention performs balancing processing based on pre-configured node parameters, and concurrently executes mapping / reduction tasks according to the balancing processing results; if the mapping / reduction task is executed to the end of the mapping stage or the end of the data shuffling and sorting stage, the disk input and output resource weights of the process where the mapping / reduction task is being executed are dynamically balanced through a completely fair queue; if the mapping / reduction task is executed to the end of the reduction stage, the corresponding distributed file system client write blocking delay is calculated and executed for the process where the mapping / reduction task is being executed, thereby solving the data skew problem caused by I / O bottlenecks under high concurrency, and realizing dynamic balancing of completely fair queue weights and dynamic write blocking of clients by using cgroup technology, thereby realizing dynamic isolation of disk input and output resources of mapping / reduction tasks and improving resource utilization.

[0075] Embodiment 2

[0076] Figure 2 1 is a flowchart of another method for optimizing the performance of a mapping-reduce framework according to Embodiment 2 of the present invention. Based on the above embodiment, this embodiment further provides specific steps for performing parameter-based balancing, as well as a specific explanation of the first formula and the second formula. Figure 2 As shown, the method includes:

[0077] S210 , pre-configure node parameters, perform balancing processing based on the pre-configured node parameters, and concurrently execute mapping / reduction tasks according to the balancing processing result.

[0078] In this embodiment, in order to achieve nearly simultaneous completion of all mapping / reducing tasks, it is necessary to configure the relevant parameters of the input and output resource scheduling of the device in advance, for example, configure the number of CPU cores of the mapping / reducing tasks, configure the number of CPU cores that can be allocated by the computing node service in the resource management and task scheduling service, etc. Then, by balancing and optimizing the configured parameters, the final number of concurrent node tasks, the number of CPU cores of each task, and other parameters are determined, and the mapping / reducing tasks are executed concurrently based on these parameters.

[0079] In an optional embodiment, balancing processing is performed based on pre-configured node parameters, including: based on the pre-configured node parameters, executing mapping / reduce task balance detection; if it is detected that the disk input / output resources are saturated and the central processing unit resources are idle, then the number of concurrent tasks of the node is adjusted downward; if it is detected that the disk input / output resources are idle and the central processing unit resources are saturated, then the number of central processing unit cores of each task is adjusted upward; if it is detected that the disk input / output resources are idle and the central processing unit resources are idle, then the number of concurrent tasks of the node is adjusted upward.

[0080] By adopting the above technical solution, the pre-configured node parameters can be balanced and optimized according to the usage saturation of the CPU resources and disk input and output resources, further improving the overall performance. Among them, when the disk input and output resources are idle and the CPU resources are saturated, adjusting the number of CPU cores of each task upward will actually reduce the number of tasks running concurrently on the node, allowing each task to perform input and output operations faster.

[0081] S220. If the mapping / reducing task is executed to the end of the mapping phase or the end of the data shuffling and sorting phase, the process number of the distributed file system data node process where the mapping / reducing task being executed in the current node is located is read.

[0082] S230. Initialize the disk input and output resource weight of the distributed file system data node process corresponding to the process number through a completely fair queue.

[0083] S240: Start the monitoring service, obtain the execution progress of each mapping / reduction task being executed, and calculate the average execution progress and the total time consumed.

[0084] S250: Calculate and modify the disk input and output resource weights of the processes where each mapping / reduce task being executed is located according to the first formula.

[0085] In an optional implementation, according to the first formula, the disk input and output resource weight of the process where each mapping / reduce task being executed is located is calculated and modified, including: according to the execution progress of the mapping / reduce task being executed and the corresponding execution progress average value and total time consumed, the first formula Math.max(L,Math.min(H,100 / Math.pow(2,d / (ma / t+1)))) is used to calculate the disk input and output resource weight of the process where the mapping / reduce task is located; according to the process number corresponding to the mapping / reduce task, the process where the mapping / reduce task is located is determined, and the calculated disk input and output resource weight is reassigned to the process; wherein L in the first formula is the lower limit of the weight, H is the upper limit of the weight, d is the difference between the execution progress of the mapping / reduce task and the execution progress average value ma, and t is the total time consumed.

[0086] In this embodiment, the first formula uses an exponential function to dynamically adjust the disk input and output resource weight of the process according to the difference between the task execution progress and the average execution progress, so as to improve the utilization rate of input and output resources and reduce data skew. For the first formula, Math.max(L,...) sets a weight lower limit of L to ensure that the weight is not too low. Math.min(H....) sets a weight upper limit of H to ensure that the weight is not too high. 100 / Math.pow(2,d / (ma / t+1)) This part uses an exponential decrease function to adjust the weight according to the value of d / (ma / t+1), which more accurately reflects the gap between the task execution progress and the average execution progress. If d is close to ma, that is, the task execution progress is close to the average execution progress, the weight will be close to its maximum value H.

[0087] By adopting the above technical solution, the disk input and output resource weight of the task process can be adaptively adjusted in multiple dimensions such as time and task progress difference through the first formula, rather than relying on a single indicator. And by setting the maximum and minimum weight values, it is ensured that the weight will not become unreasonable even in extreme cases.

[0088] S260. Add a write blocking delay parameter in the write context method of the mapping specification framework.

[0089] Among them, the timing for adding write blocking delay parameters in the write context method of the mapping specification framework is usually before the mapping / reducing task is executed, or after the mapping / reducing task is executed, but before reaching the end of the specification phase.

[0090] S270. If the mapping / reduce task is executed to the end of the reduction phase, the execution progress of each mapping / reduce task being executed, as well as the corresponding average execution progress and total time consumed, are obtained by accessing the monitoring service to obtain the CPU load of this node and the input and output throughput of each disk.

[0091] S280. For the mapping / reduce task being executed, if the execution progress of the mapping / reduce task is greater than the average execution progress, and the input / output throughput of the disk corresponding to the mapping / reduce task reaches a first threshold, or the central processing unit load reaches a second threshold, the distributed file system client write blocking delay is calculated according to the second formula and executed on the process where the mapping / reduce task is located.

[0092] In an optional implementation, the distributed file system client write blocking delay is calculated according to the second formula and executed on the process where the mapping / reduction task is located, including: according to the execution progress of the mapping / reduction task being executed and the corresponding execution progress average value and total time consumed, the distributed file system client write blocking delay is calculated using the second formula 100*d / (ma / t+1), and executed on the process where the mapping / reduction task is located; wherein d in the second formula is the difference between the execution progress of the mapping / reduction task and the execution progress average value ma, and t is the total time consumed.

[0093] In this embodiment, the second formula calculates the delay by d / (ma / t+1), which ensures that if the execution progress of the current task is greater than the average execution progress ma, and the disk input and output resources or the CPU resources are close to saturation, then the task will be appropriately delayed. And the higher the task execution progress, the longer the delay time will be, thus playing a balancing role.

[0094] By adopting the above technical solution, the second formula can be used to adaptively adjust the distributed file system client write blocking delay according to the actual progress and overall time consumption of the task, so as to ensure that different mapping / reduction tasks can be completed in a similar time and balance the overall resource usage.

[0095] The technical solution of the embodiment of the present invention performs balancing processing based on pre-configured node parameters, and concurrently executes mapping / reduction tasks according to the balancing processing results; if the mapping / reduction task is executed to the end of the mapping stage or the end of the data shuffling and sorting stage, the disk input and output resource weights of the process where the mapping / reduction task is being executed are dynamically balanced through a completely fair queue; if the mapping / reduction task is executed to the end of the reduction stage, the corresponding distributed file system client write blocking delay is calculated and executed for the process where the mapping / reduction task is being executed, thereby solving the data skew problem caused by I / O bottlenecks under high concurrency, and realizing dynamic balancing of completely fair queue weights and dynamic write blocking of clients by using cgroup technology, thereby realizing dynamic isolation of disk input and output resources of mapping / reduction tasks and improving resource utilization.

[0096] Embodiment 3

[0097] Figure 3 1 is a schematic diagram of a performance optimization device for a mapping reduction framework according to Embodiment 3 of the present invention. Figure 3 As shown, the device comprises:

[0098] The parameter balancing processing module 310 is used to perform balancing processing based on pre-configured node parameters and concurrently execute mapping / reduction tasks according to the balancing processing results;

[0099] The weight adjustment module 320 is used to dynamically balance the disk input and output resource weights of the process where the mapping / reducing task is being executed through a completely fair queue if the mapping / reducing task is executed to the end of the mapping phase or the end of the data shuffling and sorting phase;

[0100] The dynamic delay module 330 is used to calculate and execute the corresponding distributed file system client write blocking delay for the process where the mapping / reducing task is being executed if the mapping / reducing task is executed to the end of the reduction phase.

[0101] The technical solution of the embodiment of the present invention performs balancing processing based on pre-configured node parameters, and concurrently executes mapping / reduction tasks according to the balancing processing results; if the mapping / reduction task is executed to the end of the mapping stage or the end of the data shuffling and sorting stage, the disk input and output resource weights of the process where the mapping / reduction task is being executed are dynamically balanced through a completely fair queue; if the mapping / reduction task is executed to the end of the reduction stage, the corresponding distributed file system client write blocking delay is calculated and executed for the process where the mapping / reduction task is being executed, thereby solving the data skew problem caused by I / O bottlenecks under high concurrency, and realizing dynamic balancing of completely fair queue weights and dynamic write blocking of clients by using cgroup technology, thereby realizing dynamic isolation of disk input and output resources of mapping / reduction tasks and improving resource utilization.

[0102] Optionally, it also includes: a parameter configuration module, which is used to perform balancing processing based on pre-configured node parameters and concurrently execute mapping / reduction tasks according to the balancing processing results.

[0103] Set the block scheduling algorithm involving disks in the local directory parameters of the computing nodes in the resource management and task scheduling service to completely fair queues;

[0104] Configure the number of CPU cores for mapping tasks and reduction tasks;

[0105] Configure the number of CPU cores that can be allocated to computing nodes in the resource management and task scheduling services.

[0106] Optionally, the parameter equalization processing module 310 is used to:

[0107] Perform map / reduce task balance detection based on pre-configured node parameters;

[0108] If it is detected that the disk input and output resources are saturated and the CPU resources are idle, the number of concurrent tasks of the node is adjusted downward;

[0109] If it is detected that the disk I / O resources are idle and the CPU resources are saturated, the number of CPU cores for each task is adjusted upward;

[0110] If it is detected that the disk input and output resources are idle and the CPU resources are idle, the number of concurrent tasks of the node is adjusted upward.

[0111] Optionally, the weight adjustment module 320 is used to:

[0112] Read the process ID of the distributed file system data node process where the mapping / reduction task being executed in this node is located;

[0113] Initialize the disk input and output resource weights of the distributed file system data node process corresponding to the process number through the completely fair queue;

[0114] Start the monitoring service, obtain the execution progress of each mapping / reduction task being executed, and calculate the average execution progress and total time consumed;

[0115] According to the first formula, the disk input and output resource weights of the processes where each mapping / reducing task being executed resides are calculated and modified.

[0116] Optionally, the weight adjustment module 320 is used to:

[0117] According to the execution progress of the map / reduce task being executed and the corresponding average execution progress and total time consumed, the first formula Math.max(L,Math.min(H,100 / Math.pow(2,d / (ma / t+1)))) is used to calculate the disk input and output resource weight of the process where the map / reduce task is located;

[0118] According to the process number corresponding to the mapping / reduction task, determine the process where the mapping / reduction task is located, and reassign the calculated disk input and output resource weights to the process;

[0119] Among them, L in the first formula is the lower limit of the weight, H is the upper limit of the weight, d is the difference between the execution progress of the mapping / reduction task and the average execution progress ma, and t is the total time consumption.

[0120] Optionally, the dynamic delay module 330 is used to:

[0121] Added a write blocking delay parameter to the write context method of the mapping specification framework;

[0122] By accessing the monitoring service, we can obtain the execution progress of each mapping / reduction task being executed, as well as the corresponding average execution progress and total time consumption, and obtain the CPU load of this node and the input and output throughput of each disk;

[0123] For the mapping / reduce task being executed, if the execution progress of the mapping / reduce task is greater than the average execution progress, and the input and output throughput of the disk corresponding to the mapping / reduce task reaches a first threshold, or the central processing unit load reaches a second threshold, the distributed file system client write blocking delay is calculated according to the second formula and executed on the process where the mapping / reduce task is located.

[0124] Optionally, the dynamic delay module 330 is used to:

[0125] According to the execution progress of the mapping / reduce task being executed and the corresponding average execution progress and total time consumption, the second formula 100*d / (ma / t+1) is used to calculate the distributed file system client write blocking delay and execute it on the process where the mapping / reduce task is located;

[0126] Wherein, d in the second formula is the difference between the execution progress of the mapping / reduction task and the average execution progress ma, and t is the total time consumption.

[0127] The performance optimization device of the mapping reduction framework provided in the embodiment of the present invention can execute the performance optimization method of the mapping reduction framework provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0128] Embodiment 4

[0129] Figure 4 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0130] like Figure 4As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0131] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0132] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the performance optimization method of the mapping reduction framework.

[0133] In some embodiments, the performance optimization method of the mapping reduction framework may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the performance optimization method of the mapping reduction framework described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to execute the performance optimization method of the mapping reduction framework in any other appropriate manner (e.g., by means of firmware).

[0134] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0135] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0136] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0137] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0138] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0139] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0140] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0141] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A performance optimization method for a mapping reduction framework, characterized in that: include: Perform balancing based on pre-configured node parameters and concurrently execute mapping / reduction tasks based on the balancing results; If the mapping / reducing task is executed to the end of the mapping phase or the end of the data shuffling and sorting phase, the disk input and output resource weights of the process where the mapping / reducing task is being executed are dynamically balanced through a completely fair queue; If the mapping / reducing task is executed to the end of the reduction phase, the corresponding distributed file system client write blocking delay is calculated and executed for the process where the mapping / reducing task is being executed.

2. The method according to claim 1, characterized in that Before performing balancing based on pre-configured node parameters and concurrently executing mapping / reducing tasks according to the balancing results, the following is also included: Set the block scheduling algorithm involving disks in the local directory parameters of the computing nodes in the resource management and task scheduling service to completely fair queues; Configure the number of CPU cores for mapping tasks and reduction tasks; Configure the number of CPU cores that can be allocated to computing nodes in the resource management and task scheduling services.

3. The method according to claim 2, characterized in that The balancing process based on pre-configured node parameters includes: Perform map / reduce task balance detection based on pre-configured node parameters; If it is detected that the disk input and output resources are saturated and the CPU resources are idle, the number of concurrent tasks of the node is adjusted downward; If it is detected that the disk I / O resources are idle and the CPU resources are saturated, the number of CPU cores for each task is adjusted upward; If it is detected that the disk input and output resources are idle and the CPU resources are idle, the number of concurrent tasks of the node is adjusted upward.

4. The method according to claim 1, characterized in that: The method of dynamically balancing the disk input and output resource weights of the process where the mapping / reducing task being executed is located by using a completely fair queue includes: Read the process ID of the distributed file system data node process where the mapping / reduction task being executed in this node is located; Initialize the disk input and output resource weight of the distributed file system data node process corresponding to the process number through a completely fair queue; Start the monitoring service, obtain the execution progress of each of the mapping / reduction tasks being executed, and calculate the average execution progress and the total time consumed; According to the first formula, the disk input / output resource weight of the process where each of the executing mapping / reducing tasks is located is calculated and modified.

5. The method according to claim 4, characterized in that The calculating and modifying the disk input and output resource weight of the process where each of the mapping / reducing tasks being executed is located according to the first formula includes: According to the execution progress of the mapping / reduction task being executed and the corresponding average execution progress and total time, use the first formula Math.max(L,Math.min(H,100 / Math.pow(2,d / (ma / t+1)))), calculates the disk input and output resource weight of the process where the mapping / reducing task is located; According to the process number corresponding to the mapping / reduction task, determine the process where the mapping / reduction task is located, and reassign the calculated disk input and output resource weight to the process; Among them, L in the first formula is the lower limit of the weight, H is the upper limit of the weight, d is the difference between the execution progress of the mapping / reduction task and the average execution progress ma, and t is the total time consumption.

6. The method according to claim 1, characterized in that The step of calculating and executing the corresponding distributed file system client write blocking delay for the process where the mapping / reduction task being executed is located includes: Added a write blocking delay parameter to the write context method of the mapping specification framework; By accessing the monitoring service, the execution progress of each of the mapping / reduction tasks being executed, as well as the corresponding average execution progress and total time consumption, the CPU load of this node and the input and output throughput of each disk are obtained; For the mapping / reduce task being executed, if the execution progress of the mapping / reduce task is greater than the execution progress average, and the input and output throughput of the disk corresponding to the mapping / reduce task reaches a first threshold, or the central processing unit load reaches a second threshold, the distributed file system client write blocking delay is calculated according to the second formula and executed on the process where the mapping / reduce task is located.

7. The method according to claim 6, characterized in that The step of calculating the distributed file system client write blocking delay according to the second formula and executing the calculation on the process where the mapping / reduction task is located includes: According to the execution progress of the mapping / reduce task being executed and the corresponding average execution progress and total time consumption, the distributed file system client write blocking delay is calculated using the second formula 100*d / (ma / t+1), and is executed on the process where the mapping / reduce task is located; Wherein, d in the second formula is the difference between the execution progress of the mapping / reduction task and the average execution progress ma, and t is the total time consumption.

8. A performance optimization device for a mapping reduction framework, characterized in that: include: The parameter balancing processing module is used to perform balancing processing based on pre-configured node parameters and concurrently execute mapping / reduction tasks according to the balancing processing results; A weight adjustment module, used for dynamically balancing the disk input and output resource weights of the process where the mapping / reduction task is being executed through a completely fair queue if the mapping / reduction task is executed to the end of the mapping phase or the end of the data shuffling and sorting phase; The dynamic delay module is used to calculate and execute the corresponding distributed file system client write blocking delay for the process where the mapping / reducing task is being executed if the mapping / reducing task is executed to the end of the reduction stage.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the performance optimization method of the mapping reduction framework described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the performance optimization method of the mapping reduction framework described in any one of claims 1 to 7 when executed.

Images