Data shuffling control method and device and storage medium

By analyzing the historical event log of the data shuffling process, generating a dependency tree and adjusting parameters, the problem of low manual adjustment efficiency in the existing technology is solved, the stability and performance of the data shuffling process are improved, and the distributed computing cluster is reduced load and job failure risk.

CN118916681AActive Publication Date: 2024-11-08BEIJING VOLCANO ENGINE TECH CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202411182670.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-08
Estimated Expiration
2044-08-27

AI Technical Summary

Technical Problem

In the prior art, the parameter adjustment of data shuffling process depends on manual experience, and the efficiency and accuracy are poor, resulting in unstable data shuffling process and affecting the execution of big data analysis operations.

Method used

By analyzing the historical event log of the data shuffling process of the distributed computing cluster, generating a dependency tree, determining the bottleneck stage, and adjusting the data shuffling process parameters according to the read and write status of the bottleneck stage, and automatically optimizing the data shuffling process.

Benefits of technology

It improves the efficiency and accuracy of data shuffling process parameter optimization, reduces the load and job failure risk of distributed computing clusters, ensures job stability, does not rely on manual experience, and is suitable for large-scale applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118916681A_ABST
    Figure CN118916681A_ABST
Patent Text Reader

Abstract

The embodiment of the invention provides a data shuffling control method and device and a storage medium, and the method comprises the steps: determining an interdependency relationship among all read-write stages in a data shuffling process according to a historical event log of the data shuffling process of a distributed computing cluster, and generating a dependency relationship tree; determining a bottleneck stage in the data shuffling process according to the dependency relationship tree, and adjusting parameters of the data shuffling process according to read-write conditions of the bottleneck stage; wherein the bottleneck stage is a read-write stage in which the data volume of the read-written file is smaller than a preset data volume threshold value and the number of read-write times is maximum; and controlling the data shuffling process based on the adjusted parameters. According to the embodiment of the invention, the bottleneck stage of the data shuffling process is determined by analyzing the historical event log, and the parameters of the data shuffling process are automatically adjusted based on the bottleneck stage, so that the parameter optimization efficiency and accuracy of the data shuffling process can be improved, and the operation efficiency of the distributed computing cluster is improved on the premise of not influencing the operation stability and operation time of the distributed computing cluster. And the data shuffling performance is optimized.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and more particularly to a data shuffling control method, device, and storage medium. Background Art

[0002] In big data analysis systems, data shuffling is usually required. Data shuffling is the process of repartitioning and combining the data of all computing nodes. The stability of the data shuffling process is very important to the stability of big data analysis jobs. An important factor affecting the data shuffling process is that too many small files are generated during the data shuffling process. The smaller the file, the more times the data is read and written during the data shuffling process, the greater the pressure on the disk, and it will also affect the execution of big data analysis jobs.

[0003] In the prior art, by adjusting the parameters of the data shuffling process, the number of files can be effectively reduced, which means the number of read and write times can be reduced, and the file size can be increased. However, in the prior art, the adjustment of the parameters of the data shuffling process is usually done manually based on experience, and the adjustment efficiency and accuracy are relatively poor. Summary of the invention

[0004] Embodiments of the present disclosure provide a data shuffling control method, device, and storage medium to improve the efficiency and accuracy of parameter optimization during a data shuffling process.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for controlling data shuffling, comprising:

[0006] According to the historical event log of the data shuffling process of the distributed computing cluster, the interdependencies between all the read and write stages in the data shuffling process are determined, and a dependency tree is generated;

[0007] According to the dependency tree, the bottleneck stage in the data shuffling process is determined, and the parameters of the data shuffling process are adjusted according to the read and write conditions of the bottleneck stage; wherein the bottleneck stage is a read and write stage in which the amount of file data read and written is less than a preset data amount threshold and the number of read and write times is the largest;

[0008] Control the data shuffling process based on the tuned parameters.

[0009] In a second aspect, an embodiment of the present disclosure provides a data shuffling control device, including:

[0010] A generation unit, used for determining the interdependencies between all read and write stages in the data shuffling process according to the historical event log of the data shuffling process of the distributed computing cluster, and generating a dependency tree;

[0011] An adjustment unit, configured to determine a bottleneck stage in the data shuffling process according to the dependency tree, and adjust parameters of the data shuffling process according to the read and write conditions in the bottleneck stage; wherein the bottleneck stage is a read and write stage in which the amount of file data read and written is less than a preset data amount threshold and the number of read and write times is the largest;

[0012] An execution unit is used to control the data shuffling process based on the adjusted parameters.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: at least one processor and a memory; the memory stores computer-executable instructions;

[0014] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the data shuffling control method as described in the first aspect and various possible designs of the first aspect.

[0015] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the data shuffling control method described in the first aspect and various possible designs of the first aspect is implemented.

[0016] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer execution instructions. When a processor executes the computer execution instructions, it implements the data shuffling control method described in the first aspect and various possible designs of the first aspect.

[0017] The control method, device and storage medium of data shuffling provided by the disclosed embodiment determine the interdependence between all read and write stages in the data shuffling process according to the historical event log of the data shuffling process of the distributed computing cluster, and generate a dependency tree; determine the bottleneck stage in the data shuffling process according to the dependency tree, and adjust the parameters of the data shuffling process according to the read and write status of the bottleneck stage; wherein the bottleneck stage is the read and write stage in which the amount of file data read and written is less than the preset data amount threshold and the number of read and write times is the largest; and control the data shuffling process based on the adjusted parameters. In the disclosed embodiment, the bottleneck stage of the data shuffling process can be determined by analyzing the historical event log, and then the parameters of the data shuffling process can be automatically adjusted based on the bottleneck stage, which can improve the efficiency and accuracy of the optimization of the parameters of the data shuffling process, ensure that the performance of data shuffling is optimized without affecting the stability and running time of the distributed computing cluster operation, reduce the overall load of the distributed computing cluster, and reduce the risk of job failure, and do not rely on manual experience, and can be applied on a large scale. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0019] Figure 1 A schematic diagram of a scenario of a method for controlling data shuffling provided in an embodiment of the present disclosure;

[0020] Figure 2 A schematic diagram of a data shuffling control method flow chart provided by an embodiment of the present disclosure;

[0021] Figure 3 A schematic flow chart of a data shuffling control method provided by another embodiment of the present disclosure;

[0022] Figure 4 A structural block diagram of a data shuffling control device provided in an embodiment of the present disclosure;

[0023] Figure 5 A schematic diagram of the hardware structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0025] In the prior art, by adjusting the parameters of the data shuffling process, the number of files can be effectively reduced, which means the number of read and write times can be reduced, and the file size can be increased. However, in the prior art, the adjustment of the parameters of the data shuffling process is usually done manually based on experience, and the adjustment efficiency and accuracy are relatively poor.

[0026] In order to solve the above technical problems, the disclosed embodiment provides a data shuffling control method, by determining the interdependence between all read and write stages in the data shuffling process according to the historical event log of the data shuffling process of the distributed computing cluster, and generating a dependency tree; according to the dependency tree, determining the bottleneck stage in the data shuffling process, and adjusting the parameters of the data shuffling process according to the read and write status of the bottleneck stage; wherein the bottleneck stage is the read and write stage in which the file data volume read and written is less than the preset data volume threshold and the number of read and write times is the largest; and controlling the data shuffling process based on the adjusted parameters. In this embodiment, the bottleneck stage of the data shuffling process can be determined by analyzing the historical event log, and then the parameters of the data shuffling process can be automatically adjusted based on the bottleneck stage, which can improve the efficiency and accuracy of the optimization of the parameters of the data shuffling process, ensure that the performance of data shuffling is optimized without affecting the stability and running time of the distributed computing cluster operation, reduce the overall load of the distributed computing cluster, and reduce the risk of job failure, and do not rely on manual experience, and can be applied on a large scale.

[0027] The data shuffling control method of the disclosed embodiment is applicable to the data shuffling process of a distributed computing cluster, such as Figure 1 As shown, the historical event log of the data shuffling process of the distributed computing cluster can be obtained first, and a dependency tree can be generated based on the historical event log; the bottleneck stage in the data shuffling process can be determined according to the dependency tree, and the parameters of the data shuffling process can be adjusted according to the bottleneck stage; finally, the data shuffling process can be controlled based on the adjusted parameters.

[0028] The data shuffling control method disclosed in the present invention will be described in detail below in conjunction with specific embodiments.

[0029] refer to Figure 2 , Figure 2 A flow chart of a method for controlling data shuffling provided in an embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices such as terminal devices or servers included in a distributed computing cluster. The method for controlling data shuffling includes:

[0030] S201. Determine the interdependencies between all read and write phases in the data shuffling process according to the historical event log of the data shuffling process of the distributed computing cluster, and generate a dependency tree.

[0031] In this embodiment, for a distributed computing cluster such as a Spark cluster, a data shuffle process is usually required, which is a computing program that repartitions and combines the data of all computing nodes, and the data shuffle process will be recorded in an event log (such as Spark EventLog). In this embodiment, the historical event log of the distributed computing cluster can be obtained, and the historical event log related to the data cleaning process can be determined from it. The interdependence between all read and write stages in the data shuffle process is determined according to the historical event log of the data cleaning process.

[0032] The data shuffling process includes multiple write stages Shuffle Write (also known as Map Task, or InputTask), and read stages Shuffle Read (also known as Reduce Task). The read stage is responsible for reading the results of the write stage. The results output by the read stage may be used as input for another write stage. The historical event log of the data shuffling process can record the interdependencies between all read and write stages in the data shuffling process, such as which read stage is responsible for reading the results of which write stage, and which read stage outputs the results that are input into which write stage. Based on the interdependencies between all read and write stages in the data shuffling process, a dependency tree can be generated to characterize the interdependencies between all read and write stages in the data shuffling process.

[0033] Optionally, the amount of file data read in any Shuffle Read phase during the data shuffling process (that is, the amount of file data read by each Reduce Task in the read phase) can also be determined, denoted as shuffle read size (the average file data amount of each read task in the read phase, that is, avg shuffle read) and stored in the dependency tree; optionally, the number of read tasks (Reduce Task) in any Shuffle Read phase during the data shuffling process can also be determined, denoted as R, and the number of write tasks (Map Task) in the Shuffle Write phase (the output of the write phase is used as the input of the read phase) corresponding to the Shuffle Read phase can also be determined, denoted as M, which can also be stored in the dependency tree. Optionally, a specific data structure, such as shuffle read size / M / R, can be used to store the amount of file data read in any Shuffle Read phase, the number of read tasks in the read phase, and the number of write tasks in the corresponding write phase in the dependency tree for easy application in subsequent traversal processes.

[0034] S202. Determine the bottleneck stage in the data shuffling process according to the dependency tree, and adjust the parameters of the data shuffling process according to the read and write conditions of the bottleneck stage; wherein the bottleneck stage is a read and write stage in which the amount of file data read and written is less than a preset data amount threshold and the number of read and write times is the largest.

[0035] In this embodiment, the dependency tree can be traversed to filter out the bottleneck stage in the data shuffling process from the dependency tree, wherein the bottleneck stage is a read-write stage in which the amount of file data read and written is less than a preset data amount threshold and the number of read and write times is the largest. Since the parameters for adjusting the data shuffling process are shared by all read-write stages, the parameters of the data shuffling process have the greatest impact on the bottleneck stage. Therefore, after filtering out the bottleneck stage, the reason why it becomes a bottleneck can be analyzed based on the actual situation of the bottleneck stage, and then the parameters of the data shuffling process can be adjusted, so that when the subsequent data shuffling process is performed based on the adjusted parameters, the bottleneck stage will no longer become a bottleneck.

[0036] Optionally, when screening the bottleneck stage, the amount of file data read by each read stage in the dependency tree can be determined, that is, the above-mentioned shuffle read size, and the amount of file data read by each read stage in the dependency tree can be compared with a preset data volume threshold (such as 20Kb), and the read stages that are smaller than the preset data volume threshold are screened out to obtain a set of alternative read stages; the read stage with the largest number of read and write times is screened out from the set of alternative read stages, and the read stage and the corresponding write stage are determined as the bottleneck stage.

[0037] The number of read and write times of any read phase can be the product of the number of read tasks of the read phase (i.e., the above R) and the number of write tasks of the corresponding write phase (i.e., the above M), that is, the number of read and write times of any read phase is equal to M*R. In this embodiment, the number of read tasks of each read phase in the candidate read phase set and the number of write tasks of the corresponding write phase can be obtained, and the product of the number of read tasks of any read phase and the number of write tasks of the corresponding write phase can be obtained to determine the number of read and write times of the read phase; and the read phase with the largest number of read and write times is selected from the candidate read phase set, and the read phase and the corresponding write phase are determined as the bottleneck phase.

[0038] It should be noted that the shuffle read size, M, and R required in this embodiment may be pre-stored in the dependency tree as described in the above embodiment, or may not be pre-selected and stored in the dependency tree, but may be obtained when needed. Of course, the determination of the bottleneck stage is not limited to the above method, and any other feasible method may be used to screen out the reading stage and the corresponding writing stage with a sufficiently small file and a sufficiently large number of read and write times, and determine them as the bottleneck stage, which will not be elaborated here.

[0039] Furthermore, after determining the bottleneck stage in the data shuffling process, the parameters of the data shuffling process can be adjusted based on the read and write conditions (such as the number of read and write tasks) in the bottleneck stage, where the adjusted parameters may include a first parameter and / or a second parameter, where the first parameter is spark.sql.files.maxPartitionBytes, which is used to indicate the maximum amount of file data written by each write task in any write stage of the data shuffling process (the first parameter is used to control the maximum amount of data in each partition when a file is divided into multiple partitions during the data shuffling process, and one partition corresponds to one write task, which is equivalent to the maximum amount of data written by each write task), and the second parameter is spark.sql.adaptive.maxNumPostShufflePartitions, which is used to indicate the maximum number of read tasks in any read stage of the data shuffling process (that is, the maximum number of Reduce Tasks included in any read stage).

[0040] Optionally, when adjusting the parameters of the data shuffling process, the purpose is to reduce the number of read and write times in the bottleneck stage, which is the product of the number of read tasks in the read stage in the bottleneck stage and the number of write tasks in the corresponding write stage, that is, M*R. It can be seen that reducing the number of read and write times in the bottleneck stage can be achieved by reducing the number of write tasks M and / or reducing the number of read tasks R in the read stage, and M is related to the first parameter, and the larger the first parameter, the smaller M. R is related to the second parameter (R is less than or equal to the second parameter), that is, the larger the second parameter, the larger R. Therefore, the number of read and write times M*R in the bottleneck stage can be reduced by adjusting the first parameter and / or the second stage, and correspondingly, the amount of file data read in the read stage is also increased.

[0041] Optionally, when adjusting the first parameter and / or the second parameter of the data shuffling process according to the bottleneck stage, the first parameter or the second parameter may be adjusted according to the size relationship between the number of write tasks M in the write stage and the number of read tasks R in the read stage, where

[0042] If the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, the first parameter is increased, and optionally, the first parameter can be doubled; or, if the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, the second parameter is reduced, and optionally, the second parameter can be reduced by half. Through the above adjustment process of the first parameter or the second parameter, the purpose of reducing the number of reads and writes in the bottleneck phase can be achieved, so that the bottleneck phase will no longer become a bottleneck in the next data shuffling process, and a new bottleneck phase may appear in the next data shuffling process, and the above process is repeated, thereby realizing dynamic adjustment of parameters, and adjusting one parameter each time can be targeted to improve the accuracy and efficiency of the adjustment process.

[0043] S203: Control the data shuffling process based on the adjusted parameters.

[0044] In this embodiment, after adjusting the parameters of the data shuffling process, the data shuffling process can be controlled based on the adjusted parameters. It should be noted that since the dependency tree is determined based on the historical event log of the data shuffling process, when the logic of the data shuffling process remains unchanged, the data dependency tree does not change, but what changes is the file data volume, task quantity, etc. of each read and write stage in the data relationship tree, so the bottleneck stage will also change accordingly. In this embodiment, the parameters can be adjusted based on the bottleneck stage of the previous data shuffling process to guide the next data shuffling process, which can well adapt to the data change trend.

[0045] The data shuffling control method provided in this embodiment determines the interdependence between all read and write stages in the data shuffling process according to the historical event log of the data shuffling process of the distributed computing cluster, and generates a dependency tree; determines the bottleneck stage in the data shuffling process according to the dependency tree, and adjusts the parameters of the data shuffling process according to the read and write status of the bottleneck stage; wherein the bottleneck stage is the read and write stage in which the file data volume read and written is less than the preset data volume threshold and the number of read and write times is the largest; and controls the data shuffling process based on the adjusted parameters. In this embodiment, the bottleneck stage of the data shuffling process can be determined by analyzing the historical event log, and then the parameters of the data shuffling process can be automatically adjusted based on the bottleneck stage, which can improve the efficiency and accuracy of the optimization of the parameters of the data shuffling process, ensure that the performance of data shuffling is optimized without affecting the stability and running time of the distributed computing cluster operation, reduce the overall load of the distributed computing cluster, and reduce the risk of job failure, and do not rely on manual experience, and can be applied on a large scale.

[0046] On the basis of any of the above embodiments, considering that the dependency tree may be too complex, a lot of information is required, and more computing resources may be occupied and time-consuming, the present embodiment considers simplifying the dependency tree, and can analyze the historical event logs of multiple different data shuffling processes to obtain the dependency trees of different data shuffling processes, and determine the bottleneck stages in different data shuffling processes by the method in the above embodiment. By analyzing the bottleneck stages of multiple different data shuffling processes, the bottleneck stage characteristic index can be determined, that is, different bottleneck stages have similar characteristic indexes, especially in the case where the first parameter needs to be adjusted based on the bottleneck stage, the bottleneck stage characteristic index has a certain similarity, and in the case where the second parameter needs to be adjusted based on the bottleneck stage, the bottleneck stage characteristic index also has a certain similarity, and the two situations can be distinguished from each other by the bottleneck stage characteristic index, that is, when the bottleneck stage characteristic index meets a certain condition (recorded as the first condition), it is determined to adjust the first parameter, and when the bottleneck stage characteristic index meets another certain condition (recorded as the second condition), it is determined to adjust the second parameter.

[0047] Optionally, it can be further refined to determine different first parameter adjustment methods corresponding to different first conditions. Similarly, different second parameter adjustment methods can be determined corresponding to different second conditions, so as to determine which condition the bottleneck stage characteristic indicator meets and then execute the corresponding adjustment method.

[0048] Specifically, during implementation, a bottleneck stage characteristic index may be determined according to the bottleneck stages in a plurality of different data shuffling processes, and a first condition that the bottleneck stage characteristic index needs to satisfy when adjusting the first parameter and a second condition that the bottleneck stage characteristic index needs to satisfy when adjusting the second parameter may be determined according to the bottleneck stage characteristic index.

[0049] In this data shuffling, Figure 3 As shown, it may include:

[0050] S301, extracting characteristic indicators of the current bottleneck stage from the historical event log of the previous data shuffling process, and matching them with the first condition and the second condition respectively;

[0051] S302: If it is determined that the characteristic indicator of the current bottleneck stage meets the first condition, adjust the first parameter and execute the adjustment method corresponding to the first condition; or

[0052] S303: If it is determined that the characteristic indicator of the current bottleneck stage meets the second condition, adjust the second parameter and execute the adjustment method corresponding to the second condition.

[0053] In this embodiment, it is only necessary to extract the characteristic indicators of the current bottleneck stage from the historical event log of the previous data shuffling process. For example, each characteristic indicator of the current bottleneck stage can be represented as a one-dimensional vector and matched with the first condition and the second condition to determine the adjustment method of the parameters without generating a dependency tree, thereby saving computing resources and improving efficiency.

[0054] For example, the characteristic indicators of the bottleneck stage may specifically include one or more of the following Table 1:

[0055] Table 1

[0056]

[0057] When adjusting the first parameter, the first condition that the bottleneck stage characteristic indicator needs to meet may include:

[0058] If the maximum number of partitions in a single stage exceeds the first partition number threshold, the first parameter is doubled; or

[0059] If the maximum number of partitions in a single stage exceeds the second partition number threshold and the average running time of the stage is less than the first time threshold, then double the first parameter; or

[0060] If the average running time of the write task in a single stage is less than the second time threshold, and the maximum average running time of the write task in all stages is less than the third time threshold, and the average maximum memory overflow in all stages is less than the first memory overflow threshold, then double the first parameter; etc.

[0061] More first conditions are not given one by one in this embodiment. Different first conditions may involve different combinations of bottleneck stage characteristic indicators. Each bottleneck stage characteristic indicator may be configured with a corresponding threshold value, which will not be described here. It should be noted that different first conditions have priorities and can be matched with the current bottleneck stage characteristic indicator in order of priority. If a first condition is matched, the subsequent first conditions will no longer be matched.

[0062] Similarly, the second condition that the bottleneck stage characteristic indicator needs to satisfy when adjusting the second parameter may include:

[0063] If the maximum average running time of the read tasks in all stages is less than the fourth time threshold, and the maximum number of partitions is less than the second partition number threshold, the second parameter is reduced by half; or

[0064] If the maximum average running time of the read task in all stages is greater than the fifth time threshold, and the maximum number of partitions is less than the third partition number threshold, the second parameter is reduced by half; etc.

[0065] More second conditions are not given one by one in this embodiment. Different second conditions may involve different combinations of bottleneck stage characteristic indicators. Each bottleneck stage characteristic indicator may be configured with a corresponding threshold value, which will not be described in detail here.

[0066] In addition, when adjusting the first parameter or the second parameter, the first parameter or the second parameter can also be directly set to a preset value, rather than doubling or halving the original value. Of course, this adjustment method can correspond to specific conditions and is not limited here.

[0067] Corresponding to the control method of data shuffling in the above embodiment, Figure 4 A structural block diagram of a data shuffling control device provided in an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Figure 4 The data shuffling control device 400 includes: a generating unit 401, an adjusting unit 402, and an executing unit 403.

[0068] The generating unit 401 is used to determine the interdependencies between all read and write phases in the data shuffling process according to the historical event log of the data shuffling process of the distributed computing cluster, and generate a dependency tree;

[0069] The adjusting unit 402 is used to determine the bottleneck stage in the data shuffling process according to the dependency tree, and adjust the parameters of the data shuffling process according to the read and write status of the bottleneck stage; wherein the bottleneck stage is a read and write stage in which the amount of file data read and written is less than a preset data amount threshold and the number of read and write times is the largest;

[0070] The execution unit 403 is used to control the data shuffling process based on the adjusted parameters.

[0071] In one or more embodiments of the present disclosure, when determining the bottleneck stage in the data shuffling process according to the dependency tree, the adjustment unit 402 is configured to:

[0072] Determine the amount of file data read in each read phase in the dependency tree;

[0073] A set of candidate read phases whose file data volume is less than a preset data volume threshold is screened out from the dependency tree, and a read phase with the largest number of read and write times and a corresponding write phase in the set of candidate read phases are determined as the bottleneck phase.

[0074] In one or more embodiments of the present disclosure, when determining the read phase with the largest number of read and write times and the corresponding write phase in the candidate read phase set as the bottleneck phase, the adjustment unit 402 is configured to:

[0075] Obtain the number of read tasks of each read phase in the candidate read phase set and the number of write tasks of the corresponding write phase, and obtain the product of the number of read tasks of any read phase and the number of write tasks of the corresponding write phase as the number of read and write times of any read phase;

[0076] A read phase with the largest number of read and write times is determined from the candidate read phase set, and the read phase with the largest number of read and write times and a corresponding write phase are determined as the bottleneck phase.

[0077] In one or more embodiments of the present disclosure, when adjusting the parameters of the data shuffling process according to the read and write conditions of the bottleneck stage, the adjusting unit 402 is used to:

[0078] The first parameter and / or the second parameter of the data shuffling process is adjusted according to the read and write conditions of the bottleneck stage, wherein the first parameter is used to represent the maximum amount of file data written by each write task in any write stage of the data shuffling process, and the second parameter is used to represent the maximum number of read tasks in any read stage of the data shuffling process.

[0079] In one or more embodiments of the present disclosure, when the adjustment unit 402 adjusts the first parameter and / or the second parameter of the data shuffling process according to the read / write status of the bottleneck stage, it is configured to:

[0080] If the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, increasing the first parameter; or

[0081] If the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, the second parameter is reduced.

[0082] In one or more embodiments of the present disclosure, when the adjustment unit 402 increases the first parameter if the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, the adjustment unit 402 is configured to:

[0083] If the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, doubling the first parameter; or

[0084] When the adjusting unit 402 reduces the second parameter if the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, the adjusting unit 402 is configured to:

[0085] If the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, the second parameter is reduced by half.

[0086] In one or more embodiments of the present disclosure, when the adjustment unit 402 determines the bottleneck stage in the data shuffling process according to the dependency tree and adjusts the parameters of the data shuffling process according to the read and write status of the bottleneck stage, it is used to:

[0087] Determine a bottleneck stage characteristic indicator according to bottleneck stages in a plurality of different data shuffling processes, and determine, according to the bottleneck stage characteristic indicator, a first condition that the bottleneck stage characteristic indicator needs to satisfy when adjusting the first parameter, and a second condition that the bottleneck stage characteristic indicator needs to satisfy when adjusting the second parameter;

[0088] Extract the current bottleneck stage characteristic indicator from the historical event log and match them with the first condition and the second condition respectively; if it is determined that the current bottleneck stage characteristic indicator meets the first condition, adjust the first parameter; or, if it is determined that the current bottleneck stage characteristic indicator meets the second condition, adjust the second parameter.

[0089] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.

[0090] refer to Figure 5 , which shows a schematic diagram of the structure of an electronic device 500 suitable for implementing the embodiment of the present disclosure, and the electronic device 500 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0091] like Figure 5As shown, the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 to a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0092] Typically, the following devices may be connected to the I / O interface 505: input devices 506 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 508 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 509. The communication devices 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 5 The electronic device 500 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0093] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0094] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0095] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0096] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0097] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0099] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".

[0100] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0101] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0102] In a first aspect, according to one or more embodiments of the present disclosure, a method for controlling data shuffling is provided, comprising:

[0103] According to the historical event log of the data shuffling process of the distributed computing cluster, the interdependencies between all the read and write stages in the data shuffling process are determined, and a dependency tree is generated;

[0104] According to the dependency tree, the bottleneck stage in the data shuffling process is determined, and the parameters of the data shuffling process are adjusted according to the read and write conditions of the bottleneck stage; wherein the bottleneck stage is a read and write stage in which the amount of file data read and written is less than a preset data amount threshold and the number of read and write times is the largest;

[0105] Control the data shuffling process based on the tuned parameters.

[0106] According to one or more embodiments of the present disclosure, determining the bottleneck stage in the data shuffling process according to the dependency tree includes:

[0107] Determine the amount of file data read in each read phase in the dependency tree;

[0108] A set of candidate read phases whose file data volume is less than a preset data volume threshold is screened out from the dependency tree, and a read phase with the largest number of read and write times and a corresponding write phase in the set of candidate read phases are determined as the bottleneck phase.

[0109] According to one or more embodiments of the present disclosure, determining the read phase with the largest number of read and write times and the corresponding write phase in the candidate read phase set as the bottleneck phase includes:

[0110] Obtain the number of read tasks of each read phase in the candidate read phase set and the number of write tasks of the corresponding write phase, and obtain the product of the number of read tasks of any read phase and the number of write tasks of the corresponding write phase as the number of read and write times of any read phase;

[0111] A read phase with the largest number of read and write times is determined from the candidate read phase set, and the read phase with the largest number of read and write times and a corresponding write phase are determined as the bottleneck phase.

[0112] According to one or more embodiments of the present disclosure, adjusting the parameters of the data shuffling process according to the read and write conditions of the bottleneck stage includes:

[0113] The first parameter and / or the second parameter of the data shuffling process is adjusted according to the read and write conditions of the bottleneck stage, wherein the first parameter is used to represent the maximum amount of file data written by each write task in any write stage of the data shuffling process, and the second parameter is used to represent the maximum number of read tasks in any read stage of the data shuffling process.

[0114] According to one or more embodiments of the present disclosure, adjusting the first parameter and / or the second parameter of the data shuffling process according to the read / write status of the bottleneck stage includes:

[0115] If the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, increasing the first parameter; or

[0116] If the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, the second parameter is reduced.

[0117] According to one or more embodiments of the present disclosure, if the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, increasing the first parameter includes:

[0118] If the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, doubling the first parameter; or

[0119] If the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, reducing the second parameter includes:

[0120] If the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, the second parameter is reduced by half.

[0121] According to one or more embodiments of the present disclosure, determining the bottleneck stage in the data shuffling process according to the dependency tree, and adjusting the parameters of the data shuffling process according to the read and write conditions of the bottleneck stage, includes:

[0122] Determine a bottleneck stage characteristic indicator according to bottleneck stages in a plurality of different data shuffling processes, and determine, according to the bottleneck stage characteristic indicator, a first condition that the bottleneck stage characteristic indicator needs to satisfy when adjusting the first parameter, and a second condition that the bottleneck stage characteristic indicator needs to satisfy when adjusting the second parameter;

[0123] Extract the current bottleneck stage characteristic indicator from the historical event log and match them with the first condition and the second condition respectively; if it is determined that the current bottleneck stage characteristic indicator meets the first condition, adjust the first parameter; or, if it is determined that the current bottleneck stage characteristic indicator meets the second condition, adjust the second parameter.

[0124] In a second aspect, according to one or more embodiments of the present disclosure, a data shuffling control device is provided, comprising:

[0125] A generation unit, configured to determine the interdependencies between all read and write phases in the data shuffling process according to the historical event log of the data shuffling process of the distributed computing cluster, and generate a dependency tree;

[0126] An adjustment unit, configured to determine a bottleneck stage in the data shuffling process according to the dependency tree, and adjust parameters of the data shuffling process according to the read and write conditions in the bottleneck stage; wherein the bottleneck stage is a read and write stage in which the amount of file data read and written is less than a preset data amount threshold and the number of read and write times is the largest;

[0127] An execution unit is used to control the data shuffling process based on the adjusted parameters.

[0128] According to one or more embodiments of the present disclosure, when determining the bottleneck stage in the data shuffling process according to the dependency tree, the adjustment unit is configured to:

[0129] Determine the amount of file data read in each read phase in the dependency tree;

[0130] A set of candidate read phases whose file data volume is less than a preset data volume threshold is screened out from the dependency tree, and a read phase with the largest number of read and write times and a corresponding write phase in the set of candidate read phases are determined as the bottleneck phase.

[0131] According to one or more embodiments of the present disclosure, when the adjustment unit determines the read phase with the largest number of read and write times and the corresponding write phase in the candidate read phase set as the bottleneck phase, it is configured to:

[0132] Obtain the number of read tasks of each read phase in the candidate read phase set and the number of write tasks of the corresponding write phase, and obtain the product of the number of read tasks of any read phase and the number of write tasks of the corresponding write phase as the number of read and write times of any read phase;

[0133] A read phase with the largest number of read and write times is determined from the candidate read phase set, and the read phase with the largest number of read and write times and a corresponding write phase are determined as the bottleneck phase.

[0134] According to one or more embodiments of the present disclosure, when the adjustment unit adjusts the parameters of the data shuffling process according to the read and write conditions of the bottleneck stage, it is used to:

[0135] The first parameter and / or the second parameter of the data shuffling process is adjusted according to the read and write conditions of the bottleneck stage, wherein the first parameter is used to represent the maximum amount of file data written by each write task in any write stage of the data shuffling process, and the second parameter is used to represent the maximum number of read tasks in any read stage of the data shuffling process.

[0136] According to one or more embodiments of the present disclosure, when the adjustment unit adjusts the first parameter and / or the second parameter of the data shuffling process according to the read / write status of the bottleneck stage, it is used to:

[0137] If the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, increasing the first parameter; or

[0138] If the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, the second parameter is reduced.

[0139] According to one or more embodiments of the present disclosure, when the adjustment unit increases the first parameter if the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, the adjustment unit is configured to:

[0140] If the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, doubling the first parameter; or

[0141] The adjusting unit is used to reduce the second parameter when the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase:

[0142] If the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, the second parameter is reduced by half.

[0143] According to one or more embodiments of the present disclosure, when the adjustment unit determines the bottleneck stage in the data shuffling process according to the dependency tree and adjusts the parameters of the data shuffling process according to the read and write conditions of the bottleneck stage, it is used to:

[0144] Determine a bottleneck stage characteristic indicator according to bottleneck stages in a plurality of different data shuffling processes, and determine, according to the bottleneck stage characteristic indicator, a first condition that the bottleneck stage characteristic indicator needs to satisfy when adjusting the first parameter, and a second condition that the bottleneck stage characteristic indicator needs to satisfy when adjusting the second parameter;

[0145] Extract the current bottleneck stage characteristic indicator from the historical event log and match them with the first condition and the second condition respectively; if it is determined that the current bottleneck stage characteristic indicator meets the first condition, adjust the first parameter; or, if it is determined that the current bottleneck stage characteristic indicator meets the second condition, adjust the second parameter.

[0146] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0147] The memory stores computer-executable instructions;

[0148] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the data shuffling control method as described in the first aspect and various possible designs of the first aspect.

[0149] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the data shuffling control method described in the first aspect and various possible designs of the first aspect is implemented.

[0150] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the data shuffling control method described in the first aspect and various possible designs of the first aspect is implemented.

[0151] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0152] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0153] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A method for controlling data shuffling, It is characterized in that include: According to the historical event log of the data shuffling process of the distributed computing cluster, the interdependencies between all the reading and writing stages in the data shuffling process are determined, and a dependency tree is generated; According to the dependency tree, determining the bottleneck stage in the data shuffling process, and adjusting the parameters of the data shuffling process according to the read and write status of the bottleneck stage; The bottleneck stage is a read-write stage in which the file data volume read and written is less than a preset data volume threshold and the number of read and write times is the largest; Control the data shuffling process based on the tuned parameters.

2. The method according to claim 1, It is characterized in that Determining the bottleneck stage in the data shuffling process according to the dependency tree includes: Determine the amount of file data read in each read phase in the dependency tree; A set of candidate read phases whose file data volume is less than a preset data volume threshold is screened out from the dependency tree, and a read phase with the largest number of read and write times and a corresponding write phase in the set of candidate read phases are determined as the bottleneck phase.

3. The method according to claim 2, It is characterized in that The step of determining the read phase with the largest number of read and write times and the corresponding write phase in the candidate read phase set as the bottleneck phase comprises: Obtain the number of read tasks of each read phase in the candidate read phase set and the number of write tasks of the corresponding write phase, and obtain the product of the number of read tasks of any read phase and the number of write tasks of the corresponding write phase as the number of read and write times of any read phase; A read phase with the largest number of read and write times is determined from the candidate read phase set, and the read phase with the largest number of read and write times and a corresponding write phase are determined as the bottleneck phase.

4. The method according to any one of claims 1 to 3, It is characterized in that The step of adjusting the parameters of the data shuffling process according to the read / write status of the bottleneck stage includes: The first parameter and / or the second parameter of the data shuffling process is adjusted according to the read and write conditions of the bottleneck stage, wherein the first parameter is used to represent the maximum amount of file data written by each write task in any write stage of the data shuffling process, and the second parameter is used to represent the maximum number of read tasks in any read stage of the data shuffling process.

5. The method according to claim 4, It is characterized in that The step of adjusting the first parameter and / or the second parameter of the data shuffling process according to the read / write status of the bottleneck stage includes: If the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, increasing the first parameter; or If the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, the second parameter is reduced.

6. The method according to claim 5, It is characterized in that If the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, increasing the first parameter includes: If the number of write tasks in the write phase in the bottleneck phase is greater than the number of read tasks in the read phase, doubling the first parameter; or If the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, reducing the second parameter includes: If the number of write tasks in the write phase in the bottleneck phase is not greater than the number of read tasks in the read phase, the second parameter is reduced by half.

7. The method according to claim 4, It is characterized in that Determining the bottleneck stage in the data shuffling process according to the dependency tree, and adjusting the parameters of the data shuffling process according to the read and write status of the bottleneck stage, includes: Determine a bottleneck stage characteristic indicator according to bottleneck stages in a plurality of different data shuffling processes, and determine, according to the bottleneck stage characteristic indicator, a first condition that the bottleneck stage characteristic indicator needs to satisfy when adjusting the first parameter, and a second condition that the bottleneck stage characteristic indicator needs to satisfy when adjusting the second parameter; Extract the current bottleneck stage characteristic indicator from the historical event log and match them with the first condition and the second condition respectively; if it is determined that the current bottleneck stage characteristic indicator meets the first condition, adjust the first parameter; or, if it is determined that the current bottleneck stage characteristic indicator meets the second condition, adjust the second parameter.

8. A data shuffling control device, It is characterized in that include: A generation unit, configured to determine the interdependencies between all read and write phases in the data shuffling process according to the historical event log of the data shuffling process of the distributed computing cluster, and generate a dependency tree; An adjustment unit, configured to determine a bottleneck stage in the data shuffling process according to the dependency tree, and adjust parameters of the data shuffling process according to the read and write conditions of the bottleneck stage; The bottleneck stage is a read-write stage in which the file data volume read and written is less than a preset data volume threshold and the number of read and write times is the largest; An execution unit is used to control the data shuffling process based on the adjusted parameters.

9. An electronic device, It is characterized in that include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 7 is implemented.

11. A computer program product, It is characterized in that The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Optimization system and method for shuffling stage in Hadoop MapReduce

    CN110502337A

  • Data processing control method and device, computer equipment and storage medium

    CN116048817A

  • Data shuffling method, device and equipment, computer readable storage medium and product

    CN116932521A