Data washing control methods, equipment and storage media

Abstract

This disclosure provides a data shuffling control method, device, and storage medium. By analyzing historical event logs of the data shuffling process in a distributed computing cluster, the interdependencies between all read / write stages in the data shuffling process are determined, generating a dependency tree. Based on the dependency tree, bottleneck stages in the data shuffling process are identified, and the parameters of the data shuffling process are adjusted according to the read / write status of the bottleneck stage. The bottleneck stage is defined as the read / write stage where the amount of file data read / written is less than a preset data volume threshold and the number of read / write operations is the highest. The data shuffling process is controlled based on the adjusted parameters. This disclosure improves the efficiency and accuracy of data shuffling parameter optimization by analyzing historical event logs and automatically adjusting the parameters based on these bottleneck stages. This ensures that the performance of data shuffling is optimized without affecting the stability and runtime of the distributed computing cluster.

Description

Technical Field

[0001] This disclosure relates to the field of computer and network communication technology, and in particular to a control method, device and storage medium for data shuffling. Background Technology

[0002] In big data analytics systems, a data shuffle process is typically required. This process involves repartitioning and recombining the data across all computing nodes. The stability of the data shuffle process is crucial to the stability of big data analytics jobs. A significant factor affecting the stability of the data shuffle process is the generation of too many small files. Smaller files result in more read / write operations, putting greater pressure on the disk and impacting the execution of big data analytics jobs.

[0003] In existing technologies, adjusting the parameters of the data shuffling process can effectively reduce the number of files, thereby reducing the number of read / write operations and increasing file size. However, the adjustment of parameters in the data shuffling process in existing technologies is usually done manually based on experience, which results in poor efficiency and accuracy. Summary of the Invention

[0004] This disclosure provides a data shuffling control method, apparatus, and storage medium to improve the efficiency and accuracy of parameter optimization in the data shuffling process.

[0005] In a first aspect, embodiments of this disclosure provide a data shuffling control method, including:

[0006] Based on the historical event logs of the data shuffling process in the distributed computing cluster, determine the interdependencies between all read and write stages in the data shuffling process and generate a dependency tree.

[0007] Based on the dependency tree, the bottleneck stage in the data shuffling process is determined, and the parameters of the data shuffling process are adjusted according to the read and write status of the bottleneck stage; wherein the bottleneck stage is the read and write stage where the amount of file data read and written is less than the preset data amount threshold and the number of read and write operations is the largest.

[0008] The data shuffling process is controlled based on the adjusted parameters.

[0009] Secondly, embodiments of this disclosure provide a control device for data shuffling, comprising:

[0010] The generation unit is used to determine the interdependencies between all read and write stages in the data shuffling process based on the historical event logs of the data shuffling process of the distributed computing cluster, and to generate a dependency tree.

[0011] The adjustment unit is used to determine the bottleneck stage in the data shuffling process according to the dependency tree, and adjust the parameters of the data shuffling process according to the read and write status of the bottleneck stage; wherein the bottleneck stage is the read and write stage where the amount of file data read and written is less than a preset data amount threshold and the number of read and write operations is the largest.

[0012] The execution unit is used to control the data shuffling process based on the adjusted parameters.

[0013] Thirdly, embodiments of this disclosure provide an electronic device, including: at least one processor and a memory; the memory stores computer-executable instructions;

[0014] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the data shuffling control method as described in the first aspect and various possible designs of the first aspect.

[0015] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the data shuffling control method described in the first aspect and various possible designs of the first aspect.

[0016] Fifthly, embodiments of this disclosure provide a computer program product, including computer execution instructions, which, when executed by a processor, implement the data shuffling control method described in the first aspect and various possible designs of the first aspect.

[0017] The data shuffling control method, device, and storage medium provided in this disclosure determine the interdependencies between all read and write stages in the data shuffling process based on historical event logs of the distributed computing cluster's data shuffling process, generating a dependency tree. Based on the dependency tree, bottleneck stages in the data shuffling process are identified, and the parameters of the data shuffling process are adjusted according to the read / write status of the bottleneck stages. The bottleneck stage is defined as the read / write stage where the amount of file data being read / written is less than a preset data volume threshold and the number of read / write operations is the highest. The data shuffling process is controlled based on the adjusted parameters. This disclosure allows for the determination of bottleneck stages in the data shuffling process by analyzing historical event logs, and then automatically adjusts the parameters based on these bottleneck stages. This improves the efficiency and accuracy of parameter optimization, ensuring that the performance of data shuffling is optimized without affecting the stability and runtime of the distributed computing cluster, reducing the overall load of the distributed computing cluster, lowering the risk of job failure, and eliminating reliance on manual experience, thus enabling scalable application. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 A schematic diagram of a scenario for a data shuffling control method provided in an embodiment of this disclosure;

[0020] Figure 2 This is a schematic flowchart of a data shuffling control method provided in an embodiment of the present disclosure;

[0021] Figure 3 This is a schematic flowchart of a data shuffling control method provided in another embodiment of the present disclosure;

[0022] Figure 4 This is a structural block diagram of a data shuffling control device provided in an embodiment of the present disclosure;

[0023] Figure 5 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0025] In existing technologies, adjusting the parameters of the data shuffling process can effectively reduce the number of files, thereby reducing the number of read / write operations and increasing file size. However, the adjustment of parameters in the data shuffling process in existing technologies is usually done manually based on experience, which results in poor efficiency and accuracy.

[0026] To address the aforementioned technical problems, this disclosure provides a data shuffling control method. By analyzing historical event logs of the data shuffling process in a distributed computing cluster, the interdependencies between all read / write stages in the data shuffling process are determined, generating a dependency tree. Based on the dependency tree, bottleneck stages in the data shuffling process are identified, and the parameters of the data shuffling process are adjusted according to the read / write status of the bottleneck stages. The bottleneck stage is defined as the read / write stage where the amount of file data read / written is less than a preset data volume threshold and the number of read / write operations is the highest. The data shuffling process is controlled based on the adjusted parameters. This embodiment can determine the bottleneck stages of the data shuffling process by analyzing historical event logs, and then automatically adjust the parameters based on these bottleneck stages. This improves the efficiency and accuracy of parameter optimization, ensuring that the performance of data shuffling is optimized without affecting the stability and runtime of the distributed computing cluster, reducing the overall load of the distributed computing cluster, lowering the risk of job failure, and is not reliant on manual experience, allowing for scalable application.

[0027] The data shuffling control method of this disclosure is applicable to the data shuffling process in a distributed computing cluster, such as... Figure 1 As shown, the historical event logs of the data shuffling process of the distributed computing cluster can be obtained first, and a dependency tree can be generated based on the historical event logs. The bottleneck stage in the data shuffling process can be determined according to the dependency tree, and the parameters of the data shuffling process can be adjusted according to the bottleneck stage. Finally, the data shuffling process can be controlled based on the adjusted parameters.

[0028] The data shuffling control method of this disclosure will be described in detail below with reference to specific embodiments.

[0029] refer to Figure 2 , Figure 2 This is a schematic flowchart of a data shuffling control method according to an embodiment of the present disclosure. The method of this embodiment can be applied to electronic devices such as terminal devices or servers included in a distributed computing cluster. The data shuffling control method includes:

[0030] S201. Based on the historical event logs of the data shuffling process in the distributed computing cluster, determine the interdependencies between all read and write stages in the data shuffling process and generate a dependency tree.

[0031] In this embodiment, for distributed computing clusters such as Spark clusters, a data shuffle process is typically required. Data shuffle is a computational process that repartitions and recombines the data across all computing nodes, and this process is recorded in an event log (e.g., Spark EventLog). This embodiment can obtain the historical event logs of the distributed computing cluster, identify historical event logs related to the data cleaning process, and determine the interdependencies between all read / write stages during the data shuffle process based on these historical event logs.

[0032] The data shuffling process includes multiple write phases (also known as Map Tasks or Input Tasks) and read phases (also known as Reduce Tasks). The read phase is responsible for reading the results of the write phases, and the output of the read phase may be used as the input of another write phase. The historical event log of the data shuffling process can record the interdependencies between all read and write phases in the data shuffling process. For example, which read phase is responsible for reading the results of which write phase, and which read phase's output is input into which write phase. Based on the interdependencies between all read and write phases in the data shuffling process, a dependency tree can be generated to represent the interdependencies between all read and write phases in the data shuffling process.

[0033] Optionally, the amount of file data read in any Shuffle Read stage during the data shuffling process (i.e., the amount of file data read by each Reduce Task in the read stage) can be determined and denoted as the shuffle read size (which can be the average file data size of each read task in that read stage, i.e., the average shuffle read), and stored in the dependency tree. Optionally, the number of read tasks (Reduce Tasks) in any Shuffle Read stage during the data shuffling process can also be determined and denoted as R, and the number of write tasks (Map Tasks) in the corresponding Shuffle Write stage (where the output of the write stage serves as the input of the read stage) can be denoted as M, and these can also be stored in the dependency tree. Optionally, a specific data structure, such as shuffle read size / M / R, can be used to store the amount of file data read in any Shuffle Read stage, the number of read tasks in that read stage, and the number of write tasks in the corresponding write stage in the dependency tree for application in subsequent traversal processes.

[0034] S202. Based on the dependency tree, determine the bottleneck stage in the data shuffling process, and adjust the parameters of the data shuffling process according to the read and write status of the bottleneck stage; wherein the bottleneck stage is the read and write stage where the amount of file data read and written is less than the preset data amount threshold and the number of read and write operations is the largest.

[0035] In this embodiment, the dependency tree can be traversed to filter out the bottleneck stage in the data shuffling process. The bottleneck stage is the read / write stage where the amount of file data read and written is less than a preset data amount threshold and the number of read / write operations is the highest. Since the parameters of the data shuffling process are shared by all read / write stages, the parameters of the data shuffling process have the greatest impact on the bottleneck stage. Therefore, after filtering out the bottleneck stage, the reasons for its bottleneck status can be analyzed based on the actual situation of the bottleneck stage, and then the parameters of the data shuffling process can be adjusted so that the bottleneck stage will no longer become a bottleneck when subsequent data shuffling processes are performed based on the adjusted parameters.

[0036] Optionally, when filtering bottleneck stages, the amount of file data read by each read stage in the dependency tree can be determined, i.e., the shuffle read size mentioned above. The amount of file data read by each read stage in the dependency tree can be compared with a preset data size threshold (e.g., 20Kb). Read stages that are less than the preset data size threshold can be filtered out to obtain a set of candidate read stages. The read stage with the largest number of read and write operations can be selected from the set of candidate read stages, and this read stage and its corresponding write stage can be identified as the bottleneck stage.

[0037] The number of read / write operations in any read phase can be the product of the number of read tasks in that read phase (R) and the number of write tasks in the corresponding write phase (M). In other words, the number of read / write operations in any read phase is equal to M. R. In this embodiment, the number of read tasks in each read stage and the number of write tasks in the corresponding write stage can be obtained from the candidate read stage set. The product of the number of read tasks in any read stage and the number of write tasks in the corresponding write stage is determined as the read / write count of that read stage. The read stage with the largest read / write count is selected from the candidate read stage set, and that read stage and the corresponding write stage are determined as the bottleneck stage.

[0038] It should be noted that the shuffle read size, M, and R required in this embodiment can be pre-stored in the dependency tree as described in the previous embodiments, or they can be retrieved only when needed instead of being pre-stored in the dependency tree. Of course, the determination of the bottleneck stage is not limited to the above method. Other feasible methods can be used to filter out the read stage and the corresponding write stage with a sufficiently small file size and a sufficiently large number of read and write operations, and determine them as the bottleneck stage. This will not be elaborated here.

[0039] Furthermore, after identifying the bottleneck stage in the data shuffling process, the parameters of the data shuffling process can be adjusted based on the read / write status of the bottleneck stage (such as the number of read / write tasks). The adjusted parameters may include a first parameter and / or a second parameter. The first parameter is spark.sql.files.maxPartitionBytes, which represents the maximum amount of data written by each write task in any write stage of the data shuffling process (the first parameter controls the maximum amount of data in each partition when a file is split into multiple partitions during the data shuffling process, and one partition corresponds to one write task, which is equivalent to the maximum amount of data written by each write task). The second parameter is spark.sql.adaptive.maxNumPostShufflePartitions, which represents the maximum number of read tasks in any read stage of the data shuffling process (that is, the maximum number of Reduce Tasks included in any read stage).

[0040] Optionally, when adjusting the parameters of the data shuffling process, the aim is to reduce the number of read / write operations in the bottleneck stage. The number of read / write operations is the product of the number of read tasks in the read stage and the number of write tasks in the corresponding write stage within the bottleneck stage, i.e., M. Therefore, reducing the number of read / write operations during the bottleneck phase can be achieved by reducing the number of write tasks M and / or the number of read tasks R during the read phase. M is related to the first parameter; the larger the first parameter, the smaller M. R is related to the second parameter (R is less than or equal to the second parameter), meaning the larger the second parameter, the larger R. Therefore, adjusting the first parameter and / or the second phase can reduce the number of read / write operations M during the bottleneck phase. R, correspondingly, also increases the amount of file data read during the read phase.

[0041] Optionally, when adjusting the first and / or second parameters of the data shuffling process based on the bottleneck stage, the first or second parameter can be adjusted according to the relationship between the number of write tasks M in the write stage and the number of read tasks R in the read stage.

[0042] If the number of write tasks in the write phase of the bottleneck stage is greater than the number of read tasks in the read phase, then the first parameter is increased; optionally, the first parameter can be doubled. Alternatively, if the number of write tasks in the write phase of the bottleneck stage is not greater than the number of read tasks in the read phase, then the second parameter is decreased; optionally, the second parameter can be halved. By adjusting the first or second parameter as described above, the number of read and write operations in the bottleneck stage can be reduced, ensuring that the bottleneck stage no longer becomes a bottleneck in the next data shuffling process. However, a new bottleneck stage may appear in the next data shuffling process, and the above process can be repeated, thus achieving dynamic parameter adjustment. Furthermore, adjusting only one parameter at a time allows for targeted adjustments, improving the accuracy and efficiency of the adjustment process.

[0043] S203, Control the data shuffling process based on the adjusted parameters.

[0044] In this embodiment, after adjusting the parameters of the data shuffling process, the process can be controlled based on the adjusted parameters. It should be noted that since the dependency tree is determined based on the historical event logs of the data shuffling process, the data dependency tree remains unchanged when the logic of the data shuffling process remains the same. What changes are the file data volume and task quantity at each read / write stage in the data dependency tree. Therefore, the bottleneck stage will also change accordingly. In this embodiment, parameters can be adjusted based on the bottleneck stage of the previous data shuffling process to guide the subsequent data shuffling process, which can effectively adapt to data change trends.

[0045] The data shuffling control method provided in this embodiment determines the interdependencies between all read and write stages in the data shuffling process based on historical event logs of the distributed computing cluster, generating a dependency tree. Based on this dependency tree, it identifies bottleneck stages in the data shuffling process and adjusts the parameters of the data shuffling process according to the read / write status of these bottleneck stages. The bottleneck stage is defined as the read / write stage where the amount of file data being read / written is less than a preset data volume threshold and the number of read / write operations is the highest. The adjusted parameters control the data shuffling process. This embodiment can determine bottleneck stages in the data shuffling process by analyzing historical event logs, and then automatically adjust the parameters based on these bottleneck stages. This improves the efficiency and accuracy of parameter optimization, ensuring that the performance of data shuffling is optimized without affecting the stability and runtime of the distributed computing cluster, reducing the overall load of the distributed computing cluster, lowering the risk of job failure, and is scalable without relying on manual experience.

[0046] Based on any of the above embodiments, considering that the dependency tree may be too complex, require a lot of information, and may consume a lot of computing resources and take a long time, this embodiment considers simplifying the dependency tree. It can analyze the historical event logs of multiple different data shuffling processes to obtain the dependency tree of different data shuffling processes, and determine the bottleneck stage in different data shuffling processes through the method in the above embodiments. By analyzing the bottleneck stages of multiple different data shuffling processes, the characteristic indicators of the bottleneck stages can be determined. That is, different bottleneck stages have similar characteristic indicators. In particular, when it is necessary to adjust the first parameter based on the bottleneck stage, the characteristic indicators of the bottleneck stages have a certain similarity. When it is necessary to adjust the second parameter based on the bottleneck stage, the characteristic indicators of the bottleneck stages also have a certain similarity. The two situations can be distinguished by the characteristic indicators of the bottleneck stages. That is, when the characteristic indicators of the bottleneck stages meet certain conditions (denoted as the first condition), it is determined to adjust the first parameter. When the characteristic indicators of the bottleneck stages meet other certain conditions (denoted as the second condition), it is determined to adjust the second parameter.

[0047] Optionally, it can be further refined to determine different first parameter adjustment methods corresponding to different first conditions. Similarly, different second parameter adjustment methods corresponding to different second conditions can be determined to determine which condition the bottleneck stage characteristic indicators meet, and then execute the corresponding adjustment method.

[0048] In practice, bottleneck stage characteristic indicators can be determined based on the bottleneck stages in multiple different data shredding processes. Then, based on the bottleneck stage characteristic indicators, the first condition that the bottleneck stage characteristic indicators need to meet when adjusting the first parameter and the second condition that the bottleneck stage characteristic indicators need to meet when adjusting the second parameter can be determined.

[0049] In this data shuffling process, the specific details are as follows: Figure 3 As shown, it may include:

[0050] S301. Extract the current bottleneck stage feature indicators from the historical event logs of the previous data shuffling process, and match them with the first and second conditions respectively.

[0051] S302. If it is determined that the characteristic indicators of the current bottleneck stage meet the first condition, then adjust the first parameter and execute the adjustment method corresponding to the first condition; or

[0052] S303. If it is determined that the characteristic indicators of the current bottleneck stage meet the second condition, then adjust the second parameter and execute the adjustment method corresponding to the second condition.

[0053] In this embodiment, it is only necessary to extract the current bottleneck stage feature indicators from the historical event logs of the previous data shuffling process. For example, each current bottleneck stage feature indicator can be represented as a one-dimensional vector and matched with the first and second conditions to determine the adjustment method of the parameters, without having to generate a dependency tree, thus saving computing resources and improving efficiency.

[0054] For example, bottleneck stage characteristic indicators may include one or more of the indicators listed in Table 1 below:

[0055] Table 1

[0056]

[0057] The first condition that the bottleneck stage characteristic indicators must meet when adjusting the first parameter may include:

[0058] If the maximum number of partitions in a single stage exceeds the first partition number threshold, then the first parameter will be doubled; or

[0059] If the maximum number of partitions in a single stage exceeds the second partition number threshold, and the average runtime of that stage is less than the first time threshold, then the first parameter is doubled; or

[0060] If the average runtime of a single write task is less than the second time threshold, the maximum average runtime of write tasks across all stages is less than the third time threshold, and the maximum average memory overflow time across all stages is less than the first memory overflow threshold, then the first parameter will be doubled; and so on.

[0061] More first conditions will not be listed in this embodiment. Different first conditions may involve different combinations of bottleneck stage characteristic indicators, and each bottleneck stage characteristic indicator may be configured with a corresponding threshold, which will not be elaborated here. It should be noted that different first conditions have priorities and can be matched with the current bottleneck stage characteristic indicators in order of priority. If a first condition is matched, the matching of subsequent first conditions will not be performed.

[0062] Similarly, the second condition that the bottleneck stage characteristic indicators must meet when adjusting the second parameter may include:

[0063] If the maximum average runtime of read tasks across all stages is less than the fourth time threshold, and the maximum number of partitions is less than the second partition number threshold, then the second parameter is reduced by half; or

[0064] If the maximum average runtime of read tasks across all stages exceeds the fifth time threshold, and the maximum number of partitions is less than the third partition number threshold, then the second parameter is reduced by half; and so on.

[0065] More second conditions will not be listed one by one in this embodiment. Different second conditions may involve different combinations of bottleneck stage characteristic indicators, and each bottleneck stage characteristic indicator may be configured with a corresponding threshold, which will not be elaborated here.

[0066] In addition, when adjusting the first or second parameter, you can directly set the first or second parameter to a preset value instead of doubling or halving it. Of course, this adjustment method can be used for specific conditions, and there are no restrictions here.

[0067] Corresponding to the data shuffling control method in the above embodiments, Figure 4 This is a structural block diagram of a data shuffling control device provided in an embodiment of this disclosure. For ease of explanation, only the parts relevant to the embodiments of this disclosure are shown. (Refer to...) Figure 4 The data shuffling control device 400 includes: a generation unit 401, an adjustment unit 402, and an execution unit 403.

[0068] Among them, the generation unit 401 is used to determine the mutual dependencies between all read and write stages in the data shuffling process based on the historical event logs of the data shuffling process of the distributed computing cluster, and generate a dependency tree.

[0069] The adjustment unit 402 is used to determine the bottleneck stage in the data shuffling process according to the dependency tree, and adjust the parameters of the data shuffling process according to the read and write status of the bottleneck stage; wherein the bottleneck stage is the read and write stage where the amount of file data read and written is less than a preset data amount threshold and the number of read and write operations is the largest.

[0070] Execution unit 403 is used to control the data shuffling process based on the adjusted parameters.

[0071] In one or more embodiments of this disclosure, when the adjustment unit 402 determines the bottleneck stage in the data shuffling process based on the dependency tree, it is configured to:

[0072] Determine the amount of file data read in each read stage of the dependency tree;

[0073] From the dependency tree, a set of candidate read stages with file data volume less than a preset data volume threshold is selected, and the read stage with the largest number of read and write operations and its corresponding write stage in the candidate read stage set are identified as the bottleneck stage.

[0074] In one or more embodiments of this disclosure, when the adjustment unit 402 determines the read stage with the highest number of read / write operations and the corresponding write stage in the candidate read stage set as the bottleneck stage, it is used to:

[0075] Obtain the number of read tasks for each read stage in the candidate read stage set and the number of write tasks for the corresponding write stage, and determine the number of read and write operations for any read stage by multiplying the number of read tasks for any read stage by the number of write tasks for the corresponding write stage.

[0076] The read stage with the highest number of read / write operations is determined from the set of candidate read stages, and the read stage with the highest number of read / write operations and its corresponding write stage are determined as the bottleneck stage.

[0077] In one or more embodiments of this disclosure, when the adjustment unit 402 adjusts the parameters of the data shuffling process according to the read / write status of the bottleneck stage, it is used to:

[0078] The first and / or second parameters of the data shuffling process are adjusted according to the read and write status of the bottleneck stage, wherein the first parameter is used to represent the maximum amount of file data written by each write task in any write stage of the data shuffling process, and the second parameter is used to represent the maximum number of read tasks in any read stage of the data shuffling process.

[0079] In one or more embodiments of this disclosure, when the adjustment unit 402 adjusts the first and / or second parameters of the data shuffling process according to the read / write status of the bottleneck stage, it is used to:

[0080] If the number of write tasks in the write phase of the bottleneck stage is greater than the number of read tasks in the read phase, then increase the first parameter; or

[0081] If the number of write tasks in the write phase of the bottleneck phase is not greater than the number of read tasks in the read phase, then the second parameter is reduced.

[0082] In one or more embodiments of this disclosure, when the adjustment unit 402 increases the first parameter if the number of write tasks in the write phase is greater than the number of read tasks in the read phase during the bottleneck phase, it is used to:

[0083] If the number of write tasks in the write phase of the bottleneck phase is greater than the number of read tasks in the read phase, then the first parameter is doubled; or

[0084] When the adjustment unit 402 lowers the second parameter if the number of write tasks in the write phase during the bottleneck phase is not greater than the number of read tasks in the read phase, it is used to:

[0085] If the number of write tasks in the write phase of the bottleneck phase is not greater than the number of read tasks in the read phase, then the second parameter is reduced by half.

[0086] In one or more embodiments of this disclosure, when the adjustment unit 402 determines the bottleneck stage in the data shuffling process based on the dependency tree and adjusts the parameters of the data shuffling process based on the read / write status of the bottleneck stage, it is configured to:

[0087] Based on the bottleneck stages in multiple different data shuffling processes, bottleneck stage characteristic indicators are determined, and based on the bottleneck stage characteristic indicators, a first condition that the bottleneck stage characteristic indicators need to satisfy when adjusting the first parameter and a second condition that the bottleneck stage characteristic indicators need to satisfy when adjusting the second parameter are determined.

[0088] Extract the current bottleneck stage feature indicators from the historical event logs and match them with the first condition and the second condition respectively. If it is determined that the current bottleneck stage feature indicators meet the first condition, then adjust the first parameter; or, if it is determined that the current bottleneck stage feature indicators meet the second condition, then adjust the second parameter.

[0089] The device provided in this embodiment can be used to execute the technical solutions of the above method embodiments. Its implementation principle and technical effect are similar, and will not be described again here.

[0090] refer to Figure 5 The diagram illustrates a structural schematic of an electronic device 500 suitable for implementing embodiments of the present disclosure. The electronic device 500 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital radio receivers, personal digital assistants (PDAs), portable Android devices (PADs), portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0091] like Figure 5As shown, the electronic device 500 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0092] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0093] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0094] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0095] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0096] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0097] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0098] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0099] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0100] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0101] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0102] In a first aspect, according to one or more embodiments of this disclosure, a data shuffling control method is provided, comprising:

[0103] Based on the historical event logs of the data shuffling process in the distributed computing cluster, determine the interdependencies between all read and write stages in the data shuffling process and generate a dependency tree.

[0104] Based on the dependency tree, the bottleneck stage in the data shuffling process is determined, and the parameters of the data shuffling process are adjusted according to the read and write status of the bottleneck stage; wherein the bottleneck stage is the read and write stage where the amount of file data read and written is less than the preset data amount threshold and the number of read and write operations is the largest.

[0105] The data shuffling process is controlled based on the adjusted parameters.

[0106] According to one or more embodiments of this disclosure, determining the bottleneck stage in the data shuffling process based on the dependency tree includes:

[0107] Determine the amount of file data read in each read stage of the dependency tree;

[0108] From the dependency tree, a set of candidate read stages with file data volume less than a preset data volume threshold is selected, and the read stage with the largest number of read and write operations and its corresponding write stage in the candidate read stage set are identified as the bottleneck stage.

[0109] According to one or more embodiments of this disclosure, determining the read stage with the highest number of read / write operations and its corresponding write stage from the set of candidate read stages as the bottleneck stage includes:

[0110] Obtain the number of read tasks for each read stage in the candidate read stage set and the number of write tasks for the corresponding write stage, and determine the number of read and write operations for any read stage by multiplying the number of read tasks for any read stage by the number of write tasks for the corresponding write stage.

[0111] The read stage with the highest number of read / write operations is determined from the set of candidate read stages, and the read stage with the highest number of read / write operations and its corresponding write stage are determined as the bottleneck stage.

[0112] According to one or more embodiments of this disclosure, adjusting the parameters of the data shuffling process based on the read / write status of the bottleneck stage includes:

[0113] The first and / or second parameters of the data shuffling process are adjusted according to the read and write status of the bottleneck stage, wherein the first parameter is used to represent the maximum amount of file data written by each write task in any write stage of the data shuffling process, and the second parameter is used to represent the maximum number of read tasks in any read stage of the data shuffling process.

[0114] According to one or more embodiments of this disclosure, adjusting the first and / or second parameters of the data shuffling process based on the read / write status of the bottleneck stage includes:

[0115] If the number of write tasks in the write phase of the bottleneck stage is greater than the number of read tasks in the read phase, then increase the first parameter; or

[0116] If the number of write tasks in the write phase of the bottleneck phase is not greater than the number of read tasks in the read phase, then the second parameter is reduced.

[0117] According to one or more embodiments of this disclosure, increasing the first parameter if the number of write tasks in the write phase of the bottleneck phase is greater than the number of read tasks in the read phase includes:

[0118] If the number of write tasks in the write phase of the bottleneck phase is greater than the number of read tasks in the read phase, then the first parameter is doubled; or

[0119] If the number of write tasks in the write phase of the bottleneck phase is not greater than the number of read tasks in the read phase, then the second parameter is reduced, including:

[0120] If the number of write tasks in the write phase of the bottleneck phase is not greater than the number of read tasks in the read phase, then the second parameter is reduced by half.

[0121] According to one or more embodiments of this disclosure, determining the bottleneck stage in the data shuffling process based on the dependency tree, and adjusting the parameters of the data shuffling process according to the read / write status of the bottleneck stage, includes:

[0122] Based on the bottleneck stages in multiple different data shuffling processes, bottleneck stage characteristic indicators are determined, and based on the bottleneck stage characteristic indicators, a first condition that the bottleneck stage characteristic indicators need to satisfy when adjusting the first parameter and a second condition that the bottleneck stage characteristic indicators need to satisfy when adjusting the second parameter are determined.

[0123] Extract the current bottleneck stage feature indicators from the historical event logs and match them with the first condition and the second condition respectively. If it is determined that the current bottleneck stage feature indicators meet the first condition, then adjust the first parameter; or, if it is determined that the current bottleneck stage feature indicators meet the second condition, then adjust the second parameter.

[0124] Secondly, according to one or more embodiments of this disclosure, a data shuffling control device is provided, comprising:

[0125] The generation unit is used to determine the interdependencies between all read and write stages in the data shuffling process based on the historical event logs of the data shuffling process of the distributed computing cluster, and to generate a dependency tree.

[0126] The adjustment unit is used to determine the bottleneck stage in the data shuffling process according to the dependency tree, and adjust the parameters of the data shuffling process according to the read and write status of the bottleneck stage; wherein the bottleneck stage is the read and write stage where the amount of file data read and written is less than a preset data amount threshold and the number of read and write operations is the largest.

[0127] The execution unit is used to control the data shuffling process based on the adjusted parameters.

[0128] According to one or more embodiments of this disclosure, when the adjustment unit determines a bottleneck stage in the data shuffling process based on the dependency tree, it is configured to:

[0129] Determine the amount of file data read in each read stage of the dependency tree;

[0130] From the dependency tree, a set of candidate read stages with file data volume less than a preset data volume threshold is selected, and the read stage with the largest number of read and write operations and its corresponding write stage in the candidate read stage set are identified as the bottleneck stage.

[0131] According to one or more embodiments of this disclosure, when the adjustment unit determines the read stage with the highest number of read / write operations and the corresponding write stage in the set of candidate read stages as the bottleneck stage, it is configured to:

[0132] Obtain the number of read tasks for each read stage in the candidate read stage set and the number of write tasks for the corresponding write stage, and determine the number of read and write operations for any read stage by multiplying the number of read tasks for any read stage by the number of write tasks for the corresponding write stage.

[0133] The read stage with the highest number of read / write operations is determined from the set of candidate read stages, and the read stage with the highest number of read / write operations and its corresponding write stage are determined as the bottleneck stage.

[0134] According to one or more embodiments of this disclosure, when the adjustment unit adjusts the parameters of the data shuffling process based on the read / write status of the bottleneck stage, it is used to:

[0135] The first and / or second parameters of the data shuffling process are adjusted according to the read and write status of the bottleneck stage, wherein the first parameter is used to represent the maximum amount of file data written by each write task in any write stage of the data shuffling process, and the second parameter is used to represent the maximum number of read tasks in any read stage of the data shuffling process.

[0136] According to one or more embodiments of this disclosure, when the adjustment unit adjusts the first and / or second parameters of the data shuffling process based on the read / write status of the bottleneck stage, it is used to:

[0137] If the number of write tasks in the write phase of the bottleneck stage is greater than the number of read tasks in the read phase, then increase the first parameter; or

[0138] If the number of write tasks in the write phase of the bottleneck phase is not greater than the number of read tasks in the read phase, then the second parameter is reduced.

[0139] According to one or more embodiments of this disclosure, when the adjustment unit increases the first parameter if the number of write tasks in the write phase is greater than the number of read tasks in the read phase during the bottleneck phase, it is used to:

[0140] If the number of write tasks in the write phase of the bottleneck phase is greater than the number of read tasks in the read phase, then the first parameter is doubled; or

[0141] When the adjustment unit lowers the second parameter if the number of write tasks in the write phase during the bottleneck phase is not greater than the number of read tasks in the read phase, it is used to:

[0142] If the number of write tasks in the write phase of the bottleneck phase is not greater than the number of read tasks in the read phase, then the second parameter is reduced by half.

[0143] According to one or more embodiments of this disclosure, when the adjustment unit determines the bottleneck stage in the data shuffling process based on the dependency tree and adjusts the parameters of the data shuffling process according to the read / write status of the bottleneck stage, it is configured to:

[0144] Based on the bottleneck stages in multiple different data shuffling processes, bottleneck stage characteristic indicators are determined, and based on the bottleneck stage characteristic indicators, a first condition that the bottleneck stage characteristic indicators need to satisfy when adjusting the first parameter and a second condition that the bottleneck stage characteristic indicators need to satisfy when adjusting the second parameter are determined.

[0145] Extract the current bottleneck stage feature indicators from the historical event logs and match them with the first condition and the second condition respectively. If it is determined that the current bottleneck stage feature indicators meet the first condition, then adjust the first parameter; or, if it is determined that the current bottleneck stage feature indicators meet the second condition, then adjust the second parameter.

[0146] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0147] The memory stores computer-executed instructions;

[0148] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the data shuffling control method as described in the first aspect and various possible designs of the first aspect.

[0149] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the data shuffling control method described in the first aspect and various possible designs of the first aspect is implemented.

[0150] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including computer execution instructions that, when executed by a processor, implement the data shuffling control method described in the first aspect and various possible designs of the first aspect.

[0151] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0152] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0153] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A control method for data shuffling, characterized in that, include: Based on the historical event logs of the data shuffling process in the distributed computing cluster, determine the interdependencies between all read and write stages in the data shuffling process and generate a dependency tree. Based on the dependency tree, bottleneck stages in the data shuffling process are identified, and the reasons why a bottleneck stage becomes a bottleneck are determined based on the read / write status of that bottleneck stage. The parameters of the data shuffling process are adjusted based on the reasons why a bottleneck stage becomes a bottleneck to reduce the number of read / write operations in that bottleneck stage. The bottleneck stage is the read / write stage where the amount of file data read is less than a preset data volume threshold and the number of read / write operations is the highest. The parameters include at least one of a first parameter and a second parameter. The first parameter represents the maximum amount of file data written by each write task in any write stage, and the second parameter represents the maximum number of read tasks in any read stage. The data shuffling process is controlled based on the adjusted parameters. The adjustment of parameters in the data shuffling process based on the reasons why the bottleneck stage becomes a bottleneck includes: The first parameter and / or the second parameter are adjusted based on the relationship between the number of write tasks in the write phase and the number of read tasks in the read phase of the bottleneck phase; or Bottleneck stage characteristic indicators are determined based on the bottleneck stages in multiple different data shredding processes, and the first parameter and / or the second parameter are adjusted based on the bottleneck stage characteristic indicators and the reasons why the bottleneck stage becomes a bottleneck.

2. The method according to claim 1, characterized in that, The step of determining the bottleneck stage in the data shuffling process based on the dependency tree includes: Determine the amount of file data read in each read stage of the dependency tree; From the dependency tree, a set of candidate read stages with file data volume less than a preset data volume threshold is selected, and the read stage with the largest number of read and write operations and its corresponding write stage in the candidate read stage set are identified as the bottleneck stage.

3. The method according to claim 2, characterized in that, The step of identifying the read stage with the highest number of read / write operations and its corresponding write stage from the candidate read stage set as the bottleneck stage includes: Obtain the number of read tasks for each read stage in the candidate read stage set and the number of write tasks for the corresponding write stage, and determine the number of read and write operations for any read stage by multiplying the number of read tasks for any read stage by the number of write tasks for the corresponding write stage. The read stage with the highest number of read / write operations is determined from the set of candidate read stages, and the read stage with the highest number of read / write operations and its corresponding write stage are determined as the bottleneck stage.

4. The method according to claim 1, characterized in that, Adjusting the first parameter and / or the second parameter based on the relationship between the number of write tasks in the write phase and the number of read tasks in the read phase of the bottleneck phase includes: If the number of write tasks in the write phase of the bottleneck stage is greater than the number of read tasks in the read phase, then increase the first parameter; or If the number of write tasks in the write phase of the bottleneck phase is not greater than the number of read tasks in the read phase, then the second parameter is reduced.

5. The method according to claim 4, characterized in that, If the number of write tasks in the write phase of the bottleneck phase is greater than the number of read tasks in the read phase, then the first parameter is increased, including: If the number of write tasks in the write phase of the bottleneck phase is greater than the number of read tasks in the read phase, then the first parameter is doubled; or If the number of write tasks in the write phase of the bottleneck phase is not greater than the number of read tasks in the read phase, then the second parameter is reduced, including: If the number of write tasks in the write phase of the bottleneck phase is not greater than the number of read tasks in the read phase, then the second parameter is reduced by half.

6. The method according to claim 1, characterized in that, The adjustment of the first parameter and / or the second parameter based on the bottleneck stage characteristic indicators and the reasons why the bottleneck stage becomes a bottleneck includes: Based on the bottleneck stage characteristic indicators, determine the first condition that the bottleneck stage characteristic indicators need to satisfy when adjusting the first parameter, and the second condition that the bottleneck stage characteristic indicators need to satisfy when adjusting the second parameter; Extract the current bottleneck stage feature indicators from the historical event logs and match them with the first condition and the second condition respectively. If it is determined that the current bottleneck stage feature indicators meet the first condition, then adjust the first parameter; or, if it is determined that the current bottleneck stage feature indicators meet the second condition, then adjust the second parameter.

7. A control device for data shuffling, characterized in that, include: The generation unit is used to determine the interdependencies between all read and write stages in the data shuffling process based on the historical event logs of the data shuffling process of the distributed computing cluster, and to generate a dependency tree. An adjustment unit is configured to determine the bottleneck stage in the data shuffling process based on the dependency tree, determine the reason why the bottleneck stage becomes a bottleneck based on the read / write status of the bottleneck stage, and adjust the parameters of the data shuffling process based on the reason why the bottleneck stage becomes a bottleneck, so as to reduce the number of read / write operations in the bottleneck stage; wherein the bottleneck stage is the read / write stage where the amount of file data read is less than a preset data amount threshold and the number of read / write operations is the largest, and the parameters include at least one of a first parameter and a second parameter, wherein the first parameter is used to represent the maximum amount of file data written by each write task in any write stage, and the second parameter is used to represent the maximum number of read tasks in any read stage. The execution unit is used to control the data shuffling process based on the adjusted parameters. The adjustment of parameters in the data shuffling process based on the reasons why the bottleneck stage becomes a bottleneck includes: The first parameter and / or the second parameter are adjusted based on the relationship between the number of write tasks in the write phase and the number of read tasks in the read phase of the bottleneck phase; or Bottleneck stage characteristic indicators are determined based on the bottleneck stages in multiple different data shredding processes, and the first parameter and / or the second parameter are adjusted based on the bottleneck stage characteristic indicators and the reasons why the bottleneck stage becomes a bottleneck.

8. An electronic device, characterized in that, include: At least one processor and memory; The memory stores computer-executed instructions; The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes computer execution instructions, which, when executed by a processor, implement the method as described in any one of claims 1-6.

Images