Method, device, equipment and medium for adjusting parallelism of data processing tasks

Abstract

The present invention discloses a method, device, equipment and medium for adjusting the parallelism of data processing tasks. The method determines the estimated processing time of a currently executed job of a data processing system and the estimated processing time of a downstream job corresponding to the currently executed job, and judges whether the execution time of the system for the downstream job has timed out based on the estimated processing time. If so, the execution parallelism corresponding to the downstream job is adjusted based on the current throughput of the system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job and the job dependency, so as to realize dynamic adjustment of the execution parallelism of the downstream job, ensure the timely completion of the downstream job through dynamic expansion, and solve the technical problem of downstream job timeout caused by upstream job delay.

Description

Technical Field

[0001] The present invention relates to the field of big data processing technology, and in particular to a method, device, equipment and medium for adjusting the parallelism of a data processing task. Background Art

[0002] Generally, data processing is "collecting and processing data items to produce meaningful information." In this sense, it can be considered a subset of information processing, where changes (processing) in information can be detected by any means. Data processing tasks, in big data systems, usually refer to big data jobs that submit data processing. In big data processing systems, a job is a collection of parallel tasks. Generally speaking, a processing job is divided into several processing stages (such as map stage and reduce stage), and for each stage, several identical processing tasks are executed in parallel.

[0003] Taking Hadoop as an example, the process of data processing task scheduling is briefly introduced. The main purpose of Hadoop in the early days was to run large batch jobs, such as web indexing and log mining. Users submit jobs to queues, and the cluster runs them in sequence. When a job is submitted to Hadoop's management node, it is divided into several stages (such as map stage and reduce stage), and each stage is divided into several tasks. Tasks are scheduled based on priority and the resources available on each processing node in the cluster, including first-in-first-out, fair scheduling, and capacity-related scheduling strategies.

[0004] However, in the existing technical solutions, a fixed degree of parallelism is usually set for each job. If the upstream job times out, the downstream job uses the fixed parallelism quota without any additional processing, which causes the downstream job to time out as well. Summary of the invention

[0005] The present invention provides a method, device, equipment and medium for adjusting the parallelism of data processing tasks, which realizes dynamic adjustment of the execution parallelism of downstream jobs and solves the technical problem of downstream job timeout caused by upstream job delay.

[0006] According to one aspect of the present invention, a method for adjusting the parallelism of a data processing task is provided, the method comprising:

[0007] Determine an estimated processing time of a currently executed job of a data processing system and a downstream job corresponding to the currently executed job;

[0008] Determining whether the execution time of the data processing system for the downstream job has exceeded based on the estimated processing time;

[0009] If so, the execution parallelism corresponding to the downstream job is adjusted based on the current throughput of the data processing system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and the job dependency.

[0010] Optionally, determining a currently executed job of the data processing system and an estimated processing time of a downstream job corresponding to the currently executed job includes:

[0011] Get historical job statistics of data processing system;

[0012] An estimated processing time of a currently executed job of the data processing system and a downstream job corresponding to the currently executed job is determined based on the historical job statistical information.

[0013] Optionally, determining the estimated processing time of a currently executed job of the data processing system and a downstream job corresponding to the currently executed job based on the historical job statistical information includes:

[0014] Determine the throughput of the historical task corresponding to the historical job based on the historical job statistical information;

[0015] Determine the throughput of the currently executed job and the downstream job corresponding to the currently executed job based on the throughput of the historical task;

[0016] According to the throughputs of the currently executed job and the downstream job, the estimated processing time of the currently executed job and the downstream job is determined.

[0017] Optionally, judging, based on the estimated processing time, whether the execution time of the downstream job corresponding to the currently executed job by the data processing system has timed out includes:

[0018] Obtaining the preset processing time of the currently executed job;

[0019] If the difference between the estimated processing time of the currently executed job and the preset processing time is greater than a preset threshold, and the estimated processing time is greater than the preset processing time, it is determined that the execution time of the downstream job corresponding to the currently executed job of the data processing system has timed out.

[0020] Optionally, the method further includes:

[0021] Determining whether the execution time of the downstream job by the data processing system is advanced based on the estimated processing time;

[0022] If so, the execution parallelism corresponding to the downstream job is adjusted based on the current throughput, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and the job dependency.

[0023] Optionally, the method further includes:

[0024] Obtaining a directed acyclic graph corresponding to all jobs of the data processing system, wherein the directed acyclic graph includes connection direction information;

[0025] The job dependency is determined based on the connection direction information.

[0026] Optionally, the method further includes:

[0027] Determining whether the currently executed job is the final executed job of the data processing system;

[0028] If not, after the current execution job is completed, the next execution job corresponding to the current execution job is determined as the current execution job, and the operation of performing judgment and determining the estimated processing time of the current execution job and the downstream job corresponding to the current execution job is returned.

[0029] According to another aspect of the present invention, a device for adjusting the parallelism of a data processing task is provided, the device comprising:

[0030] A time prediction module, used to determine the estimated processing time of a currently executed job of the data processing system and a downstream job corresponding to the currently executed job;

[0031] A timeout determination module, configured to determine whether the execution time of the data processing system for the downstream job has timed out based on the estimated processing time;

[0032] The downstream adjustment module is used to adjust the execution parallelism corresponding to the downstream job based on the current throughput of the data processing system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job and the job dependency.

[0033] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0034] at least one processor; and

[0035] a memory communicatively connected to the at least one processor; wherein,

[0036] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the method for adjusting the parallelism of a data processing task described in any embodiment of the present invention.

[0037] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for adjusting the parallelism of a data processing task described in any embodiment of the present invention when executed.

[0038] The technical solution of the embodiment of the present invention determines the estimated processing time of the currently executed job of the data processing system and the estimated processing time of the downstream job corresponding to the currently executed job, and judges whether the system's execution time for the downstream job has timed out based on the estimated processing time. If so, the execution parallelism corresponding to the downstream job is adjusted based on the current throughput of the system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and the job dependencies, so as to realize dynamic adjustment of the execution parallelism of the downstream job, ensure the timely completion of the downstream job through dynamic expansion, and solve the technical problem of downstream job timeout caused by upstream job delay.

[0039] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0041] Figure 1 It is a flowchart of a method for adjusting the degree of parallelism of a data processing task provided in the first embodiment of the present invention;

[0042] Figure 2A It is a flowchart of a method for adjusting the degree of parallelism of a data processing task provided in the second embodiment of the present invention;

[0043] Figure 2B It is a flowchart of a method for adjusting the degree of parallelism of a data processing task provided in the second embodiment of the present invention;

[0044] Figure 3 It is a flowchart of a method for adjusting the degree of parallelism of a data processing task provided in Embodiment 3 of the present invention;

[0045] Figure 4 It is a structural diagram of a device for adjusting the degree of parallelism of a data processing task provided by a fourth embodiment of the present invention;

[0046] Figure 5 It is a structural schematic diagram of an electronic device provided in Embodiment 5 of the present invention. DETAILED DESCRIPTION

[0047] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0048] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0049] Embodiment 1

[0050] Figure 1 The present invention provides a flowchart of a method for adjusting the parallelism of a data processing task in accordance with the first embodiment of the present invention. This embodiment can be applied to the case where the execution parallelism of downstream jobs is dynamically adjusted according to the estimated processing time of each job during the process of processing each job by the data processing system. The method can be executed by a device for adjusting the parallelism of data processing tasks. The device for adjusting the parallelism of data processing tasks can be implemented in the form of hardware and / or software. The device for adjusting the parallelism of data processing tasks can be configured in electronic devices such as mobile phones, computers, and tablets. Figure 1 As shown, the method includes:

[0051] S110: Determine a currently executed job of a data processing system and an estimated processing time of a downstream job corresponding to the currently executed job.

[0052] Specifically, the data processing system can be a system that executes each task in a data processing job. In the data processing system, each data processing job can be understood as a collection of parallel tasks. Generally speaking, a data processing job can be divided into several processing stages (such as map stage and reduce stage). For each stage, several identical processing tasks are executed in parallel.

[0053] In this embodiment, during the process of the data processing system executing various tasks of the data processing job, the current job being executed by the data processing system can be determined in real time, that is, the job to which the currently processed task belongs can be determined; further, the time for the data processing system to complete the current job being executed, that is, the estimated processing time, can be estimated; and the time for the data processing system to complete the downstream job corresponding to the current job being executed can be estimated. The downstream job corresponding to the current job being executed can be a job that can only be executed after the completion of the current job being executed.

[0054] For example, the estimated processing time of the currently executed job and downstream jobs can be determined by the historical average execution time of each job. Alternatively, the estimated processing time of the currently executed job and downstream jobs can be determined by the current throughput of the data processing system, the available resources of the system, and the amount of data to be processed corresponding to the currently executed job.

[0055] S120: Determine whether the execution time of the data processing system for the downstream job has timed out based on the estimated processing time.

[0056] In this embodiment, after determining the estimated processing time of the currently executed job and the downstream job, further, it is possible to judge whether the downstream job will be executed overtime based on the estimated processing time. For example, a corresponding processing completion time can be set in advance for each job. After determining the estimated processing time of the currently executed job and the downstream job, the estimated completion time of the downstream job is calculated based on the estimated processing time of the currently executed job and the downstream job, and the preset processing completion time corresponding to the downstream job is compared with the estimated completion time. If the estimated completion time exceeds the processing completion time, it can be determined that the downstream job will be executed overtime.

[0057] In an optional embodiment, a corresponding preset processing time may be set in advance for each data processing job, and then by comparing the preset processing time corresponding to the currently executing job with the expected processing time, it can be determined whether the currently executing job has timed out. If so, it can be determined that the execution time of the downstream job corresponding to the currently executing job has exceeded.

[0058] Exemplarily, judging whether the execution time of the downstream job corresponding to the currently executing job by the data processing system has timed out based on the estimated processing time includes: obtaining the preset processing time of the currently executing job; if the difference between the estimated processing time of the currently executing job and the preset processing time is greater than a preset threshold, and the estimated processing time is greater than the preset processing time, then determining that the execution time of the downstream job corresponding to the currently executing job by the data processing system has timed out.

[0059] The preset threshold value may be a preset difference critical value, which may be adjusted according to actual business needs. Specifically, for the currently executed job, if the corresponding estimated processing time is greater than the preset processing time, and the difference between the two is greater than the preset threshold value, it can be determined that the currently executed job is about to be completed by timeout, and further it can be determined that the execution time of the downstream job corresponding to the currently executed job has timed out, that is, the time for the data processing system to execute the downstream job exceeds the preset time, and the downstream job cannot be completed within the specified time.

[0060] S130: If yes, adjusting the execution parallelism corresponding to the downstream job based on the current throughput of the data processing system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and job dependencies.

[0061] The execution parallelism corresponding to the downstream job can be the number of data processing tasks that the data processing system (such as MapReduce, Spark and other clusters) executes in parallel for the downstream job. The parallelism on a single node and the number of nodes determine the parallelism of the cluster.

[0062] Specifically, if it is determined that the execution time of the downstream job corresponding to the currently executed job has timed out, the execution parallelism corresponding to the downstream job can be adjusted according to the current throughput of the data processing system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and the job dependency. The current throughput includes the computing throughput, input and output throughput, and network throughput of the data processing system.

[0063] In this embodiment, the job dependency may be an execution dependency between jobs. If the execution of one data processing job depends on the completion of another data processing job, then there is a job dependency between the two jobs, for example, job 1 → job 2 → job 4, job 1 → job 3.

[0064] Exemplarily, the job dependency can be determined by the following method, that is, the method also includes: obtaining a directed acyclic graph corresponding to all jobs of the data processing system, wherein the directed acyclic graph includes connection direction information; and determining the job dependency based on the connection direction information.

[0065] The directed acyclic graph may be a directed graph without loops, which includes connection direction information between jobs, and the execution dependency relationship corresponding to each job may be determined according to the connection direction information of the directed acyclic graph. For example, in the directed acyclic graph, jobs with execution dependency relationships may be connected by directional connecting lines, and the pointed job needs to be executed after the job pointing to it is completed.

[0066] Specifically, based on the current throughput of the data processing system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job and the job dependencies, the execution parallelism corresponding to the downstream job is adjusted, which can be: determining the execution delay time corresponding to the downstream job according to the estimated processing time of the currently executed job and the estimated processing time of the downstream job, determining the target parallelism corresponding to the downstream job according to the execution delay time and the current throughput of the data processing system, and adjusting the execution parallelism corresponding to the downstream job based on the target parallelism to increase the execution parallelism corresponding to the downstream job, that is, to achieve capacity expansion of the data processing system.

[0067] The target parallelism corresponding to the downstream job is determined according to the execution delay time and the current throughput of the data processing system. Specifically, the target parallelism corresponding to the downstream job is determined according to the execution delay time, the current throughput and the system available resources. The system available resources can be the number of nodes (servers) available to the system, or the number of pods available to the system. In this way, the idle computing resources of the system can be effectively utilized to compensate for the delay of the downstream job caused by the timeout of the upstream job.

[0068] It is understandable that if the number of available resources of the data processing system is insufficient to adjust the execution parallelism to ensure that downstream jobs are completed on time, multiple blank nodes can be introduced into the data processing system to increase the available resources of the data processing system, thereby ensuring that downstream jobs are completed on time.

[0069] It should be noted that the method provided in this embodiment can be repeatedly executed in the process of the data processing system executing each data processing job. That is, the method also includes: judging whether the currently executed job is the final executed job of the data processing system; if not, after the currently executed job is completed, determining the next executed job corresponding to the currently executed job as the currently executed job, and returning to the operation of performing the judgment to determine the estimated processing time of the currently executed job and the downstream job corresponding to the currently executed job.

[0070] In other words, the method provided in this embodiment can be implemented in the entire process of the data processing system executing data processing jobs. For each data processing job except the final execution job, a judgment is made as to whether the downstream job has timed out. If it times out, the execution parallelism of the downstream job corresponding to the currently executing job is adjusted until all data processing jobs are completed.

[0071] In this way, it can be ensured that all jobs processed by the data processing system can be completed on time, avoiding the timeout of one of the downstream jobs; and the execution parallelism corresponding to the downstream jobs can be readjusted according to the estimated processing time of each currently executed job, thereby realizing dynamic adjustment of the execution parallelism and further ensuring the timely completion of downstream jobs.

[0072] The technical solution of this embodiment determines the estimated processing time of the currently executed job of the data processing system and the estimated processing time of the downstream job corresponding to the currently executed job, and judges whether the system's execution time for the downstream job has timed out based on the estimated processing time. If so, the execution parallelism corresponding to the downstream job is adjusted based on the current throughput of the system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and the job dependencies, so as to achieve dynamic adjustment of the execution parallelism of the downstream job, ensure timely completion of the downstream job through dynamic expansion, and solve the technical problem of downstream job timeout caused by upstream job delays.

[0073] Embodiment 2

[0074] Figure 2A A flowchart of a method for adjusting the degree of parallelism of a data processing task provided in the second embodiment of the present invention. Based on the above embodiments, the present embodiment optionally includes: obtaining historical job statistics of the data processing system; determining the current job of the data processing system and the estimated processing time of the downstream job corresponding to the current job based on the historical job statistics. Figure 2A As shown, the method includes:

[0075] S210: Obtain historical job statistics information of the data processing system, and determine the estimated processing time of a currently executed job of the data processing system and a downstream job corresponding to the currently executed job based on the historical job statistics information.

[0076] The historical job statistics may include the historical completion time, historical computing volume, and system throughput (including computing, I / O, network, etc. throughput) of each historical job of the data processing system. Exemplarily, the historical job statistics of all historical jobs of the data processing system may be extracted offline, and the task-level system throughput may be calculated based on the extracted information, and then the estimated processing time of the currently executed job may be determined based on the calculated system throughput and the amount of data to be processed of the currently executed job.

[0077] In an optional embodiment, determining the estimated processing time of a currently executed job of the data processing system and a downstream job corresponding to the currently executed job based on the historical job statistical information includes: determining the throughput of a historical task corresponding to the historical job based on the historical job statistical information; determining the throughput of the currently executed job and the downstream job corresponding to the currently executed job based on the throughput of the historical task; and determining the estimated processing time of the currently executed job and the downstream job based on the throughput of the currently executed job and the downstream job.

[0078] The throughput of historical tasks includes computing throughput, I / O throughput (input and output throughput), and network throughput. That is, the throughput of the currently executed job can be determined based on the computing throughput, input and output throughput, and network throughput of each historical task included in the historical job, and then the estimated processing time of the currently executed job can be calculated based on the throughput.

[0079] S220: Determine whether the execution time of the data processing system for the downstream job has timed out based on the estimated processing time.

[0080] S230: If yes, adjusting the execution parallelism corresponding to the downstream job based on the current throughput of the data processing system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and job dependencies.

[0081] For example, Figure 2BAs shown, a flow chart of a method for adjusting the parallelism of a data processing task is shown. The method extracts job statistics offline, and calculates task-level computing, I / O, network and other throughput information through the statistics. The task-level computing, I / O and other throughput information calculated at the task level is used to estimate the task-level execution time (i.e., the expected processing time) of the current execution job and the downstream job, and determines whether the expected processing time meets the corresponding time requirements (e.g., whether the task-level execution time of the current execution job does not exceed the preset processing time). If so, the existing parallelism configuration can be maintained, and the execution parallelism of the downstream job will not be adjusted. If not, the historical job execution time ratio can be queried through the calculated task-level computing, I / O and other throughput information, and the execution parallelism of the downstream job can be adjusted according to the directed acyclic graph and the historical job execution time ratio.

[0082] The technical solution of this embodiment obtains historical job statistics information of the data processing system, and determines the estimated processing time of the currently executed job of the data processing system based on the acquired historical job statistics information, thereby achieving accurate estimation of the estimated processing time of the currently executed job and further ensuring the timely completion of downstream jobs.

[0083] Embodiment 3

[0084] Figure 3 A flowchart of a method for adjusting the parallelism of a data processing task provided in the third embodiment of the present invention. Based on the above embodiments, this embodiment optionally further includes: judging whether the execution time of the downstream job by the data processing system is advanced based on the estimated processing time; if so, adjusting the execution parallelism corresponding to the downstream job based on the current throughput, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and the job dependencies. Figure 3 As shown, the method includes:

[0085] S310: Determine a currently executed job of a data processing system and an estimated processing time of a downstream job corresponding to the currently executed job.

[0086] S320. Determine whether the execution time of the downstream job by the data processing system has timed out based on the estimated processing time. If so, adjust the execution parallelism corresponding to the downstream job based on the current throughput of the data processing system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and job dependencies.

[0087] S330. Determine whether the execution time of the downstream job by the data processing system is advanced based on the estimated processing time. If so, adjust the execution parallelism corresponding to the downstream job based on the current throughput, the estimated processing time of the currently executed job, the estimated processing time of the downstream job and the job dependencies.

[0088] Specifically, the estimated processing time of the currently executing job can be compared with the preset processing time of the currently executing job. If the difference between the estimated processing time and the preset processing time is less than the set threshold, and the estimated processing time is less than the preset processing time, it can be determined that the currently executing job can be completed in advance, and then the execution time of the downstream job corresponding to the currently executing job can be advanced.

[0089] Furthermore, after determining that the execution time of the downstream job is advanced, the execution parallelism of the downstream job can be adjusted according to the current throughput of the data processing system, the estimated processing time of the currently executed job, and the job dependencies, so as to reduce the execution parallelism of the downstream job and release the resources of the data processing system, that is, to achieve the reduction of the capacity of the data processing system.

[0090] The technical solution of this embodiment determines whether the execution time of the downstream job of the data processing system is advanced through the estimated processing time. If so, the execution parallelism corresponding to the downstream job is adjusted according to the current throughput, the estimated processing time of the currently executed job, the estimated processing time of the downstream job and the job dependencies, thereby realizing dynamic expansion and contraction of the data processing system. While ensuring that the downstream jobs can be completed on time, the system resources are released so that the released resources can be used by other tasks in the system.

[0091] Embodiment 4

[0092] Figure 4 This is a schematic diagram of the structure of a device for adjusting the degree of parallelism of a data processing task provided by the fourth embodiment of the present invention. Figure 4 As shown, the device includes: a time prediction module 410, a timeout determination module 420 and a downstream adjustment module 430.

[0093] A time prediction module 410, used to determine an estimated processing time of a currently executed job of a data processing system and a downstream job corresponding to the currently executed job;

[0094] A timeout determination module 420, configured to determine whether the execution time of the data processing system for the downstream job has timed out based on the estimated processing time;

[0095] The downstream adjustment module 430 is used to adjust the execution parallelism corresponding to the downstream job based on the current throughput of the data processing system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job and the job dependency.

[0096] In this embodiment, by determining the estimated processing time of the currently executed job of the data processing system and the estimated processing time of the downstream job corresponding to the currently executed job, it is judged whether the system's execution time for the downstream job has timed out based on the estimated processing time. If so, the execution parallelism corresponding to the downstream job is adjusted based on the system's current throughput, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and the job dependencies, so as to achieve dynamic adjustment of the execution parallelism of the downstream job, ensure timely completion of the downstream job through dynamic expansion, and solve the technical problem of downstream job timeout caused by upstream job delays.

[0097] Optionally, the time prediction module 410 includes a historical information extraction unit and a current time estimation unit. The historical information extraction unit is used to obtain historical job statistical information of the data processing system; the current time estimation unit is used to determine the estimated processing time of the currently executed job of the data processing system and the downstream job corresponding to the currently executed job based on the historical job statistical information.

[0098] Optionally, the current time estimation unit is specifically used to:

[0099] Determine the throughput of the historical tasks corresponding to the historical jobs based on the historical job statistics; determine the throughput of the currently executed job and the downstream jobs corresponding to the currently executed job based on the throughput of the historical tasks; determine the estimated processing time of the currently executed job based on the throughput of the currently executed job and the downstream jobs.

[0100] Optionally, the timeout determination module 420 is specifically configured to:

[0101] Obtain the preset processing time of the currently executing job; if the difference between the estimated processing time of the currently executing job and the preset processing time is greater than a preset threshold, and the estimated processing time is greater than the preset processing time, it is determined that the data processing system has timed out the execution time of the downstream job corresponding to the currently executing job.

[0102] Optionally, the device also includes an advance judgment module, which is used to judge whether the execution time of the downstream job by the data processing system is advanced based on the expected processing time; if so, based on the current throughput, the expected processing time of the currently executed job, the expected processing time of the downstream job and the job dependency, adjust the execution parallelism corresponding to the downstream job.

[0103] Optionally, the device also includes a dependency determination module, which is used to obtain a directed acyclic graph corresponding to all jobs of the data processing system, wherein the directed acyclic graph includes connection direction information; and determine the job dependency based on the connection direction information.

[0104] Optionally, the device also includes a return execution module, which is used to determine whether the current execution job is the final execution job of the data processing system; if not, after the current execution job is completed, the next execution job corresponding to the current execution job is determined as the current execution job, and the operation of returning to execute and determining the estimated processing time of the current execution job and the downstream job corresponding to the current execution job is returned.

[0105] The device for adjusting the degree of parallelism of a data processing task provided in an embodiment of the present invention can execute the method for adjusting the degree of parallelism of a data processing task provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0106] Embodiment 5

[0107] Figure 5 This is a schematic diagram of the structure of an electronic device provided in Embodiment 5 of the present invention. Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0108] like Figure 5As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0109] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0110] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 executes the various methods and processes described above, such as a method for adjusting the degree of parallelism of a data processing task.

[0111] In some embodiments, the method for adjusting the degree of parallelism of a data processing task may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the method for adjusting the degree of parallelism of the data processing task described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the method for adjusting the degree of parallelism of a data processing task in any other appropriate manner (e.g., by means of firmware).

[0112] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0113] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0114] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0115] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0116] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0117] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0118] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0119] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for adjusting the parallelism of a data processing task, characterized in that: include: Determine an estimated processing time of a currently executed job of a data processing system and an estimated processing time of a downstream job corresponding to the currently executed job; Determining whether the execution time of the downstream job by the data processing system has timed out based on the estimated processing time of the currently executed job and the estimated processing time of the downstream job; If so, the execution parallelism corresponding to the downstream job is adjusted based on the current throughput of the data processing system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job and the job dependencies; wherein the execution parallelism is the number of data processing tasks of the downstream job that are simultaneously executed in parallel by the data processing system.

2. The method according to claim 1, characterized in that The determining of the estimated processing time of a currently executed job of the data processing system and the estimated processing time of a downstream job corresponding to the currently executed job includes: Get historical job statistics of data processing system; An estimated processing time of a currently executed job of the data processing system and an estimated processing time of a downstream job corresponding to the currently executed job are determined based on the historical job statistical information.

3. The method according to claim 2, characterized in that The determining, based on the historical job statistical information, an estimated processing time of a currently executed job of the data processing system and an estimated processing time of a downstream job corresponding to the currently executed job comprises: Determine the throughput of the historical task corresponding to the historical job based on the historical job statistical information; Based on the throughput of the historical tasks, determining the throughput of the currently executed job and the throughput of the downstream jobs corresponding to the currently executed job; According to the throughput of the currently executed job and the throughput of the downstream job, an estimated processing time of the currently executed job and an estimated processing time of the downstream job are determined.

4. The method according to claim 1, characterized in that The determining, based on the estimated processing time, whether the execution time of the downstream job corresponding to the currently executed job by the data processing system has timed out includes: Obtaining the preset processing time of the currently executed job; If the difference between the estimated processing time of the currently executed job and the preset processing time is greater than a preset threshold, and the estimated processing time is greater than the preset processing time, it is determined that the execution time of the downstream job corresponding to the currently executed job of the data processing system has timed out.

5. The method according to claim 1, characterized in that The method further comprises: Determining whether the execution time of the downstream job by the data processing system is advanced based on the estimated processing time; If so, the execution parallelism corresponding to the downstream job is adjusted based on the current throughput, the estimated processing time of the currently executed job, the estimated processing time of the downstream job, and the job dependency.

6. The method according to claim 1, characterized in that The method further comprises: Obtaining a directed acyclic graph corresponding to all jobs of the data processing system, wherein the directed acyclic graph includes connection direction information; The job dependency is determined based on the connection direction information.

7. The method according to claim 1, characterized in that The method further comprises: Determining whether the currently executed job is the final executed job of the data processing system; If not, after the current execution job is completed, the next execution job corresponding to the current execution job is determined as the current execution job, and the operation of returning to the execution judgment to determine the estimated processing time of the current execution job and the estimated processing time of the downstream job corresponding to the current execution job is performed.

8. A device for adjusting the parallelism of a data processing task, characterized in that: The device comprises: A time prediction module, used to determine an estimated processing time of a currently executed job of the data processing system and an estimated processing time of a downstream job corresponding to the currently executed job; A timeout judgment module, used for judging whether the execution time of the data processing system for the downstream job has timed out based on the estimated processing time of the currently executed job and the estimated processing time of the downstream job; A downstream adjustment module is used to adjust the execution parallelism corresponding to the downstream job based on the current throughput of the data processing system, the estimated processing time of the currently executed job, the estimated processing time of the downstream job and the job dependencies; wherein the execution parallelism is the number of the data processing tasks of the downstream job that are simultaneously executed in parallel by the data processing system.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can perform the method for adjusting the parallelism of the data processing task described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the method for adjusting the parallelism of a data processing task according to any one of claims 1 to 7 when the processor executes the computer instructions.

Images