Extensible distributed task execution system and method, electronic equipment, storage medium and program product

Abstract

The invention provides an extensible distributed task execution system and method, and relates to the field of distributed computing task scheduling, and the system comprises a task scheduling module, a task execution module and a task monitoring module. The task scheduling module receives a current task request and schedules the task to a plurality of task execution nodes after fragmenting the task; the task execution module executes the distributed task fragments on each task execution node, and reports an execution state to the task monitoring module; the task monitoring module judges whether all the tasks are executed or not according to the reported state, and if yes, the next task request is received; otherwise, the task monitoring module monitors the task execution state and records a log, and re-dispatches the failed task fragments. The high-performance extensible distributed task execution system provided by the invention supports multiple task types and scheduling strategies, provides a uniform interface, facilitates system integration and secondary development, and is widely applied to scenes such as timing data processing, system maintenance, distributed computing, batch processing job scheduling and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of distributed computing task scheduling, and particularly relates to an extensible distributed task execution system and method. BACKGROUND

[0002] With the rapid development of cloud computing and big data technology, distributed computing task scheduling systems have emerged as the times require, gradually replacing traditional task scheduling systems and being widely applied in various industries. Traditional task scheduling systems usually adopt centralized architecture, which has problems such as single point of failure and performance bottleneck, and is difficult to meet the needs of large-scale and high-concurrency task execution.

[0003] In order to improve the reliability and scalability of the system, the industry has begun to design task scheduling systems using distributed architecture. Although the existing distributed task scheduling systems have solved the problems of single point of failure and performance bottleneck of traditional task scheduling systems to some extent, there are still many deficiencies. On the one hand, most of the existing distributed task scheduling systems lack comprehensive monitoring and management of task execution, cannot guarantee reliable execution of tasks, and cannot support advanced functions such as failure retry. On the other hand, the existing distributed task scheduling systems have limited scalability and cannot flexibly support multiple task types and scheduling strategies, making it difficult to adapt to different scenarios. In addition, most systems lack unified API interfaces, which brings inconvenience to system integration and secondary development.

[0004] In order to solve the above problems, it is currently necessary to provide a high-performance, scalable distributed task execution system, so as to support the scheduling, execution and monitoring of a large number of tasks, guarantee the reliable execution of tasks, support advanced functions such as failure retry, and at the same time, the required system should have good scalability, so as to support multiple task types and scheduling strategies, and provide unified API interfaces to facilitate system integration and secondary development. SUMMARY

[0005] The present application shows an extensible distributed task execution system and method, in order to solve the problem that the prior art cannot support large-scale, high-performance and scalable distributed task execution requirements.

[0006] In a first aspect, the present application shows an extensible distributed task execution system, which comprises a task scheduling module, a task execution module and a task monitoring module. The task scheduling module receives a current task request and schedules the task shards to multiple task execution nodes after the task is divided; The task execution module executes the assigned task shards on each task execution node and reports the execution status to the task monitoring module; The task monitoring module determines whether all tasks are executed according to the reported state, if yes, the next task request is received, if no, the task monitoring module monitors the task execution state and records the log, and retries the scheduling of the failed task fragments.

[0007] Optionally, the system further comprises an interface module and a horizontal expansion module. The interface module has a RESTful style API interface for external system access and integration. The horizontal expansion module dynamically increases or decreases the task execution node according to the task load, and realizes the horizontal expansion of the distributed task execution system.

[0008] Optionally, the task scheduling module is specifically configured to: According to the task type and the preset scheduling strategy, the current task is split into multiple task fragments; Based on the load condition of the task execution node, the task fragments are scheduled to the corresponding task execution node; The task fragments are distributed to the corresponding task execution node through the message queue or the server.

[0009] Optionally, the task execution module is specifically configured to: Receive the assigned task fragments, and construct the corresponding task script according to the task type; Execute the task script, and report the execution state to the task monitoring module in real time, and perform local retry execution on the failed task fragments.

[0010] Optionally, the task monitoring module is specifically configured to: Collect the task execution state reported by each task execution node in real time; Analyze and record the task execution state to generate a task execution log; Retry scheduling of the failed task fragments, and schedule the failed tasks to other task execution nodes except the current task execution node for execution.

[0011] In the second aspect, an extensible distributed task execution method is shown, applied to the distributed task execution system in any of the above aspects, and the method comprises: The task scheduling module receives the current task request, and schedules the task fragments to multiple task execution nodes; The task execution module executes the assigned task fragments on each task execution node, and reports the execution state to the task monitoring module; The task monitoring module determines whether all tasks are executed according to the reported state, if yes, the next task request is received, if no, the task monitoring module monitors the task execution state and records the log, and retries the scheduling of the failed task fragments.

[0012] Optionally, the task scheduling module receives the current task request, and the step of scheduling the task fragments to the plurality of task execution nodes after fragmenting the task, comprises: The task scheduling module splits the current task into a plurality of task fragments according to the task type and the preset scheduling strategy; The task fragments are scheduled to the corresponding task execution nodes based on the load condition of the task execution nodes; The task fragments are distributed to the corresponding task execution nodes through the message queue or the server.

[0013] Optionally, the task execution module executes the assigned task fragments on each task execution node, and reports the execution state to the task monitoring module, and the step comprises: The task monitoring module receives the assigned task fragments, and constructs the corresponding task script according to the task type; The task script is executed, and the execution state is reported to the task monitoring module in real time, and the failed task fragments are executed locally.

[0014] Optionally, the step of monitoring the task execution state and recording the log, and retrying the scheduling of the failed task fragments by the task monitoring module, comprises: The task monitoring module collects the task execution state reported by each task execution node in real time; The task execution state is analyzed and recorded to generate the task execution log; The failed task fragments are scheduled to other task execution nodes except the current task execution node for execution.

[0015] In a third aspect, the present application shows an electronic device, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the method of any of the above aspects.

[0016] In a fourth aspect, the present application shows a non-transitory computer-readable storage medium, when the instructions in the storage medium are executed by the processor of an electronic device, the electronic device can execute the method of any of the above aspects.

[0017] In a fifth aspect, the present application shows a computer program product, when instructions in the computer program product are executed by a processor of an electronic device, the electronic device can execute the method according to any one of the above aspects.

[0018] The technical scheme provided by the present application can include the following beneficial effects: (1) retry function, solves the problem that the existing system lacks comprehensive monitoring and management of task execution and cannot guarantee reliable task execution, improves the reliability and stability of task execution; (2) by supporting horizontal expansion and multiple task types and scheduling strategies, improve the scalability of the system, can flexibly adapt to the needs of different scenarios, meet the needs of large-scale, high-concurrency task execution, solve the problem of limited scalability of the existing system; (3) by providing RESTful style API interface, it is convenient for system integration and secondary development, solves the problem that the existing system lacks unified API interface; (4) by designing distributed architecture, improve the performance and concurrent processing capacity of the system, solve the single point failure and performance bottleneck problem existing in the existing centralized architecture; (5) by reasonable resource utilization and scheduling strategy, improve the resource utilization rate, solve the problem of unreasonable resource utilization and allocation of the existing system. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 A structural diagram of the scalable distributed task execution system provided by the present application is shown in the figure; Figure 2 Another structural diagram of the scalable distributed task execution system provided by the present application is shown in the figure; Figure 3 A flowchart of the scalable distributed task execution method provided by the present application is shown in the figure; Figure 4 Another flowchart of the scalable distributed task execution method provided by the present application is shown in the figure Figure 5 A block diagram of an electronic device provided by the present application is shown in the figure; Figure 6 Another block diagram of an electronic device provided by the present application is shown in the figure. DETAILED DESCRIPTION

[0020] The technical scheme in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0021] To solve the above problems, the application provides an extensible distributed task execution system and method to solve the problem that the prior art cannot support large-scale, high-performance and extensible distributed task execution requirements.

[0022] Example One Please refer to Figure 1 and Figure 2 , wherein, Figure 1 is a structural diagram of an extensible distributed task execution system provided by the application, which comprises a task scheduling module 110, a task execution module 120 and a task monitoring module 130.

[0023] The task scheduling module 110 receives a current task request and schedules the task shards to multiple task execution nodes after the task is fragmented; The task execution module 120 executes the assigned task shards on each task execution node and reports the execution status to the task monitoring module; The task monitoring module 130 determines whether the task is completely executed according to the reported status, if yes, it receives the next task request; if no, the task monitoring module monitors the task execution status and records the log, and retries the scheduling of the failed task shards.

[0024] Further, the system further comprises an interface module and a horizontal expansion module; the interface module has a RESTful style API interface for external system access and integration, for example, it can provide task submission, task query, task control and other API interfaces, which facilitates the integration of other systems with the system; the horizontal expansion module dynamically increases or decreases the task execution nodes according to the task load, to realize the horizontal expansion of the distributed task execution system. For example, based on the CPU, memory and other resource utilization of the task execution nodes, the number of task execution nodes can be dynamically increased or decreased to meet the requirements of different task loads.

[0025] In one case, the task scheduling module 110 is specifically configured to split the current task into multiple task shards according to the task type and the preset scheduling strategy; based on the load condition of the task execution node, the task shards are scheduled to the corresponding task execution node; the task shards are distributed to the corresponding task execution node through a message queue or a server.

[0026] Specifically, for batch data processing tasks, the tasks can be split into multiple shards according to the data volume, and each shard processes a part of the data; for timing tasks, the tasks can be split into multiple shards according to the time window, and each shard executes the task of a time period. In one case, the task shards can be dispatched to the nodes with lower load for execution by using a round-robin scheduling, a minimum load priority scheduling or the like. Specifically, a high-performance message queue such as Kafka can be used to send the task shard information to the corresponding queue, and the task execution nodes acquire and execute the task shard information from the queue.

[0027] In one case, the task execution module 120 is specifically configured to receive the assigned task shards and construct a corresponding task script according to the task type; execute the task script and report the execution status to the task monitoring module in real time, and perform local retry execution on the failed task shards.

[0028] Specifically, for SQL tasks, a corresponding execution script can be generated according to the SQL statement; for Shell tasks, a corresponding execution script can be generated according to the Shell command. The task execution status can be reported to the monitoring module through an HTTP interface or a message queue, including the task start time, end time, execution result and the like. Preferably, the maximum number of retries can be set to 3 times, and if the task still fails after 3 retries, the failure information is reported to the monitoring module. It should be noted that the application does not limit the specific value of the maximum number of retries.

[0029] In one case, the task monitoring module 130 is specifically configured to collect the task execution status reported by each task execution node in real time; analyze and record the task execution status to generate a task execution log; and perform retry scheduling on the failed task shards to dispatch the failed tasks to other task execution nodes except the current task execution node for execution.

[0030] In one case, the task execution status can be stored in a database, and a Web interface is provided to view the task execution status. A failover strategy can be used to dispatch the failed task shards to other available nodes for execution.

[0031] In one specific example, as shown in FIG. 1, Figure 2 The task scheduling module 110 includes two sub-modules, i.e., a task management module and a distributed scheduler. Specifically, the task management module receives a current task request and dispatches the task shards to various task execution nodes such as an extensible executor (i.e., the task execution module 120) through the distributed scheduler; and then analyzes and records the task execution status through the monitoring and log module (i.e., the task monitoring module 130) to generate a task execution log and store the task execution log in a persistent storage layer.

[0032] Compared with the prior art, the distributed task execution system provided by the application has the following beneficial effects: (1) By supporting the scheduling, execution and monitoring of a large number of tasks, and guaranteeing the reliable execution and failure retry function of the tasks, the problem that the existing system lacks comprehensive monitoring and management of task execution and cannot guarantee reliable task execution is solved, and the reliability and stability of task execution are improved; (2) By supporting horizontal expansion and multiple task types and scheduling strategies, the scalability of the system is improved, which can flexibly adapt to the needs of different scenarios, meet the needs of large-scale and high-concurrency task execution, and solve the problem of limited scalability of the existing system; (3) By providing a RESTful style API interface, system integration and secondary development are facilitated, and the problem of lack of unified API interface in the existing system is solved; (4) By designing a distributed architecture, the performance and concurrent processing capacity of the system are improved, and the single point of failure and performance bottleneck problems existing in the existing centralized architecture are solved; (5) By reasonable resource utilization and scheduling strategy, the resource utilization rate is improved, and the problem of unreasonable resource utilization and allocation in the existing system is solved.

[0033] Example Two See Figure 3 and Figure 4 , wherein, Figure 3 is a flowchart of an extensible distributed task execution method provided by the application, applied to the distributed task execution system described in any one of the above, the method comprising: Step S201: The task scheduling module receives a current task request and schedules the task shards to multiple task execution nodes after the task is divided.

[0034] In one case, the task scheduling module splits the current task into multiple task shards according to the task type and the preset scheduling strategy; based on the load condition of the task execution node, the task shards are scheduled to the corresponding task execution node; the task shards are distributed to the corresponding task execution node through a message queue or a server.

[0035] Step S202: The task execution module executes the allocated task shards on each task execution node and reports the execution status to the task monitoring module.

[0036] In one case, the task monitoring module receives the allocated task shards and constructs the corresponding task script according to the task type; executes the task script and reports the execution status to the task monitoring module in real time, and performs local retry execution on the failed task shards.

[0037] Step S203: The task monitoring module determines whether all tasks are completed according to the reported state; if yes, step S204 is executed; if no, step S205 is executed. Step S204: A next task request is received. Step S205: The task monitoring module monitors the task execution state and records a log, and retries scheduling of a failed task fragment.

[0038] In one case, the task monitoring module collects the task execution state reported by each task execution node in real time; analyzes and records the task execution state to generate a task execution log; and retries scheduling of a failed task fragment, and schedules the failed task to other task execution nodes except the current task execution node for execution.

[0039] It should be noted that, for the method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the action sequence described, because according to the present application, certain steps can be performed in other sequences or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification are all optional embodiments, and the actions involved are not necessarily required by the present application.

[0040] Example Three Optionally, the present application also provides an electronic device, which comprises a processor, a memory, a computer program stored in the memory and executable on the processor, the computer program is executed by the processor to realize each process of the above method embodiments, and the same technical effects can be achieved, to avoid repetition, here is no longer repeated.

[0041] The present application also provides a computer readable storage medium, the computer readable storage medium stores a computer program, the computer program is executed by the processor to realize each process of the above method embodiments, and the same technical effects can be achieved, to avoid repetition, here is no longer repeated. The computer readable storage medium, such as read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disk, etc.

[0042] Figure 5 A block diagram of an electronic device 800 is provided for the present application. For example, the electronic device 800 can be a mobile phone, a computer, a digital broadcast terminal, a message transmission device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0043] Reference Figure 5The electronic device 800 can include one or more of the following components: a processing component 802, a memory 804, a power supply component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0044] The processing component 802 usually controls overall operations of the electronic device 800, such as operations associated with displaying, making phone calls, data communications, camera operations and recording operations. The processing component 802 can include one or more processors 820 to execute instructions to complete all or part of steps of the above methods. In addition, the processing component 802 can include one or more modules to facilitate

[0045] The memory 804 is configured to store various types of data to support operations of the electronic device 800. Examples of these data include instructions for any application or method operating on the electronic device 800, contact data, phonebook data, messages, images, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic or optical disk.

[0046] The power supply component 806 provides power for the various components of the electronic device 800. The power supply component 806 can include a power supply management system, one or more power supplies, and other components associated with generating, managing and distributing power for the electronic device 800.

[0047] The multimedia component 808 includes a screen to provide an output interface between the electronic device 800 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. The front camera and / or the rear camera can receive external multimedia data when the electronic device 800 is in an operating mode, such as a shooting mode or a video mode. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zooming capability.

[0048] The audio component 810 is configured to output and / or input an audio signal. For example, the audio component 810 includes a microphone (MIC) to receive an external audio signal when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 further includes a speaker to output an audio signal.

[0049] The I / O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0050] The sensor component 814 includes one or more sensors to provide various state assessments for the electronic device 800. For example, the sensor component 814 can detect an open / closed state of the device 800, relative positioning of components, such as a display and a keypad of the electronic device 800, a change in position of the electronic device 800 or a component of the electronic device 800, presence or absence of user contact with the electronic device 800, a direction or acceleration / deceleration of the electronic device 800, and a change in temperature of the electronic device 800. The sensor component 814 can include a proximity sensor to detect presence of an object in proximity to the electronic device 800 without any physical contact. The sensor component 814 can also include a light sensor, such as a CMOS or CCD image sensor, to use in an imaging application. In some embodiments, the sensor component 814 can further include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0051] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as WiFi, a cellular network (e.g., 2G, 3G, 4G or 5G), or a combination thereof. In an example embodiment, the communication component 816 receives broadcast signals or broadcast operation information from an external broadcast management system via a broadcast channel. In an example embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques and other techniques.

[0052] In an example embodiment, the electronic device 800 can be implemented with one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic elements, for performing the above-described methods.

[0053] In an example embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, is also provided, which can be executed by the processor 820 of the electronic device 800 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0054] Example Four Figure 6 Another block diagram of an electronic device 1900 is provided for the present application. For example, the electronic device 1900 can be provided as a server.

[0055] Referring to Figure 6 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by a memory 1932, for storing instructions, such as application programs, executable by the processing component 1922. The application programs stored in the memory 1932 can include one or more modules each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute the instructions to perform the above-described methods.

[0056] The electronic device 1900 can further include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 can operate based on an operating system stored in the memory 1932, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0057] Example Five In a fifth aspect, the present application provides a computer program product, which, when instructions in the computer program product are executed by a processor of an electronic device, enables the electronic device to perform the method according to any one of the preceding aspects.

[0058] It should be noted that, in this document, the terms "comprise", "comprising", or any other variant thereof are intended to cover non-exclusive inclusions, such that processes, methods, articles, or apparatuses that comprise a list of elements are not limited to those elements, but can also include other elements not expressly listed, or inherent to such processes, methods, articles, or apparatuses. Without further limitation, an element defined by the phrase "comprising a" does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0059] From the above description of the embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be realized by means of software and necessary general hardware platforms, of course, they can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk) and includes a number of instructions for causing a terminal (which can be a mobile phone, a computer, a server, an air conditioner, or a network device, etc.) to execute the methods described in the various embodiments of the present application.

[0060] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the specific embodiments described above, which are merely illustrative rather than restrictive, and those of ordinary skill in the art can make many forms under the inspiration of the present application without departing from the scope of the present application and the scope of protection of the claims.

[0061] Those skilled in the art can clearly understand the unit and algorithm steps of each example described in combination with the embodiments disclosed in the present application can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software manner depends on the specific application and design constraints of the technical solution. The skilled person can use different methods to realize the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0062] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0063] In the embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented in other ways. For example, the above-described device embodiments are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0064] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0065] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0066] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a U disk, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc.

[0067] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A scalable distributed task execution system, characterized in that, The system comprises a task scheduling module, a task execution module and a task monitoring module; The task scheduling module receives a current task request and schedules task shards to multiple task execution nodes after the task is fragmented; The task execution module executes the assigned task shards on each task execution node and reports the execution status to the task monitoring module; The task monitoring module determines whether the task is completely executed according to the reported status, if yes, receives the next task request, if no, the task monitoring module monitors the task execution status and records the log, and retries the scheduling of the failed task shards.

2. The extensible distributed task execution system of claim 1, wherein, The system further comprises an interface module and a horizontal expansion module; The interface module has a RESTful style API interface for external system access and integration; The horizontal expansion module dynamically increases or decreases the task execution nodes according to the task load, so as to realize the horizontal expansion of the distributed task execution system.

3. The extensible distributed task execution system of claim 1, wherein, The task scheduling module is specifically configured to: According to the task type and the preset scheduling strategy, the current task is split into multiple task shards; Based on the load condition of the task execution node, the task shards are scheduled to the corresponding task execution node; The task shards are distributed to the corresponding task execution node through a message queue or a server.

4. The extensible distributed task execution system of claim 1, wherein, The task execution module is specifically configured to: Receive the assigned task shards and construct the corresponding task script according to the task type; Execute the task script and report the execution status to the task monitoring module in real time, and perform local retry execution on the failed task shards.

5. The extensible distributed task execution system of claim 1, wherein, The task monitoring module is specifically configured to: Collect the task execution status reported by each task execution node in real time; Analyze and record the task execution status to generate a task execution log; Retry scheduling the failed task shards to other task execution nodes except the current task execution node for execution.

6. A scalable distributed task execution method, characterized by, The method applied to the distributed task execution system of any one of claims 1 to 5, the method comprises: The task scheduling module receives a current task request and schedules task shards to multiple task execution nodes after the task is fragmented; The task execution module executes the assigned task shards on each task execution node and reports the execution status to the task monitoring module; The task monitoring module determines whether the task is completely executed according to the reported status, if yes, receives the next task request, if no, the task monitoring module monitors the task execution status and records the log, and retries the scheduling of the failed task shards.

7. The scalable distributed task execution method of claim 6, wherein, The step that the task scheduling module receives a current task request and schedules task shards to multiple task execution nodes after the task is fragmented, comprises: The task scheduling module splits the current task into multiple task shards according to the task type and the preset scheduling strategy; Based on the load condition of the task execution node, the task shards are scheduled to the corresponding task execution node; The task shards are distributed to the corresponding task execution node through a message queue or a server.

8. The scalable distributed task execution method of claim 6, wherein, The step that the task execution module executes the assigned task shards on each task execution node and reports the execution status to the task monitoring module, comprises: The task monitoring module receives the assigned task fragments and constructs a corresponding task script according to the task type; The task script is executed, and the execution state is reported to the task monitoring module in real time, and the failed task fragments are locally retried and executed.

9. The scalable distributed task execution method of claim 6, wherein, The task monitoring module monitors the task execution state and records logs, and the step of retrying and scheduling the failed task fragments comprises: The task monitoring module collects the task execution state reported by each task execution node in real time; The task execution state is analyzed and recorded to generate a task execution log; The failed task fragments are retried and scheduled to be executed on other task execution nodes except the current task execution node.

10. An electronic device, comprising: Comprise: A processor, a memory, and a computer program stored on the memory and executable on the processor, the computer program being executed by the processor to implement the method of any one of claims 6 to 9.

11. A computer readable storage medium, characterized in that, The computer program is stored on the computer readable storage medium, and the computer program is executed by the processor to implement the method of any one of claims 6 to 9.

12. A computer program product, characterised in that, When the instructions in the computer program product are executed by the processor of the electronic device, the electronic device implements the method of any one of claims 6 to 9.

Images