Spark kernel-based task execution method, system and computer device

By using a task execution method based on the Spark kernel, combined with Akka and Spark, and generating parallel task streams using configuration files, the problems of low data synchronization and development efficiency in existing technologies are solved, achieving efficient ETL task execution and cluster deployment.

CN112486468BActive Publication Date: 2024-11-26ENYIKE (BEIJING) DATA TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011472995.3
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-15
Publication Date
2024-11-26
Estimated Expiration
2040-12-15

AI Technical Summary

Technical Problem

Existing technologies have problems in data applications, such as lack of support for real-time data synchronization, lack of support for cluster deployment, low development efficiency, lack of notification mechanism for task failure, and slow ETL computing speed.

Method used

It adopts a task execution method based on the Spark kernel, combining Akka and Spark, using configuration files to generate task streams for parallel execution, using an actor model to allocate and execute tasks, and defining data sources and synchronization data formats through configuration files, supporting multi-task parallel processing.

Benefits of technology

It improves the execution efficiency of ETL tasks, lowers the development threshold and maintenance costs, enables real-time data synchronization and cluster deployment, supports multi-task parallel processing, and reduces the risk of data loss.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112486468B_ABST
    Figure CN112486468B_ABST
Patent Text Reader

Abstract

The present application relates to a task execution method, system, and computer device based on the spark kernel, wherein the task execution method includes: a task flow generation step, in which the spark driver node reads and generates a task flow according to a configuration file; a task allocation step, in which akka allocates tasks according to the task flow; a task execution step, in which a scheduler distributes tasks to corresponding execution nodes according to the information of the allocated tasks, and sends the execution results to the spark driver node. By combining akka and spark, the task flow of etl and each task in the task flow can be executed in parallel, thereby improving the execution efficiency; and the cost of writing code can be saved by configuring the configuration file, thereby improving the development efficiency.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of big data technology, and in particular to a task execution method, system, and computer device based on the spark kernel. Background Art

[0002] With the advent of the big data cloud era, all walks of life have an increasing demand for data applications. In data applications, the synchronization and calculation of multi-heterogeneous data are the basis of data applications. How to efficiently synchronize and quickly develop ETL between different data sources has become a major problem facing the industry.

[0003] In the prior art, ETL development is generally performed by writing MapReduce code or Spark code. However, there are the following disadvantages:

[0004] 1. Does not support real-time data synchronization;

[0005] 2. Cluster deployment is not supported. It is limited by the performance of a single server, has a slow speed, and is prone to memory overflow.

[0006] 3. Does not support multi-tasking configuration;

[0007] 4. There is no notification mechanism for task failure;

[0008] 5. The speed of performing ETL calculation is slow;

[0009] 6. It is necessary to write code, package and deploy the project, which has low development efficiency and high difficulty.

[0010] Currently, no effective solution has been proposed to address the problem of low development efficiency in related technologies. Summary of the invention

[0011] The embodiments of the present application provide a task execution method, system, and computer device based on the Spark kernel, which lower the development threshold, improve development efficiency, reduce maintenance costs, increase ETL speed, and improve the robustness of the system by setting configuration files.

[0012] In a first aspect, an embodiment of the present application provides a task execution method based on the spark kernel, comprising the following steps:

[0013] In the task flow generation step, the spark driver node reads and generates a task flow according to a configuration file;

[0014] Task allocation step: Akka allocates tasks according to the task flow;

[0015] In the task execution step, a scheduler distributes the task to the corresponding execution node according to the information of the assigned task, and sends the execution result to the spark driver node.

[0016] In some embodiments, the task flow generation step specifically includes:

[0017] By writing spark and akka codes, the configuration file is read and parsed to generate a dag graph.

[0018] In some embodiments, the task allocation step specifically includes:

[0019] Traverse each task of the dag graph and determine its status. If it is not executed, determine whether there is a parent node task. If there is a parent node task, wait for the parent node task to be successfully executed. If there is no parent node task or all parent node tasks have been successfully executed, distribute the task to the executor for execution through the actor model.

[0020] In some embodiments, the task execution steps specifically include:

[0021] Sending the execution result to the spark driver node through the actor model, and the spark driver node receives and records its status;

[0022] When the task fails to execute, a retry is performed. When the number of retries exceeds a preset number, the task is determined to have failed and the program is exited, and an alarm message is output;

[0023] When the task is successfully executed, the sub-node tasks continue to be executed until all the tasks are completed.

[0024] In some of the embodiments, the configuration file uses the JSON format to define the data source, synchronization data format, and specify data conversion logic.

[0025] In a second aspect, an embodiment of the present application provides a task execution system based on the spark kernel, applying the task execution method based on the spark kernel described in the first aspect above, including:

[0026] The task flow generation module, the spark driver node reads and generates a task flow according to a configuration file;

[0027] Task allocation module, Akka allocates tasks according to the task flow;

[0028] The task execution module is a scheduler that distributes the task to the corresponding execution node according to the information of the assigned task, and sends the execution result to the spark driver node.

[0029] In some of the embodiments, the task flow generation module generates a DAG graph by writing Spark and Akka codes and reading and parsing the configuration file.

[0030] In some of the embodiments, the task allocation module traverses each task of the dag graph and determines its status. If it is not executed, it determines whether there is a parent node task. If there is a parent node task, it waits for the parent node task to be executed successfully. If there is no parent node task or all parent node tasks have been executed successfully, the task is distributed to the executor for execution through the actor model.

[0031] In some embodiments, the task execution module includes:

[0032] A state recording unit, which sends the execution result to the spark driver node through the actor model, and the spark driver node receives and records its state;

[0033] The status judgment unit performs a retry when the task fails to execute. When the number of retries exceeds a preset number, the task is judged to have failed and the program is exited, and an alarm message is output; when the task is successfully executed, the sub-node tasks are continued to be executed until the task is fully completed.

[0034] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the task execution method based on the spark kernel as described in the first aspect above is implemented.

[0035] Compared with the related art, the task execution method based on the spark kernel provided in the embodiment of the present application improves the execution efficiency by combining akka and spark to achieve parallel execution of the task flow of etl and parallel execution of each task in the task flow; and saves the cost of writing code through configuration files, thereby improving development efficiency.

[0036] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0038] Figure 1is a flowchart of a task execution method based on the spark kernel according to an embodiment of the present application;

[0039] Figure 2 is a preferred flow chart of a task execution method according to an embodiment of the present application;

[0040] Figure 3 is a flowchart of the task execution steps according to the preferred embodiment of the present application;

[0041] Figure 4 It is a structural block diagram of a task execution system based on the spark kernel according to an embodiment of the present application;

[0042] Figure 5 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application. Description of the drawings:

[0044] 1. Task flow generation module; 2. Task allocation module; 3. Task execution module;

[0045] 31. Status recording unit; 32. Status judging unit; 81. Processor;

[0046] 82. Memory; 83. Communication interface; 80. Bus. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not intended to limit the present application. Based on the embodiments provided in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0048] Obviously, the drawings described below are only some examples or embodiments of the present application. For ordinary technicians in this field, the present application can also be applied to other similar scenarios based on these drawings without creative work. In addition, it can also be understood that although the efforts made in this development process may be complicated and lengthy, for ordinary technicians in this field related to the content disclosed in this application, some changes in design, manufacturing or production based on the technical content disclosed in this application are just conventional technical means, and should not be understood as insufficient content disclosed in this application.

[0049] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those of ordinary skill in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0050] Unless otherwise defined, the technical terms or scientific terms involved in this application should be understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the" and the like involved in this application do not indicate a quantitative limitation, and may represent the singular or plural. The terms "include", "comprise", "have" and any of their variations involved in this application are intended to cover non-exclusive inclusions; for example, a process, method, system, product or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units that are not listed, or may also include other steps or units inherent to these processes, methods, products or devices. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there may be three relationships, for example, "A and / or B" can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the objects before and after are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0051] Apache Spark is a fast, general-purpose distributed query and processing engine designed for large-scale data processing. It provides the flexibility and scalability of MapReduce, but at a significantly higher speed: it is 100 times faster than Apache Hadoop when data is stored in memory and up to 10 times faster when accessing disk. Apache Spark allows users to read, transform, aggregate data, and easily train and deploy complex statistical models. The Spark API is accessible in Java, Scala, Python, R, and SQL.

[0052] Akka is a toolkit and runtime for building highly concurrent, distributed, and fault-tolerant applications on the JAVA virtual machine platform. Akka is written in Scala and provides development interfaces for both Scala and Java. Akka's approach to concurrency is based on the Actor model, and the only mechanism for communication between Actors is message passing.

[0053] Akka's actor model is a lightweight message communication model in Akka.

[0054] The DAG data structure tracks the computation and assignment of values ​​and variables within a basic block; values ​​used from elsewhere in the block are represented as leaf nodes; operations on values ​​are represented as internal nodes; and assignment of new values ​​is represented by appending the name of the target variable or temporary variable to the node representing the assignment.

[0055] This embodiment provides a task execution method based on the spark kernel. Figure 1 is a flowchart of a task execution method based on the spark kernel according to an embodiment of the present application, such as Figure 1 As shown, the process includes the following steps:

[0056] In the task flow generation step S1, the spark driver node reads and generates a task flow according to a configuration file;

[0057] Task allocation step S2, Akka allocates tasks according to the task flow;

[0058] In the task execution step S3, a scheduler distributes the task to the corresponding execution node according to the information of the assigned task, and sends the execution result to the spark driver node.

[0059] In actual applications, based on the spark kernel, the actor model of Akka is used as a distributed communication method. Both spark and Akka have a distributed architecture. By writing spark and Akka codes, functions such as reading configuration files, parsing and generating dag graphs, distributing tasks, and recording and judging task status are completed. At the same time, input and output modules are also set up. The above input and output modules must support the reading and writing functions of multiple data sources such as mysql, hdfs, clickhouse, hbase, hive, postgresql, etc.

[0060] Through the above steps, the method is completely developed based on the spark kernel. Spark is based on memory computing. The combination of Akka and Spark can achieve parallel execution of the task flow of ETL and parallel execution of each task in the task flow, which can greatly improve the computing efficiency and execution efficiency. At the same time, the configuration file is used to synchronize the data source configuration and manage task dependencies, and the calculation is performed by writing SQL, which reduces the development threshold and difficulty. Different configuration files can be developed according to different needs, without the need to modify and package the code, which is simple to deploy and easy to maintain. There is no data transmission between components, which reduces the possibility of data loss.

[0061] In some embodiments, the task flow generation step specifically includes:

[0062] By writing spark and akka code, read and parse the configuration file to generate the DAG graph.

[0063] In this embodiment, each leaf node in the DAG graph corresponding to the task can be generated according to the configuration file, so as to split the task corresponding to the configuration file into several simple subtasks through the leaf nodes of the DAG graph to execute the task corresponding to the task request. In addition, since the leaf nodes of the DAG graph have been determined, the total number of nodes of the DAG graph can be further determined according to the configuration file, so that when the downstream nodes are generated by executing the task later, it can be determined when the task is completed (for example, when all the nodes of the DAG graph are generated).

[0064] Specifically, the data required to be input for the corresponding task can be determined according to the configuration file, and then the leaf nodes in the DAG graph can be generated according to the determined data. Alternatively, when the configuration file already carries the data required to execute the task, the server can also directly generate the leaf nodes in the DAG graph according to the carried data. Of course, the above two methods of generating leaf nodes can be used alone or in combination, and this specification does not limit them.

[0065] In some embodiments, the task allocation step specifically includes:

[0066] Traverse each task of the dag graph and determine its status. If it is not executed, determine whether there is a parent node task. If there is a parent node task, wait for the parent node task to be executed successfully. If there is no parent node task or all parent node tasks have been executed successfully, distribute the task to the executor for execution through the actor model.

[0067] The specific tasks to be executed are sent in batches by the above mechanism, and the execution of each task is executed in parallel through the spark core engine. According to this solution, each task is executed in parallel, and tasks can also be executed in parallel, which greatly improves the execution efficiency.

[0068] In some embodiments, the task execution steps specifically include:

[0069] The execution results are sent to the spark driver node through the actor model, and the spark driver node receives and records its status;

[0070] When the task fails to execute, a retry is performed. When the number of retries exceeds a preset number, the task is determined to have failed and the program is exited, and an alarm message is output;

[0071] When the task is successfully executed, continue to execute the sub-node tasks until all tasks are completed.

[0072] In this embodiment, the Spark task can dynamically allocate resources based on the YARN cluster, which is easy to expand and has a fast synchronization speed. When a task fails, enterprise WeChat, DingTalk, and email alarms can be performed according to the configuration file for the detection personnel to make adjustments.

[0073] In some of the embodiments, the configuration file uses the JSON format to define the data source, synchronization data format, and specify data conversion logic.

[0074] The embodiments of the present application are described and illustrated below through preferred embodiments.

[0075] Figure 2 is a preferred flow chart of the task execution method according to an embodiment of the present application, such as Figure 2 As shown, the task execution method includes the following steps:

[0076] S201, Configuration File

[0077] The file script configuration format uses the JSON format. Users can define the ETL process by writing a file script in the JSON format, including defining the data source, synchronizing the data format, and specifying the data conversion logic.

[0078] S202, read the configuration file and generate a dag graph;

[0079] The task is started by writing spark and akka codes. The driver in the service reads and parses the configuration file content, and generates the corresponding dag graph (directed acyclic graph) based on the parsed configuration file content.

[0080] S203, Distribute Tasks

[0081] Traverse each task in the DAG graph and determine its status. If it is not executed, determine whether there is a parent node task. If there is a parent node task, wait for the parent node task to be successfully executed. If there is no parent node task or all parent node tasks have been successfully executed, the task to be executed is assigned through the Akka actor model mechanism. After the Spark scheduler receives the message from the actor model, it distributes the task to the corresponding executor for execution.

[0082] S204, record and judge task status

[0083] Executor executes the task and sends the task status to spark driver through akka. After receiving it, spark driver records its status. If it fails, it will retry. If the number of retries exceeds, the task is considered to have failed and the program will exit. An alarm email will be sent, etc. If it succeeds, it will continue to judge and execute the sub-node tasks until all tasks are completed.

[0084] Through the above steps, through script configuration (no need to write code), the Spark core engine is used for efficient ETL. The existing MapReduce development or Spark development can also implement the ETL process, but it is not as convenient and efficient as this method.

[0085] Figure 3 is a flowchart of the task execution steps according to the preferred embodiment of the present application, such as Figure 3 As shown, tasks 1, 2, and 3 are tasks executed in parallel, task 4 is a subtask of tasks 1 and 2, task 5 is a subtask of task 3, and task 6 is a subtask of task 4. The task execution steps include the following steps:

[0086] The driver executes tasks 1, 2, and 3 in parallel. After tasks 1 and 2 are completed, it executes task 4. After task 4 is completed, it executes task 6. After task 3 is completed, it executes task 5. After tasks 5 and 6 are completed, the driver task is completed.

[0087] It should be noted that the steps shown in the above process or the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0088] This embodiment also provides a task execution system based on the spark kernel, which is used to implement the above embodiments and preferred implementation modes, and will not be repeated here. As used below, the terms "module", "unit", "subunit", etc. can be a combination of software and / or hardware that implements a predetermined function. Although the system described in the following embodiments is preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0089] Figure 4 is a structural block diagram of a task execution system based on the spark kernel according to an embodiment of the present application, such as Figure 4 As shown, the system includes:

[0090] Task flow generation module 1, the spark driver node reads and generates a task flow according to a configuration file;

[0091] Task allocation module 2, Akka allocates tasks according to task flows;

[0092] Task execution module 3, a scheduler distributes tasks to corresponding execution nodes according to the information of task allocation, and sends the execution results to the spark driver node.

[0093] In actual applications, the configuration file uses the JSON format to define the data source, synchronize the data format, and specify the data conversion logic.

[0094] In some of the embodiments, the task flow generation module 1 generates a DAG graph by writing Spark and Akka codes and reading and parsing the configuration file.

[0095] In some of the embodiments, the task allocation module 2 traverses each task of the DAG graph and determines its status. If it is not executed, it determines whether there is a parent node task. If there is a parent node task, it waits for the parent node task to be executed successfully. If there is no parent node task or all parent node tasks have been executed successfully, the task is distributed to the executor for execution through the Akka actor model.

[0096] In some embodiments, the task execution module 3 includes:

[0097] The state recording unit 31 sends the execution result to the driver node through the actor model, and the driver node receives and records its state;

[0098] The status judgment unit 32 performs a retry when the task fails to execute. When the number of retries exceeds a preset number, the task is considered to have failed and the program is exited, and an alarm message is output; when the task is successfully executed, the sub-node tasks continue to be executed until the tasks are all completed.

[0099] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0100] In addition, combined Figure 1 The task execution method based on the spark kernel in the embodiment of the present application described can be implemented by a computer device. Figure 5 Schematic diagram of the hardware structure of a computer device according to an embodiment of the present application.

[0101] The computer device may include a processor 81 and a memory 82 storing computer program instructions.

[0102] Specifically, the processor 81 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present application.

[0103] Among them, the memory 82 may include a large-capacity memory for data or instructions. By way of example and not limitation, the memory 82 may include a hard disk drive (HDD), a floppy disk drive, a solid-state drive (SSD), a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 82 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 82 may be inside or outside the data processing device. In a specific embodiment, the memory 82 is a non-volatile memory. In a specific embodiment, the memory 82 includes a read-only memory (ROM) and a random access memory (RAM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (Programmable Read-Only Memory, PROM for short), an erasable PROM (Erasable ProgrammableRead-Only Memory, EPROM for short), an electrically erasable PROM (Electrically Erasable ProgrammableRead-Only Memory, EEPROM for short), an electrically alterable ROM (Electrically Alterable Read-Only Memory, EAROM for short) or a flash memory (FLASH) or a combination of two or more of these. Under appropriate circumstances, the RAM can be a static random access memory (SRAM) or a dynamic random access memory (DRAM), wherein the DRAM can be a fast page mode dynamic random access memory (FPMDRAM), an extended data output dynamic random access memory (EDODRAM), a synchronous dynamic random access memory (SDRAM), etc.

[0104] The memory 82 may be used to store or cache various data files required for processing and / or communication, as well as possible computer program instructions executed by the processor 81 .

[0105] The processor 81 implements any one of the task execution methods in the above embodiments by reading and executing computer program instructions stored in the memory 82 .

[0106] In some of these embodiments, the computer device may further include a communication interface 83 and a bus 80. Figure 5 As shown, the processor 81, the memory 82, and the communication interface 83 are connected via a bus 80 and communicate with each other.

[0107] The communication interface 83 is used to implement communication between the modules, devices, units and / or equipment in the embodiment of the present application. The communication port 83 can also implement data communication with other components such as: external devices, image / data acquisition equipment, databases, external storage, and image / data processing workstations.

[0108] The bus 80 includes hardware, software or both, and couples the components of the computer device to each other. The bus 80 includes but is not limited to at least one of the following: a data bus, an address bus, a control bus, an expansion bus, and a local bus. By way of example and not limitation, bus 80 may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side Bus (FSB), a Hyper Transport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a Micro Channel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable buses or a combination of two or more of these. Where appropriate, bus 80 may include one or more buses. Although embodiments of the present application describe and illustrate a particular bus, the present application contemplates any suitable bus or interconnect.

[0109] The computer device can realize the combination based on the acquired task allocation information. Figure 1 Describes the task execution method.

[0110] In addition, in combination with the task execution method in the above embodiment, the embodiment of the present application can provide a computer-readable storage medium for implementation. The computer-readable storage medium stores computer program instructions; when the computer program instructions are executed by a processor, any one of the task execution methods in the above embodiment is implemented.

[0111] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0112] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A task execution method based on the spark kernel, characterized in that: The following steps are involved: In the task flow generation step, the spark driver node reads and generates a task flow according to a configuration file; Task allocation step: Akka allocates tasks according to the task flow; In the task execution step, a scheduler distributes the task to the corresponding execution node according to the information of the assigned task, and sends the execution result to the spark driver node; The task flow generation step specifically includes: By writing spark and akka codes, the configuration file is read and parsed to generate a dag graph, each leaf node in the dag graph corresponding to the task is generated according to the configuration file, and the task corresponding to the configuration file is split into several subtasks to execute the task corresponding to the task request, and the total number of nodes of the dag graph is determined according to the configuration file, which is used to determine when the task is completed when the downstream nodes are generated by executing the task later; The task allocation step specifically includes: Traverse each task of the dag graph and determine its status. If it is not executed, determine whether there is a parent node task. If there is a parent node task, wait for the parent node task to be successfully executed. If there is no parent node task or all parent node tasks have been successfully executed, distribute the task to the executor for execution through the actor model; The task execution steps specifically include: Sending the execution result to the spark driver node through the actor model, and the spark driver node receives and records its status; When the task fails to execute, a retry is performed. When the number of retries exceeds a preset number, the task is determined to have failed and the program is exited, and an alarm message is output; When the task is successfully executed, continue to execute the sub-node tasks until all the tasks are completed; The configuration file uses the json format to define the data source, synchronization data format and specify data conversion logic.

2. A task execution system based on the spark kernel, applying the task execution method based on the spark kernel described in claim 1, characterized in that: include: The task flow generation module, the spark driver node reads and generates a task flow according to a configuration file; Task allocation module, Akka allocates tasks according to the task flow; A task execution module, a scheduler distributes tasks to corresponding execution nodes according to the information of the assigned tasks, and sends the execution results to the spark driver node; The task flow generation module writes spark and akka codes, reads and parses the configuration file to generate a dag graph, generates leaf nodes in the dag graph corresponding to the task according to the configuration file, and splits the task corresponding to the configuration file into several subtasks to execute the task corresponding to the task request, and determines the total number of nodes of the dag graph according to the configuration file, which is used to determine when the task is completed when the downstream nodes are generated by executing the task later; The task allocation module traverses each task of the dag graph and determines its status. If it is not executed, it determines whether there is a parent node task. If there is a parent node task, it waits for the parent node task to be successfully executed. If there is no parent node task or all parent node tasks have been successfully executed, the task is distributed to the executor for execution through the actor model. Wherein, the task execution module includes: A state recording unit, which sends the execution result to the spark driver node through the actor model, and the spark driver node receives and records its state; A status judgment unit, when the task fails to execute, retries are performed, and when the number of retries exceeds a preset number, the task is determined to have failed and the program is exited, and an alarm message is output; when the task is successfully executed, the sub-node tasks are continued to be executed until the task is fully completed; The configuration file uses the json format to define the data source, synchronization data format and specify data conversion logic.

3. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the spark kernel-based task execution method as described in claim 1 is implemented.