A data job scheduling method and system based on role model and message mailbox

Abstract

The present invention provides a data job scheduling method and system based on a role model and a message mailbox, comprising: defining the role type of a task node according to a data processing function; defining a message mailbox according to different business scenarios; defining the message type of the role based on the event type processed by the role; determining the data job flow based on the data business, and configuring the data job based on the data job flow; establishing a scheduling flow based on the data job configuration result to complete data job scheduling. The present invention does not require complex scheduling logic, does not need to wait for other concurrent threads, implements supervision of task nodes, supports restarting nodes when task nodes are abnormal, and does not need to restart the entire task, thereby implementing data job scheduling based on a role model and a message mailbox.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a data job scheduling method and system based on a role model and a message mailbox. Background Art

[0002] In recent years, with the vigorous development of cloud computing and Internet technology, data has shown an explosive and continuous growth pattern, and the era of big data has quietly arrived. How to efficiently schedule user jobs has become an urgent problem to be solved.

[0003] The general process of the existing job scheduling method is as follows: create a global scheduler and create a scheduler for each task; the task scheduler is responsible for task scheduling, and the scheduler communicates with each task node; the task node accepts the scheduler to execute the task, and notifies the scheduler after completing the task; when the task node is multi-threaded concurrently, the node needs to maintain the status of multiple threads, and among multiple threads, the thread that completes first needs to join and wait; after the task is completed, the scheduler modifies the task status.

[0004] However, the existing scheduling methods have the following problems:

[0005] 1) Use a scheduler to coordinate the work between components, which usually requires writing complex scheduling logic code;

[0006] 2) When multiple threads of a task node are running concurrently, state synchronization needs to be achieved through thread joining. The thread that finishes executing first needs to wait, occupying system resources.

[0007] 3) The scheduler needs to maintain the status of each task node, which is complex to maintain and consumes a lot of resources;

[0008] 4) When a task node fails, it is usually necessary to restart or abandon the entire task. Summary of the invention

[0009] The present invention provides a data job scheduling method and system based on a role model and a message mailbox, and realizes data job scheduling based on the role model and the message mailbox.

[0010] The present invention provides a data job scheduling method based on a role model and a message mailbox, comprising:

[0011] Step 1: Define the role type of the task node according to the data processing function;

[0012] Step 2: Define the message mailbox according to different business scenarios;

[0013] Step 3: Define the role's message type based on the event type the role handles;

[0014] Step 4: Determine the data operation process based on the data service, and perform data operation configuration based on the data operation process;

[0015] Step 5: Based on the data job configuration results, establish a scheduling process to complete the data job scheduling.

[0016] In one possible way;

[0017] In step 1, the role types defined include:

[0018] Data collector: used to read source data and send the data stream to the next node message mailbox;

[0019] Processor: used to read the data in the message mailbox, process the data according to the preset rules, and send the results to the message mailbox of the next node;

[0020] Writer: used to read data from the message mailbox and write the data into various data carriers;

[0021] Supervisor: used to supervise the data collectors, processors, and writers. When the data collectors, processors, and writers encounter problems that they cannot solve themselves, the supervisors will handle them in a unified manner;

[0022] Task Manager: Used to manage created tasks.

[0023] In one possible way;

[0024] In step 2, the message mailboxes defined include:

[0025] General mailbox: a regular message mailbox, where messages follow the first-in-first-out principle, and each role has an independent mailbox;

[0026] Priority mailbox: Based on the general mailbox, the priority of the message can be set, and the role will read the high-priority messages first;

[0027] Fast mailbox: There are multiple senders, one receiver, and overflowing messages are discarded;

[0028] Shared mailbox: A mailbox shared by all roles.

[0029] In one possible way;

[0030] In step 3, the message types of the role are defined as: start message, end message, data message, exception message, and broadcast message.

[0031] In one possible way;

[0032] Determining a data operation process based on the data service, and configuring the data operation based on the data operation process includes:

[0033] Analyze the data service to determine a plurality of subtasks, sort the plurality of subtasks, and establish a sequence identifier for each subtask according to the sorting result;

[0034] According to the attributes of the plurality of subtasks, a task identifier is established for each subtask;

[0035] Based on the sequence identifier and the task identifier, the data operation process is established, and based on the data operation process, the task nodes of the data service are deployed;

[0036] Perform data operation configuration on the task node according to the task characteristics of the task node.

[0037] In one possible way;

[0038] In step 5, based on the data job configuration results, a scheduling process is established to complete the data job scheduling, including:

[0039] Based on the data operation process, determine the scheduling task, create a task manager for the scheduling task, and send a start message to the message mailbox corresponding to the task manager;

[0040] After receiving the start message, the task manager parses the task configuration based on the data job configuration result and obtains the configuration resources;

[0041] According to the configuration resources, the task manager creates corresponding types of data takers, processors, and data writers, and sends a startup message to the message mailboxes corresponding to the data takers, processors, and data writers respectively;

[0042] After receiving the start message, the data acquirer initializes the data source according to its own type, starts to read data, and packages the read data into a data message type to obtain the target data, and sends the target data to the message mailbox of the processor corresponding to the next node;

[0043] When it is detected that the data fetcher has completed data reading, an end message is sent to the message mailbox of the processor corresponding to the next node, and the resources configured for the data fetcher are recovered;

[0044] After receiving the start message and the target data, the processor processes the target data according to the configuration rules to obtain result data, packages the result data into a data message type, and sends it to the next node. The type of the next node corresponding to the processor is other processors and writers;

[0045] When it is detected that the processor has completed data processing, an end message is sent to the corresponding message mailbox of the next node, and the resources configured for the processor are recovered;

[0046] After receiving the start message and the result data, the writer parses the result data to obtain the write data, and writes the write data into a pre-configured data carrier;

[0047] When it is detected that the writer has completed data writing, the resources configured for the writer are recovered;

[0048] When the scheduled task is detected to be completed, a completion message is sent to the task manager;

[0049] After receiving the end message, the task manager updates the parameters and status of the scheduled task and reclaims all resources configured for the scheduled task.

[0050] In one possible way;

[0051] In step 5, based on the data job configuration results, a scheduling process is established to complete the data job scheduling, which also includes:

[0052] When it is detected that an unhandled exception occurs in the process of processing the scheduling task by the data acquirer, processor, or writer, an error message is sent to the task manager;

[0053] After receiving the error message, the task manager destroys the erroneous data collector, processor, and writer, and creates a new data collector, processor, and writer to continue to execute the corresponding task;

[0054] When the task manager is unable to fix the anomaly, an error message is sent to the supervisor;

[0055] When the supervisor receives the error message sent by the task manager, it reclaims the resources of the task manager and its corresponding data getters, processors, and writers, and restarts or discards the scheduled task according to the system exception handling mechanism configuration.

[0056] In one possible way;

[0057] According to the task characteristics of the task node, configuring the data operation of the task node includes:

[0058] Obtaining the position information of the task node in the entire task node graph, and parsing the subtasks corresponding to the task node to determine the task type;

[0059] Based on the location information and the task type, setting a role type for the task node, and performing a first data operation configuration for the task node according to the role type;

[0060] After the first data job configuration is completed, role instance data is set for the task node according to the data processing capability of the task node, and a second data job configuration is performed for the task node based on the role instance data;

[0061] After the second data job configuration is completed, a third data job configuration is performed on the task node according to the task characteristics of the task node, specifically:

[0062] When the role type is a task manager, configuring input source information for the task node;

[0063] When the role type is a data getter, configuring data type conversion information for the task node;

[0064] When the role type is processor, configuring data rule processing information for the task node;

[0065] When the role type is a writer, configuring data carrier information for the task node;

[0066] When the role type is supervisor, configuring processing mode information for the task node;

[0067] After the third data operation configuration is completed, a transfer path between the task nodes is determined according to the data processing relationship between the task nodes, and a fourth data operation configuration is performed on the task nodes according to the transfer path.

[0068] In one possible way;

[0069] The task node graph is determined according to the deployment result of the task node for deploying the data service.

[0070] A data job scheduling system based on a role model and a message mailbox, comprising:

[0071] Definition module, used to define role types, message mailboxes, and message types according to data processing functions;

[0072] A configuration module, used to determine a data operation process according to a data service, and perform data operation configuration based on the data operation process;

[0073] The scheduling module is used to establish a scheduling process and complete data job scheduling based on the data job configuration results.

[0074] The present invention has the following advantages:

[0075] (1) Jobs are message-driven and do not require complex scheduling logic.

[0076] (2) The state of the node is maintained by the node itself, thus effectively avoiding concurrency issues such as consistency and atomicity.

[0077] (3) Multi-thread state synchronization is achieved through the message broadcast mode, and threads do not need to wait for other concurrent threads.

[0078] (4) The multi-level supervision mode of task managers and supervisors supports node restart when task nodes run abnormally, without restarting the entire task.

[0079] The protection points of the present invention are:

[0080] 1. A data job scheduling method based on a role model and a message mailbox, including but not limited to the definition and classification of role types, the definition and classification of message mailboxes, the definition and classification of message types, and a data job scheduling process driven by message types.

[0081] 2. Data job scheduling system based on role model and message mailbox, including task configuration management, task scheduling management, task overview, etc.

[0082] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0083] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0085] Figure 1 It is a flow chart of a data job scheduling method based on a role model and a message mailbox in an embodiment of the present invention;

[0086] Figure 2 Another flow chart of a data job scheduling method based on a role model and a message mailbox in an embodiment of the present invention;

[0087] Figure 3 The structure diagram of a data job scheduling system based on a role model and a message mailbox in an embodiment of the present invention. DETAILED DESCRIPTION

[0088] The preferred embodiments of the present invention are described below in conjunction with the accompanying drawings. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0089] Example 1

[0090] A data job scheduling method based on role model and message mailbox, such as Figure 1 As shown, including:

[0091] Step 1: Define the role type of the task node according to the data processing function;

[0092] Step 2: Define the message mailbox according to different business scenarios;

[0093] Step 3: Define the role's message type based on the event type the role handles;

[0094] Step 4: Determine the data operation process based on the data service, and perform data operation configuration based on the data operation process;

[0095] Step 5: Based on the data job configuration results, establish a scheduling process to complete the data job scheduling.

[0096] In this embodiment, the task node is a node for executing one of the tasks determined by the data service.

[0097] In this embodiment, the role type is an abstraction of all task nodes in the entire scheduling system. Each role receives the message of the previous role, performs corresponding actions according to the internal business logic of the role, and sends the message to the subsequent role. Each role is a message consumer of the previous role and a message producer of the next role. The color calibration types include: data collector, which is mainly used for reading data, processor, which is mainly used for processing data, writer, which is mainly used for writing data, supervisor, which is mainly used for supervising other roles, and task manager, which is used to create tasks.

[0098] In this embodiment, the message mailbox is a carrier for message transmission between all roles. According to different business scenarios, the message mailbox can be divided into a general mailbox, a priority mailbox, a quick mailbox, and a shared mailbox.

[0099] In this embodiment, the role transmits messages through mailboxes and performs corresponding actions based on the messages. The entire scheduling system is driven by messages. The message type defines the event type that the role or task node can handle, including start messages, end messages, data messages, exception messages, and broadcast messages.

[0100] In this embodiment, the data operation process refers to a service execution process determined according to the data service.

[0101] In this embodiment, the data job configuration is specifically to configure the role, message mailbox, and message type of the task node according to the data service.

[0102] In this embodiment, the scheduling process is a scheduling of reading, writing, processing, etc. of data determined according to data services.

[0103] The beneficial effects of the above design scheme are: by defining role types, message mailboxes, and message types, the work between each task node is realized, and the work is driven according to the message type. There is no need for complex scheduling logic. The state of each task node is maintained by the node itself, avoiding concurrency problems such as consistency and atomicity. Multi-threaded state synchronization is achieved through message mailboxes without waiting for other concurrent threads. By setting the roles of task managers and supervisors, task nodes can be supervised, and the restart of nodes when task nodes are abnormal is supported without restarting the entire task, thereby realizing data job scheduling based on role models and message mailboxes.

[0104] Example 2

[0105] Based on Example 1, the embodiment of the present invention provides a data job scheduling method based on a role model and a message mailbox. In step 1, the defined role types include:

[0106] Data collector: used to read source data and send the data stream to the next node message mailbox;

[0107] Processor: used to read the data in the message mailbox, process the data according to the preset rules, and send the results to the message mailbox of the next node;

[0108] Writer: used to read data from the message mailbox and write the data into various data carriers;

[0109] Supervisor: used to supervise the data collectors, processors, and writers. When the data collectors, processors, and writers encounter problems that they cannot solve themselves, the supervisors will handle them in a unified manner;

[0110] Task Manager: Used to manage created tasks.

[0111] In this embodiment, the data acquirer can be subdivided into MysqlReader and OracleReader for reading relational databases, KafkaReader and RabbitMQReader for reading middleware, and HttpReader and WebServiceReader for reading data interfaces, according to different data source types.

[0112] In this embodiment, the processors can be subdivided into data conversion processors, data filtering processors, data statistics processors, data desensitization processors, data encryption processors, and data decryption processors according to different data processing rules.

[0113] In this embodiment, the writer can be subdivided into MysqlWriter, SqlServerWriter, MongoDBWriter, etc. according to different data carriers.

[0114] In this embodiment, the processing methods uniformly handled by the supervisor include: d) interrupting the current job scheduling, destroying the current erroneous role, creating a new role and trying again, etc.

[0115] The beneficial effect of the above design is: by defining different types of roles and clarifying different divisions of labor, a basis is provided for realizing task scheduling.

[0116] Example 3

[0117] Based on Example 1, the embodiment of the present invention provides a data job scheduling method based on a role model and a message mailbox. In step 2, the defined message mailbox includes:

[0118] General mailbox: a regular message mailbox, where messages follow the first-in-first-out principle, and each role has an independent mailbox;

[0119] Priority mailbox: Based on the general mailbox, the priority of the message can be set, and the role will read the high-priority messages first;

[0120] Fast mailbox: There are multiple senders, one receiver, and overflowing messages are discarded;

[0121] Shared mailbox: A mailbox shared by all roles.

[0122] The beneficial effect of the above design is that by defining the type of mailbox as a carrier for message transmission between task nodes or roles, a basis is provided for implementing task scheduling.

[0123] Example 4

[0124] Based on Example 1, an embodiment of the present invention provides a data job scheduling method based on a role model and a message mailbox, characterized in that in step 3, the message types defining the roles include: start message, end message, data message, exception message, and broadcast message.

[0125] In this embodiment, the message type defines the event type that the role can handle, and the entire system is driven to work based on the message.

[0126] In this embodiment, the startup message is used to initialize related resources and enter the ready state after receiving the startup message; the end message is used to release its own resources and destroy its own objects after receiving the end message; the data message is the business data that needs to be processed by the task node; the exception information is when the task node encounters a problem that it cannot solve by itself, it will send this information to the supervisor, and the supervisor will solve the problem; the broadcast message is used for state synchronization during multi-threaded concurrency. The thread that completes first will send a broadcast to all threads, and other threads will perform counting operations after receiving the broadcast information. The last completed thread modifies the task node state.

[0127] The beneficial effect of the above design is that by defining the message type, the scheduling system works through message drive, does not require complex scheduling logic, and simplifies the scheduling process.

[0128] Example 5

[0129] Based on Example 1, the embodiment of the present invention provides a data job scheduling method based on a role model and a message mailbox, such as Figure 2 As shown, based on the data service, determining the data operation process, and based on the data operation process, performing data operation configuration includes:

[0130] Step 41: Analyze the data service, determine multiple subtasks, sort the multiple subtasks, and establish a sequence identifier for each subtask according to the sorting result;

[0131] Step 42: creating a task identifier for each subtask according to the attributes of the plurality of subtasks;

[0132] Step 43: establishing the data operation process based on the sequence identifier and the task identifier, and deploying the task nodes of the data service based on the data operation process;

[0133] Step 44: Perform data operation configuration on the task node according to the task characteristics of the task node.

[0134] In this embodiment, the plurality of subtasks include reading, processing, and writing data.

[0135] In this embodiment, the plurality of subtasks are sorted according to the order in which the subtasks are executed.

[0136] In this embodiment, the attributes of the multiple subtasks are reading, processing, writing, etc.

[0137] In this embodiment, the data operation flow includes the execution time and execution functions of multiple subtasks.

[0138] In this embodiment, the task nodes correspond to role types.

[0139] In this embodiment, the task feature of the task node is related to the task performed by the task node.

[0140] The beneficial effect of the above design scheme is: by determining the data operation process based on the data business, and configuring the data operation based on the data operation process, a resource basis is provided for data operation scheduling.

[0141] Example 6

[0142] Based on Example 1, the embodiment of the present invention provides a data job scheduling method based on a role model and a message mailbox. In step 5, a scheduling process is established based on the data job configuration result to complete the data job scheduling, including:

[0143] Based on the data operation process, determine the scheduling task, create a task manager for the scheduling task, and send a start message to the message mailbox corresponding to the task manager;

[0144] After receiving the start message, the task manager parses the task configuration based on the data job configuration result and obtains the configuration resources;

[0145] According to the configuration resources, the task manager creates corresponding types of data takers, processors, and data writers, and sends a startup message to the message mailboxes corresponding to the data takers, processors, and data writers respectively;

[0146] After receiving the start message, the data acquirer initializes the data source according to its own type, starts to read data, and packages the read data into a data message type to obtain the target data, and sends the target data to the message mailbox of the processor corresponding to the next node;

[0147] When it is detected that the data fetcher has completed data reading, an end message is sent to the message mailbox of the processor corresponding to the next node, and the resources configured for the data fetcher are recovered;

[0148] After receiving the start message and the target data, the processor processes the target data according to the configuration rules to obtain result data, packages the result data into a data message type, and sends it to the next node. The type of the next node corresponding to the processor is other processors and writers;

[0149] When it is detected that the processor has completed data processing, an end message is sent to the corresponding message mailbox of the next node, and the resources configured for the processor are recovered;

[0150] After receiving the start message and the result data, the writer parses the result data to obtain the write data, and writes the write data into a pre-configured data carrier;

[0151] When it is detected that the writer has completed data writing, the resources configured for the writer are recovered;

[0152] When the scheduled task is detected to be completed, a completion message is sent to the task manager;

[0153] After receiving the end message, the task manager updates the parameters and status of the scheduled task and reclaims all resources configured for the scheduled task.

[0154] In this embodiment, the configuration resources are resources that support roles and task processing between roles.

[0155] In this embodiment, the processor's configuration rules are pre-set according to the processing tasks of this task node.

[0156] The beneficial effect of the above design scheme is: by creating task managers, data getters, processors, and data writers, data job scheduling based on role models and message mailboxes is implemented through message mailboxes according to the message type, thereby meeting the requirements of no need for complex scheduling logic, self-maintenance of node status, and no need for threads to wait, thereby avoiding resource consumption.

[0157] Example 7

[0158] Based on Example 6, the embodiment of the present invention provides a data job scheduling method based on a role model and a message mailbox. In step 5, a scheduling process is established based on the data job configuration result to complete the data job scheduling, and the method further includes:

[0159] When it is detected that an unhandled exception occurs during the processing of the scheduling task by the data acquirer, processor, or writer, an error message is sent to the task manager;

[0160] After receiving the error message, the task manager destroys the erroneous data collector, processor, and writer, and creates a new data collector, processor, and writer to continue to execute the corresponding task;

[0161] When the task manager is unable to fix the anomaly, an error message is sent to the supervisor;

[0162] When the supervisor receives the error message sent by the task manager, it reclaims the resources of the task manager and its corresponding data getters, processors, and writers, and restarts or discards the scheduled task according to the system exception handling mechanism configuration.

[0163] The beneficial effects of the above design are: monitoring task scheduling through the supervisor, and supporting the destruction of roles and the re-creation of new roles after the role runs abnormally, without restarting the entire task, thereby ensuring the efficiency of data job scheduling.

[0164] Example 8

[0165] Based on Example 5, the embodiment of the present invention provides a data job scheduling method based on a role model and a message mailbox, and configuring the data job of the task node according to the task characteristics of the task node includes:

[0166] Obtaining the position information of the task node in the entire task node graph, and parsing the subtasks corresponding to the task node to determine the task type;

[0167] Based on the location information and the task type, setting a role type for the task node, and performing a first data operation configuration for the task node according to the role type;

[0168] After the first data job configuration is completed, role instance data is set for the task node according to the data processing capability of the task node, and a second data job configuration is performed for the task node based on the role instance data;

[0169] After the second data job configuration is completed, a third data job configuration is performed on the task node according to the task characteristics of the task node, specifically:

[0170] When the role type is a task manager, configuring input source information for the task node;

[0171] When the role type is a data getter, configuring data type conversion information for the task node;

[0172] When the role type is processor, configuring data rule processing information for the task node;

[0173] When the role type is a writer, configuring data carrier information for the task node;

[0174] When the role type is supervisor, configuring processing mode information for the task node;

[0175] After the third data operation configuration is completed, a transfer path between the task nodes is determined according to the data processing relationship between the task nodes, and a fourth data operation configuration is performed on the task nodes according to the transfer path.

[0176] In this embodiment, performing a first data operation on the task node configures resources for the role type of the task node.

[0177] In this embodiment, the role instance data is a data processing instance in the historical data job scheduling of the role.

[0178] In this embodiment, performing the second data operation configuration on the task node provides an example basis for data processing of the task node, thereby ensuring the operation of the task node.

[0179] In this embodiment, performing the third data operation configuration on the task node provides configuration resources for a specific execution task of the task node.

[0180] In this embodiment, performing the fourth data operation configuration on the task nodes provides configuration resources for message transmission between task nodes.

[0181] The beneficial effect of the above design scheme is: by configuring data operations for the task nodes according to the task characteristics of the task nodes, the normal operation of the task nodes is ensured, and a resource basis is provided for data operation scheduling.

[0182] Example 9

[0183] Based on Example 8, an embodiment of the present invention provides a data job scheduling method based on a role model and a message mailbox, wherein the task node graph is determined according to the deployment result of the task node for deploying the data service.

[0184] The beneficial effect of the above design scheme is: providing a basis for data job configuration of task nodes.

[0185] Example 10

[0186] A data job scheduling system based on role model and message mailbox, such as Figure 3 As shown, including:

[0187] Definition module, used to define role types, message mailboxes, and message types according to data processing functions;

[0188] A configuration module, used to determine a data operation process according to a data service, and perform data operation configuration based on the data operation process;

[0189] The scheduling module is used to establish a scheduling process and complete data job scheduling based on the data job configuration results.

[0190] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A data job scheduling method based on role model and message mailbox, characterized in that: include: Step 1: Define the role type of the task node according to the data processing function; Step 2: Define the message mailbox according to different business scenarios; Step 3: Define the role's message type based on the event type the role handles; Step 4: Determine the data operation process based on the data service, and perform data operation configuration based on the data operation process; Step 5: Based on the data job configuration results, establish a scheduling process to complete the data job scheduling; In step 5, based on the data job configuration results, a scheduling process is established to complete the data job scheduling, including: Based on the data operation process, determine the scheduling task, create a task manager for the scheduling task, and send a start message to the message mailbox corresponding to the task manager; After receiving the start message, the task manager parses the task configuration based on the data job configuration result and obtains the configuration resources; According to the configuration resources, the task manager creates corresponding types of data takers, processors, and data writers, and sends a startup message to the message mailboxes corresponding to the data takers, processors, and data writers respectively; After receiving the start message, the data acquirer initializes the data source according to its own type, starts to read data, and packages the read data into a data message type to obtain the target data, and sends the target data to the message mailbox of the processor corresponding to the next node; When it is detected that the data fetcher has completed data reading, an end message is sent to the message mailbox of the processor corresponding to the next node, and the resources configured for the data fetcher are recovered; After receiving the start message and the target data, the processor processes the target data according to the configuration rules to obtain result data, packages the result data into a data message type, and sends it to the next node. The type of the next node corresponding to the processor is other processors and writers; When it is detected that the processor has completed data processing, an end message is sent to the corresponding message mailbox of the next node, and the resources configured for the processor are recovered; After receiving the start message and the result data, the writer parses the result data to obtain the write data, and writes the write data into a pre-configured data carrier; When it is detected that the writer has completed data writing, the resources configured for the writer are recovered; When the scheduled task is detected to be completed, a completion message is sent to the task manager; After receiving the end message, the task manager updates the parameters and status of the scheduled task and reclaims all resources configured for the scheduled task.

2. A data job scheduling method based on role model and message mailbox according to claim 1, characterized in that: In step 1, the role types defined include: Data collector: used to read source data and send the data stream to the next node message mailbox; Processor: used to read the data in the message mailbox, process the data according to the preset rules, and send the results to the message mailbox of the next node; Writer: used to read data from the message mailbox and write the data into various data carriers; Supervisor: used to supervise the data collectors, processors, and writers. When the data collectors, processors, and writers encounter problems that they cannot solve themselves, the supervisors will handle them in a unified manner; Task Manager: Used to manage created tasks.

3. A data job scheduling method based on role model and message mailbox according to claim 1, characterized in that: In step 2, the message mailboxes defined include: General mailbox: a regular message mailbox, where messages follow the first-in-first-out principle, and each role has an independent mailbox; Priority mailbox: Based on the general mailbox, the priority of the message can be set, and the role will read the high-priority messages first; Fast mailbox: There are multiple senders, one receiver, and overflowing messages are discarded; Shared mailbox: A mailbox shared by all roles.

4. A data job scheduling method based on role model and message mailbox according to claim 1, characterized in that: In step 3, the message types of the role are defined as: start message, end message, data message, exception message, and broadcast message.

5. The data job scheduling method based on role model and message mailbox according to claim 1 is characterized in that: Determining a data operation process based on the data service, and configuring the data operation based on the data operation process includes: Analyze the data service to determine a plurality of subtasks, sort the plurality of subtasks, and establish a sequence identifier for each subtask according to the sorting result; According to the attributes of the plurality of subtasks, a task identifier is established for each subtask; Based on the sequence identifier and the task identifier, the data operation process is established, and based on the data operation process, the task nodes of the data service are deployed; Perform data operation configuration on the task node according to the task characteristics of the task node.

6. A data job scheduling method based on role model and message mailbox according to claim 1, characterized in that: In step 5, based on the data job configuration results, a scheduling process is established to complete the data job scheduling, which also includes: When it is detected that an unhandled exception occurs in the process of processing the scheduling task by the data acquirer, processor, or writer, an error message is sent to the task manager; After receiving the error message, the task manager destroys the erroneous data collector, processor, and writer, and creates a new data collector, processor, and writer to continue to execute the corresponding task; When the task manager is unable to fix the anomaly, an error message is sent to the supervisor; When the supervisor receives the error message sent by the task manager, it reclaims the resources of the task manager and its corresponding data getters, processors, and writers, and restarts or discards the scheduled task according to the system exception handling mechanism configuration.

7. A data job scheduling method based on role model and message mailbox according to claim 5, characterized in that: According to the task characteristics of the task node, configuring the data operation of the task node includes: Obtaining the position information of the task node in the entire task node graph, and parsing the subtasks corresponding to the task node to determine the task type; Based on the location information and the task type, setting a role type for the task node, and performing a first data operation configuration for the task node according to the role type; After the first data job configuration is completed, role instance data is set for the task node according to the data processing capability of the task node, and a second data job configuration is performed for the task node based on the role instance data; After the second data job configuration is completed, a third data job configuration is performed on the task node according to the task characteristics of the task node, specifically: When the role type is a task manager, configuring input source information for the task node; When the role type is a data getter, configuring data type conversion information for the task node; When the role type is processor, configuring data rule processing information for the task node; When the role type is a writer, configuring data carrier information for the task node; When the role type is supervisor, configuring processing mode information for the task node; After the third data operation configuration is completed, a transfer path between the task nodes is determined according to the data processing relationship between the task nodes, and a fourth data operation configuration is performed on the task nodes according to the transfer path.

8. A data job scheduling method based on role model and message mailbox according to claim 7, characterized in that: The task node graph is determined according to the deployment result of the task node for deploying the data service.

9. A data job scheduling system based on a role model and a message mailbox, used to implement the scheduling method according to any one of claims 1 to 8, characterized in that: include: Definition module, used to define role types, message mailboxes, and message types according to data processing functions; A configuration module, used to determine a data operation process according to a data service, and perform data operation configuration based on the data operation process; The scheduling module is used to establish a scheduling process and complete data job scheduling based on the data job configuration results.

Images