Data processing method and device, equipment, storage medium and program product

Abstract

The invention provides a data processing method and device, equipment, a storage medium and a program product, and relates to the technical field of computers.The data processing method comprises the steps that a task chain is constructed according to task identifiers of data tasks and the execution sequence of the data tasks; in response to an execution instruction for the task chain, executing a first data task of the task chain; and in response to the fact that the first data task is not the last data task of the task chain, executing the second data task according to the execution sequence of the data tasks. The task chain can be flexibly and conveniently constructed, the data task of the task chain can be accurately and automatically executed, and user intervention is not needed.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a data processing method, device, equipment, storage medium and program product. Background Art

[0002] In the application system, many different types of data tasks may be involved. For example, in the financial dispute resolution application system, there are multiple data tasks, including batch case evidence pulling tasks, batch case evidence download and packaging tasks, batch case push tasks, and batch case report generation tasks.

[0003] Currently, the application programming interface (API) provided by CompletableFuture (an asynchronous programming tool class) is usually used to predefine the task scheduling of multiple data tasks in the application, and then a single node and a single process are used to execute each data task. However, executing data tasks in this way is not flexible enough. Summary of the invention

[0004] The present application provides a data processing method, apparatus, device, storage medium and program product to solve the problem of insufficient flexibility in executing data tasks through current methods.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] Build a task chain based on the task identifier of the data task and the execution order of the data task;

[0007] In response to an execution instruction for the task chain, executing a first data task of the task chain;

[0008] In response to the first data task not being the last data task of the task chain, the second data task is executed according to the execution order of the data tasks.

[0009] In a second aspect, the present application provides a data processing device, including:

[0010] A construction module, used to construct a task chain according to the task identifier of the data task and the execution order of the data task;

[0011] An execution module, configured to execute a first data task of the task chain in response to an execution instruction of the task chain;

[0012] The processing module is used for executing the second data task according to the execution order of the data tasks in response to the first data task not being the last data task of the task chain.

[0013] In a third aspect, the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor;

[0014] Memory stores computer-executable instructions;

[0015] The processor executes the computer-executable instructions stored in the memory to implement the data processing method as described in the first aspect of the present application.

[0016] In a fourth aspect, the present application provides a computer-readable storage medium, in which computer program instructions are stored. When the computer program instructions are executed, the data processing method described in the first aspect of the present application is implemented.

[0017] In a fifth aspect, the present application provides a computer program product, including a computer program, which implements the data processing method described in the first aspect of the present application when the computer program is executed.

[0018] The data processing method, device, equipment, storage medium and program product provided by the present application construct a task chain according to the task identifier of the data task and the execution order of the data task, which can realize flexible and convenient task scheduling of the data task, rather than predefining the task scheduling of the data task in a hard-coded manner; in response to the execution instruction of the task chain, the first data task of the task chain is executed, and in response to the first data task not being the last data task of the task chain, the second data task is executed according to the execution order of the data tasks. In the present application, data tasks are managed through task chains, and the execution of the data tasks of the task chain is automatically triggered according to the execution order of the data tasks of the task chain until all the data tasks of the task chain are executed, which can accurately and automatically execute the data tasks of the task chain without user intervention. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0020] Figure 1 A flowchart for processing multiple batch data tasks provided for related technologies;

[0021] Figure 2 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0022] Figure 3 A flowchart of a data processing method provided in one embodiment of the present application;

[0023] Figure 4 A flowchart of a data processing method provided by another embodiment of the present application;

[0024] Figure 5 A schematic diagram of an application flow of a data processing method provided in one embodiment of the present application;

[0025] Figure 6 A schematic diagram of a task chain provided in an embodiment of the present application;

[0026] Figure 7 A schematic diagram of the structure of a data processing device provided in one embodiment of the present application;

[0027] Figure 8 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0029] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0030] First, some technical terms involved in this application are explained:

[0031] A batch data task refers to a data task that needs to process multiple data during execution. The batch data task is considered completed only after all the data in the data task are processed.

[0032] Task scheduling is to complete a certain business by arranging a series of batch data tasks into a chain in sequence. When a batch data task is completed, another one or more batch data tasks are executed until all batch data tasks are completed.

[0033] Batch data refers to a batch of data consisting of one or more data sets with the same data structure.

[0034] A message queue is a container that holds messages during their transmission. A message queue manager acts as an intermediary when relaying messages from the source of the message queue manager to the destination of the message queue manager. The main purpose of a message queue is to provide routing and guarantee the delivery of messages. If the recipient is unavailable when a message is sent, the message queue retains the message until it can be successfully delivered.

[0035] In the application system, there may be many different types of data tasks involved, especially batch data tasks. For example, in the financial dispute resolution application system, there are multiple data tasks, including batch case evidence pulling tasks, batch case evidence downloading and packaging tasks, batch case push tasks, and batch case report generation tasks. And these data tasks have an execution order. For example, the batch case evidence downloading and packaging task must depend on the batch case evidence pulling task, and the batch case report generation task must depend on the batch case push task.

[0036] At present, for multiple data tasks, the API provided by CompletableFuture is usually used to implement the orchestration of multiple data tasks in a Java (an object-oriented programming language) application, and then a single node and a single process are used to process multiple data tasks. Among them, data tasks can be divided into the following four categories according to the number of dependencies of data tasks: zero-dependency data tasks, single-dependency data tasks, dual-dependency data tasks, and multiple-dependency data tasks. Figure 1 A flowchart for processing multiple data tasks provided for related technologies, such as Figure 1 As shown, the process of processing multiple data tasks through CompletableFuture can include the following five steps:

[0037] S101, data task starts, data task 1 and data task 2 are started.

[0038] Data Task 1 and Data Task 2 are zero-dependency data tasks, that is, they do not depend on other data tasks. Therefore, after being triggered, Data Task 1 and Data Task 2 will be executed in parallel.

[0039] S102, execute data task 1.

[0040] S103, execute data task 2.

[0041] Data task 3 and data task 5 are single-dependent data tasks, data task 3 depends on data task 1, and data task 5 depends on data task 2; therefore, when data task 1 is completed, step S104 is executed and data task 3 is executed; when data task 2 is completed, step S105 is executed and data task 5 is executed.

[0042] Data task 4 is a dual-dependent data task. When data task 1 and data task 2 are both completed, step S106 is executed to execute data task 4.

[0043] Data task 3, data task 4 and quantity data task 5 are executed in parallel.

[0044] Data task 6 has multiple dependencies. When data task 3, data task 4 and data task 5 are all completed, step S107 is executed to execute data task 6.

[0045] After data task 6 is completed, it means that all data tasks are completed.

[0046] However, when using CompletableFuture to process multiple data tasks, there are the following disadvantages: (1) The API provided by CompletableFuture uses a hard-coded method to predefine the task scheduling for multiple data tasks. Therefore, there is a problem that the task scheduling is not flexible enough. (2) At present, in order to ensure the high availability of application systems, application systems are usually deployed in clusters to prevent single point failures. CompletableFuture is based on a single node and a single process. Since it does not provide communication between nodes, it cannot be applied to cluster deployment scenarios, and thus cannot give full play to the advantages of the server cluster where the application system is located. In addition, CompletableFuture mainly processes simple data tasks with small data volumes. For data tasks with large data volumes and long execution times, the execution time of a single node is long, resulting in low efficiency. (3) CompletableFuture does not provide detailed information on data task processing, making it impossible to determine when a data task is executed and when it is completed, and it is impossible to report and count the execution time of data tasks.

[0047] Based on the above problems, the present application provides a data processing method, which allows users to configure task chains according to actual business needs and currently existing data tasks, wherein users can independently select the data tasks included in the task chain and arrange the dependencies between the data tasks, so that the task chain can be constructed according to the user's configuration operations, and each data task in the task chain can be automatically executed according to the execution order of each data task in the task chain, thereby realizing flexible and convenient task scheduling of the task chain and automatic execution of each data task in the task chain without user intervention.

[0048] In the following, the application scenarios of the solutions provided in this application are first illustrated.

[0049] Figure 2 This is a schematic diagram of an application scenario provided by an embodiment of the present application. Figure 2 As shown, the application scenario may include: a client 21 and a server cluster 22. The server cluster 22 includes multiple servers 221 (i.e., nodes) and a storage 222, and the client 21 communicates with the server cluster 22 via a wireless network or a wired network. In this application scenario, the user configures the task chain through the client 21 according to business needs and currently existing data tasks. The user selects the batch task data corresponding to the data task of the task chain, and then executes the task chain. The nodes in the server cluster 22 (i.e., servers 221) respond to the execution instructions of the task chain and automatically execute each data task in the task chain according to the execution order of each data task in the task chain. The relevant data of the task chain is stored in the storage 222.

[0050] It should be noted that Figure 2 This is only a schematic diagram of an application scenario provided by the embodiment of the present application. Figure 2 does not limit the equipment included in Figure 2 The positional relationship between the devices is limited.

[0051] The technical solution of the present application is described in detail below through specific embodiments. It should be noted that the following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0052] Figure 3 This is a flow chart of a data processing method provided in one embodiment of the present application. The data processing method may be executed by software and / or hardware devices. For example, the hardware device may be a data processing device, and the data processing device may be an electronic device or a processing chip in the electronic device. Figure 3 As shown, the method of the embodiment of the present application includes:

[0053] S301: Construct a task chain according to the task identifier of the data task and the execution order of the data task.

[0054] In the embodiment of the present application, the task chain is constructed based on the configuration operation of the data task corresponding to the business demand. For example, the user can create a new task chain through the interactive interface according to the business demand and the currently existing data tasks, configure the multiple data tasks included in the task chain, and configure the execution order of each data task according to the dependency relationship between different data tasks in the multiple data tasks.

[0055] Optionally, in response to a configuration operation on a data task corresponding to a business requirement, an execution order of the data task is obtained.

[0056] For example, after configuring the data tasks corresponding to the business needs, the execution order of the data tasks can be obtained. Thus, a task chain can be constructed according to the task identifier of the data task and the execution order of the data task. For example, the task chain includes multiple sequentially executed data tasks such as batch case evidence pulling tasks, batch case evidence download and packaging tasks, batch case push tasks, and batch case report generation tasks.

[0057] S302: In response to an execution instruction for a task chain, execute a first data task of the task chain.

[0058] In this step, the execution instruction of the task chain is, for example, input by a user to an electronic device executing an embodiment of the present method, or is sent by other devices to the electronic device executing an embodiment of the present method. The electronic device executing the embodiment of the present method executes the first data task of the task chain in response to the execution instruction of the task chain. Exemplarily, the first data task of the task chain can be determined according to the execution order of the data tasks in the task chain, so as to execute the first data task. Specifically, the first data task can be executed by a server cluster to improve the processing efficiency of the data tasks of the task chain. For how to specifically execute the first data task of the task chain, please refer to the subsequent embodiments.

[0059] Further, optionally, before executing the first data task of the task chain, it can be determined that the first data task meets the execution conditions based on the execution order of the data tasks in the task chain, and then the first data task is executed, and the execution conditions include that the first data task does not depend on other data tasks or that all other data tasks on which it depends have been executed.

[0060] It can be understood that when executing the first data task of the task chain, it is necessary to first determine whether the first data task meets the execution conditions, wherein the execution conditions include that the first data task does not depend on other data tasks or that all other data tasks it depends on have been completed. If the execution conditions are met, the first data task is executed. If the execution conditions are not met, for example, if other data tasks that the first data task depends on have not been completed, it is necessary to wait until all other dependent data tasks have been completed before triggering the execution of the first data task.

[0061] Optionally, before executing the first data task of the task chain, it is also possible to determine whether the first data task is the first data task in the task chain; if the first data task is the first data task in the task chain, the pre-configured batch task data is determined as the batch task data of the first data task; if the first data task is not the first data task in the task chain, then according to the execution order of the data tasks, the previous third data task of the first data task is determined, and the batch task data completed by the third data task is determined as the batch task data of the first data task.

[0062] Exemplarily, the batch task data corresponding to the data task of the task chain configured by the user through the interactive interface is, for example, 100 business data. If the first data task is the first data task of the task chain, the pre-configured 100 business data are determined as the batch task data of the first data task. If the first data task is not the first data task of the task chain, the previous third data task of the first data task can be determined according to the execution order of the data tasks, and the batch task data completed by the third data task can be determined as the batch task data of the first data task. For example, the pre-configured batch task data is, for example, 100 business data, if the batch task data completed by the third data task is 90 business data, then these 90 business data are determined as the batch task data of the first data task. Among them, the batch task data completed by the third data task can be understood as the batch task data that has been successfully executed.

[0063] S303: In response to the first data task not being the last data task of the task chain, executing the second data task according to the execution order of the data tasks.

[0064] Exemplarily, a task chain is constructed based on the task identifier of the data task and the execution order of the data tasks. During the execution of the first data task, it can be determined whether the first data task is the last data task in the task chain, or it can be determined whether the first data task is the last data task in the task chain when the first data task is completed. If the first data task is not the last data task in the task chain, it means that there are still data tasks in the task chain that need to be processed, and the second data task is executed according to the execution order of the data tasks. The execution process of the second data task can refer to the above-mentioned steps S302 and S303, and so on, until all the data tasks in the task chain are executed. If the first data task is the last data task in the task chain, it means that all data tasks in the task chain have been executed and the entire task chain processing is completed.

[0065] The data processing method provided in the embodiment of the present application constructs a task chain according to the task identifier of the data task and the execution order of the data task, so as to realize flexible and convenient task scheduling of the data task, instead of predefining the task scheduling of the data task in a hard-coded manner; in response to the execution instruction of the task chain, the first data task of the task chain is executed, and in response to the first data task not being the last data task of the task chain, the second data task is executed according to the execution order of the data tasks. In the embodiment of the present application, the data tasks are managed through the task chain, and the execution of the data tasks of the task chain is automatically triggered according to the execution order of the data tasks of the task chain until all the data tasks of the task chain are executed, so that the data tasks of the task chain can be accurately and automatically executed without user intervention.

[0066] Based on the above embodiments, considering the processing of data tasks, it can be processed in the application system of one node, or each batch task data of the data task is stored in the message queue, and consumed by all nodes of the server cluster where the application system is located. Usually, for data tasks with relatively small data volume and less execution time, they can be processed in the application system of one node; if the amount of data involved in a data task is relatively large, and the processing time of each batch task data is also long, you can consider using the message queue processing method to let all nodes in the server cluster participate in the processing. For example, suppose there is a data task that needs to process 100,000 data, and the processing time of each batch task data is 1 second. If a single node executes 10 threads, the execution time is: 100,000 * 1 second / 10 threads = 10,000 seconds; if the server cluster contains 10 nodes, each node corresponds to 10 threads, and the execution time is: 100,000 * 1 second / 10 threads / 10 nodes = 1,000 seconds, which provides 10 times the performance of single-node execution.

[0067] Therefore, considering that the data tasks of the task chain are processed by the nodes in the server cluster, Figure 4 This is a flow chart of a service data processing method provided by another embodiment of the present application. Based on the above embodiment, the present application embodiment further illustrates the data processing method. Figure 4 As shown, the method of the embodiment of the present application may include:

[0068] S401. In response to a configuration operation of a data task corresponding to a business requirement, an execution order of the data task is obtained through a producer node in a server cluster.

[0069] In the embodiment of the present application, the server cluster is a collection of multiple servers, which provides high-performance, high-availability and scalable computing services through mutual collaboration and sharing of resources. Each server in the server cluster can be called a node, that is, the server cluster includes multiple nodes. In this step, the producer node in the server cluster can also be understood as any node in the server cluster. Exemplarily, the user can create a new task chain through the interactive interface according to business needs and the current existing data tasks, configure the multiple data tasks contained in the task chain, and configure the execution order of each data task according to the dependencies between different data tasks in the multiple data tasks. The task chain, for example, includes multiple data tasks such as batch case evidence pulling tasks, batch case evidence download packaging tasks, batch case push tasks, and batch case report generation tasks. Accordingly, in response to the user's configuration operation on the task chain according to business needs and the current existing data tasks, the task chain is stored through the producer node in the server cluster, such as storing the relevant information of the task chain in the database. Table 1 is a task chain table for storing task chain related information provided by an embodiment of the present application. As shown in Table 1, the task chain table includes a task chain name, a task chain number, and a task code (i.e., task identifier) ​​set corresponding to the task chain; wherein the task code set is a set of task identifiers of data tasks determined in the order of execution of all data tasks included in the task chain. According to the task code set, the last data task of the task chain can be determined.

[0070] Table 1

[0071]

[0072] After configuring the task chain, the user can select the batch task data that needs to be processed by the data task of the task chain. For example, the batch task data is for 10,000 users, that is, multiple data tasks included in the task chain are executed for these 10,000 users. Correspondingly, the batch task data is stored in the database through the producer node in the server cluster. Table 2 is a task data table for storing batch task data corresponding to data tasks provided by an embodiment of the present application. As shown in Table 2, the task data table includes a task code (i.e., task identifier), a batch number, a business identifier (IDentity, ID), a status, and an Internet Protocol (IP) address (Internet Protocol, IP) address of the data task execution.

[0073] Table 2

[0074]

[0075]

[0076] In this step, in response to the user's configuration operation on the task chain according to business needs and the currently existing data tasks, the task identifier of the data task and the execution order of the data task are obtained through the producer node in the server cluster. It can be understood that by allowing users to independently select the data tasks included in the task chain according to actual business needs and the currently existing data tasks, and then choreograph the task chain, the arrangement of the task chain can be made more convenient and flexible.

[0077] S402: Build a task chain according to the task identifier of the data task and the execution order of the data task through the producer node in the server cluster.

[0078] In this step, based on Figure 3 The relevant description of S301 in the illustrated embodiment is that after obtaining the task identifier of the data task and the execution order of the data task through the producer node in the server cluster, the task chain can be constructed according to the task identifier of the data task and the execution order of the data task through the producer node in the server cluster.

[0079] In the embodiment of the present application, Figure 3 The step S302 may further include the following three steps S403 to S405:

[0080] S403: In response to the execution instruction of the task chain, the task identifier and batch task data of the first data task are obtained through the producer node of the server cluster.

[0081] It can be understood that the server cluster includes multiple nodes, any of which may be a producer node or a consumer node. The producer node can also be understood as a production node, as a producer of a message queue, for storing batch task messages in the message queue; the consumer node can also be understood as a consumption node, as a consumer of a message queue, for consuming batch task messages in the message queue. Among them, the message queue is a container for storing messages during the transmission of the message. In the embodiment of the present application, batch task messages are stored in a message queue, that is, for each data task included in the task chain, a message queue shared by the data tasks is abstracted, so that the producer node in the server cluster stores the batch task messages in the message queue, and the consumer nodes in the server cluster consume the batch task messages stored in the message queue in parallel.

[0082] In this step, the producer node in the server cluster acts as a production node, and in response to the execution instruction of the task chain, automatically obtains the task identifier of the first data task and the batch task data corresponding to the first data task, and initializes the batch task data corresponding to the first data task. For example, if the first data task is the first data task in the task chain, the batch task data corresponding to the first data task is initialized to the pre-configured batch task data.

[0083] S404: Generate a batch task message of the first data task according to the task identifier and batch task data of the first data task through the producer node of the server cluster, and send the batch task message to the message queue.

[0084] In this step, if it is determined that the first data task meets the execution conditions based on the execution order of the first data task, the producer node in the server cluster encapsulates the task identifier of the first data task and the batch task data corresponding to the first data task, generates a batch task message corresponding to the first data task, and sends the batch task message to the message queue.

[0085] Further, optionally, generating a batch task message of the first data task based on the task identifier and batch task data of the first data task may include: obtaining a preset number of target batch task data from the batch task data in batches based on the task identifier of the first data task; and generating a batch task message of the first data task based on the target batch task data and the task identifier of the first data task.

[0086] For example, assuming that there are 100 batch task data, and the preset number is, for example, 10, then 10 target batch task data can be taken from the 100 batch task data at a time according to the task identifier of the first data task, that is, a paging query method is adopted, and one page contains 10 target batch task data, so that a batch task message of the first data task is generated according to the 10 target batch task data and the task identifier of the first data task. A total of 10 batch task messages can be generated from the 100 batch task data, and the 10 batch task messages are stored in the message queue.

[0087] S405 , consuming batch task messages in the message queue in parallel through the consumer nodes of the server cluster.

[0088] In this step, any of the multiple nodes included in the server cluster may be a consumer node. The consumer node acts as a consumer of the message queue. Multiple consumer nodes consume batch task messages in the message queue and process the batch task data corresponding to the data task in parallel according to the task identifier of the data task in the consumed batch task messages.

[0089] Exemplarily, the consumer node in the server cluster can route to a specific consumer (i.e., a specific business processing module corresponding to the task identifier) ​​to process the batch task message according to the task identifier of the first data task in the consumed batch task message, thereby enabling multiple consumer nodes to process the batch task data corresponding to each batch task message in the message queue in parallel. The consumer node can also determine whether each batch task message has been processed based on the task identifier of the first data task. For example, Table 2 can be queried based on the task identifier of the first data task to obtain the processing status (status) of all batch task data corresponding to the task identifier. Based on the processing status, it can be determined whether each batch task data has been processed. If it is determined that each batch task data has been processed, it is determined that the first data task has been executed; if there is batch task data that has not been processed, the next batch task message in the message queue continues to be processed.

[0090] S406: In response to the first data task not being the last data task of the task chain, executing the second data task through the consumer node of the server cluster according to the execution order of the data tasks.

[0091] For example, based on Figure 3 The relevant description of S303 in the illustrated embodiment can be that according to the task identifier of the first data task, query table 2 to obtain the task chain number of the task chain corresponding to the task identifier; then query table 1 according to the task chain number to obtain the task code set (tasks) corresponding to the task chain number; according to the task code set, it can be determined whether the first data task is the last data task of the task chain. If it is determined that the first data task is not the last data task of the task chain, the second data task is executed according to the execution order of the data tasks through the consumer node in the server cluster, that is, the consumer node acts as a new producer node and executes step S403. If it is determined that the first data task is the last data task of the task chain, the data task execution of the task chain ends.

[0092] Combined with steps S403 to S406, it can be seen that for multiple data tasks with dependencies included in the task chain, the producer node in the server cluster automatically obtains the first data task in the task chain in response to the execution instruction of the task chain, and initializes the data of the first data task. This data task can be understood as the first data task. Since it is the first data task and does not depend on other data tasks, after obtaining the data task, the producer node in the server cluster generates a batch task message corresponding to the data task according to the task identifier of the data task and the batch task data corresponding to the data task, and stores the batch task message in the message queue. The consumer node in the server cluster processes the batch task messages in the message queue in parallel. When it is determined that the first data task is completed, the consumer node in the server cluster obtains the next data task (i.e., the second data task). When it is determined that the next data task meets the execution condition, the next data task is used as the new first data task, and the consumer node in the server cluster is used as the new producer node to execute step S403. Among them, the execution condition includes, for example, that the next data task does not depend on other data tasks or that all other data tasks that it depends on are completed. It can be understood that for each data task included in the task chain, steps S403 to S406 are executed until each data task in the task chain is completed.

[0093] The data processing method provided in the embodiment of the present application, by responding to the configuration operation of the data task corresponding to the business demand, obtains the execution order of the data task through the producer node in the server cluster, and constructs the task chain through the producer node in the server cluster according to the task identifier of the data task and the execution order of the data task, so as to realize flexible and convenient task scheduling of the data task, instead of predefining the task scheduling of the data task in a hard-coded manner. In response to the execution instruction of the task chain, the task identifier and batch task data of the first data task are obtained through the producer node of the server cluster, and the batch task message of the first data task is generated through the producer node of the server cluster according to the task identifier and batch task data of the first data task, and the batch task message is sent to the message queue; the batch task message of the message queue is consumed in parallel by the consumer node of the server cluster, that is, the whole cluster parallel processing of the batch task data corresponding to the data task of the task chain is realized, instead of processing the data task by a single node and a single process, on the basis of ensuring the accurate execution of the data task of the task chain, the processing efficiency of the data task can be effectively improved, and it has the distributed characteristics of high performance, high scalability and high availability. In response to the fact that the first data task is not the last data task in the task chain, the second data task is executed through the consumer node of the server cluster according to the execution order of the data tasks. That is, the data tasks are managed through the task chain, and the execution of the data tasks of the task chain is automatically triggered according to the execution order of the data tasks in the task chain until all the data tasks in the task chain are completed. The data tasks of the task chain can be accurately and automatically executed without user intervention.

[0094] On the basis of the above embodiments, optionally, after executing the first data task of the task chain, the data processing method provided in the embodiments of the present application may also include: recording consumption information of the consumer node consuming batch task data, the consumption information including the consumption status of the batch task data; and / or, recording the execution information of the first data task, the execution information including at least one of the task start execution time, task completion time, task execution time and task execution status of the first data task.

[0095] This embodiment can be understood as performing task monitoring during the processing of batch task messages. The task monitoring refers to automatically recording the processing information of the first data task on the batch task data when each data task in the task chain is executed, so as to facilitate the troubleshooting of problems and the performance analysis of data tasks. Exemplarily, Table 3 is a task monitoring summary table for monitoring data tasks provided by an embodiment of the present application, as shown in Table 3, showing the various fields included in the task monitoring summary table. Table 4 is a task monitoring details table for monitoring batch task data provided by an embodiment of the present application, as shown in Table 4, showing the various fields included in the task monitoring details table, wherein the consumption status of batch task data includes, for example, initialization, successful processing, and failed processing.

[0096] Table 3

[0097]

[0098] Table 4

[0099]

[0100] Based on the above embodiments, Figure 5 The following is a schematic diagram of the application flow of the data processing method provided in one embodiment of the present application. Figure 5 As shown, the application process of the embodiment of the present application may include:

[0101] S501, the user configures a task chain, the task chain includes multiple data tasks with dependencies; the user selects batch task data and executes the task chain. Correspondingly, the server cluster stores the task chain and batch task data.

[0102] For example, it is assumed that a user wants to complete a certain business, and the business needs to execute four data tasks. Figure 6 A schematic diagram of a task chain provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the user creates a task chain and configures the task chain; the task chain contains 4 data tasks, and the dependencies of the 4 data tasks are as follows Figure 6 As shown, in the task chain list, data task 1, data task 2, data task 3 and data task 4 will be recorded.

[0103] S502. The server cluster responds to the execution instruction of the task chain, obtains the first data task, i.e., data task 1, from the task chain, starts data task 1, queries the database for batch task data corresponding to data task 1, and encapsulates the batch task data into a batch task message and sends it to the message queue.

[0104] S503: The server cluster consumes the batch task message corresponding to the data task 1 in the message queue, and processes the batch task message.

[0105] S504, the server cluster determines whether all batch task messages corresponding to data task 1 have been executed. If not, it continues to consume and process the batch task messages corresponding to data task 1. If all batch task messages corresponding to data task 1 have been executed, it continues to execute step S505.

[0106] S505. The server cluster checks whether data task 1 is the last data task in the task chain. If it is determined that data task 1 is not the last data task in the task chain, the subsequent data tasks of data task 1 are triggered to continue to execute. At this time, data task 2 and data task 3 are triggered, that is, steps S506 to S509 are executed, and steps S510 to S513 are executed.

[0107] It should be noted that the embodiment of the present application does not limit the order in which steps S506 to S509 and steps S510 to S513 are executed.

[0108] S506 , the server cluster starts data task 2 , encapsulates the batch task data corresponding to data task 2 into a batch task message and sends it to the message queue.

[0109] S507: The server cluster consumes the batch task message of data task 2 and processes the batch task message.

[0110] S508. The server cluster determines whether all batch task messages of data task 2 have been executed. If not, the server cluster continues to consume batch task messages. If the batch task messages have been executed, the server cluster continues to execute step S509.

[0111] S509, the server cluster checks whether data task 2 is the last data task in the task chain. At this time, it can be determined that data task 2 is not the last data task in the task chain, and data task 4 is triggered. However, data task 4 depends on data task 2 and data task 3, and data task 3 has not been completed at this time. Therefore, data task 4 cannot be started.

[0112] S510 , the server cluster starts data task 3 , encapsulates the batch task data corresponding to data task 3 into a batch task message and sends it to the message queue.

[0113] S511. The server cluster consumes the batch task message of data task 3 and processes the batch task message.

[0114] S512. The server cluster determines whether all batch task messages of data task 3 have been executed. If not, the server cluster continues to consume batch task messages. If the batch task messages have been executed, the server cluster executes step S513.

[0115] S513, the server cluster checks whether data task 3 is the last data task in the task chain. At this time, it can be determined that data task 3 is not the last data task in the task chain, and data task 4 is triggered. However, data task 4 depends on data task 2 and data task 3, and at this time, data task 2 and data task 3 have been executed. Therefore, data task 4 needs to be started, that is, execute step S514.

[0116] S514. The server cluster starts data task 4, encapsulates the batch task data corresponding to data task 4 into a batch task message and sends it to the message queue.

[0117] S515. The server cluster consumes the batch task message of data task 4 and processes the batch task message.

[0118] S516. The server cluster determines whether all batch task messages of data task 4 have been executed. If not, the server cluster continues to consume batch task messages. If the batch task messages have been executed, the server cluster continues to execute step S517.

[0119] S517, the server cluster checks whether data task 4 is the last data task in the task chain. At this time, it can be determined that data task 4 is the last data task in the task chain, so the task chain execution ends.

[0120] Understandably, Figure 5 During the execution of the data tasks in the task chain shown, the embodiment of the present application will record task monitoring data during the execution of the data tasks. Specifically, for example, when did data task 1, data task 2, data task 3, and data task 4 start execution, when did they finish execution, and how much time did the execution process take. Detailed records are made for all batch task data corresponding to data task 1, such as when the first batch task data in data task 1 starts processing, when the processing is completed, how long it takes to process, and whether the processing status is normal. By automatically recording the execution status data related to the data tasks, an effective basis is provided for later troubleshooting and checking the execution performance of the data tasks.

[0121] The data processing method provided in the embodiment of the present application allows users to independently select and then arrange task chains based on actual business needs and currently existing data tasks, so that the arrangement of task chains can be more convenient and flexible. After the data task is started, the server cluster will automatically execute according to the configured task chain without user intervention; based on the message queue, any node in the server cluster can be used as a producer node to store batch task messages in the message queue, and any node in the server cluster can be used as a consumer node to consume batch task messages stored in the message queue, so as to realize the whole cluster and perform processing, which can effectively improve the processing efficiency of data tasks and has the distributed characteristics of high performance, high scalability and high availability.

[0122] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0123] Figure 7This is a schematic diagram of the structure of a data processing device provided in an embodiment of the present application, which is applied to a server cluster. Figure 7 As shown, the data processing device 700 of the embodiment of the present application includes: a construction module 701, an execution module 702 and a processing module 703. Among them:

[0124] The construction module 701 is used to construct a task chain according to the task identifier of the data task and the execution order of the data task.

[0125] The execution module 702 is used to execute the first data task of the task chain in response to the execution instruction of the task chain.

[0126] The processing module 703 is configured to execute the second data task according to the execution order of the data tasks in response to the first data task not being the last data task of the task chain.

[0127] Optionally, the execution module 702 can also be used to: before executing the first data task of the task chain, determine whether the first data task meets the execution conditions based on the execution order of the data tasks in the task chain, and then execute the first data task, the execution conditions including that the first data task does not depend on other data tasks or that all other data tasks on which it depends have been executed.

[0128] Optionally, the construction module 701 may also be used to obtain the execution order of the data tasks in response to the configuration operation of the data tasks corresponding to the business requirements before constructing the task chain according to the task identifier of the data task and the execution order of the data tasks.

[0129] In some embodiments, the execution module 702 can be specifically used to: obtain the task identifier and batch task data of the first data task through the producer node of the server cluster; generate a batch task message of the first data task according to the task identifier and batch task data of the first data task through the producer node of the server cluster, and send the batch task message to the message queue; and consume the batch task messages of the message queue in parallel through the consumer node of the server cluster.

[0130] Optionally, when the execution module 702 is used to generate a batch task message of the first data task based on the task identifier and batch task data of the first data task, it is specifically used to: obtain a preset number of target batch task data from the batch task data in batches according to the task identifier of the first data task; and generate a batch task message of the first data task according to the target batch task data and the task identifier of the first data task.

[0131] Optionally, the execution module 702 is also used to: after executing the first data task of the task chain, record the consumption information of the consumer node consuming the batch task data, the consumption information including the consumption status of the batch task data; and / or, record the execution information of the first data task, the execution information including at least one of the task start execution time, task completion time, task execution time and task execution status of the first data task.

[0132] Optionally, the execution module 702 is also used to: before executing the first data task of the task chain, determine whether the first data task is the first data task in the task chain; if the first data task is the first data task in the task chain, determine the pre-configured batch task data as the batch task data of the first data task; if the first data task is not the first data task in the task chain, determine the previous third data task of the first data task according to the execution order of the data tasks, and determine the batch task data completed by the third data task as the batch task data of the first data task.

[0133] Optionally, the execution module 702 is further configured to: in response to the first data task being completed, determine that the first data task is not the last data task of the task chain.

[0134] The device of the embodiment of the present application can be used to execute the technical solution of any of the above-mentioned method embodiments, and its implementation principle and technical effect are similar, which will not be repeated here.

[0135] Figure 8 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 8 As shown, the electronic device 800 may include: at least one processor 801 and a memory 802 .

[0136] The memory 802 is used to store programs. Specifically, the programs may include program codes, and the program codes include computer-executable instructions.

[0137] The memory 802 may include a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0138] The processor 801 is used to execute the computer-executable instructions stored in the memory 802 to implement the data processing method described in the aforementioned method embodiment. Among them, the processor 801 may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application. Specifically, when implementing the data processing method described in the aforementioned method embodiment, the electronic device may be, for example, an electronic device with processing functions such as a server cluster.

[0139] Optionally, the electronic device 800 may further include a communication interface 803. In a specific implementation, if the communication interface 803, the memory 802 and the processor 801 are implemented independently, the communication interface 803, the memory 802 and the processor 801 may be interconnected through a bus and communicate with each other. The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc., but it does not mean that there is only one bus or one type of bus.

[0140] Optionally, in a specific implementation, if the communication interface 803, the memory 802 and the processor 801 are integrated on a chip, the communication interface 803, the memory 802 and the processor 801 can communicate through an internal interface.

[0141] The present application also provides a computer-readable storage medium, in which computer program instructions are stored. When a processor executes the computer program instructions, the above data processing method is implemented.

[0142] The present application also provides a computer program product, including a computer program, which implements the above data processing method when executed by a processor.

[0143] The computer-readable storage medium mentioned above can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0144] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in a dedicated integrated circuit. Of course, the processor and the readable storage medium can also exist as discrete components in a data processing device.

[0145] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A data processing method, characterized in that: include: Constructing a task chain according to the task identifiers of the data tasks and the execution order of the data tasks; In response to an execution instruction for the task chain, executing a first data task of the task chain; In response to the first data task not being the last data task of the task chain, the second data task is executed according to the execution order of the data tasks.

2. The data processing method according to claim 1, characterized in that: Before executing the first data task of the task chain, the method further includes: If it is determined according to the execution order of the data tasks in the task chain that the first data task meets the execution condition, the first data task is executed. The execution condition includes that the first data task is not dependent on other data tasks or that all other dependent data tasks have been completed.

3. The data processing method according to claim 1, characterized in that: Before constructing the task chain according to the task identifier of the data task and the execution order of the data task, the method further includes: In response to a configuration operation on a data task corresponding to a business requirement, an execution order of the data task is obtained.

4. The data processing method according to any one of claims 1 to 3, characterized in that: The executing the first data task of the task chain includes: Obtaining the task identifier and batch task data of the first data task through the producer node of the server cluster; Generate a batch task message of the first data task according to the task identifier and batch task data of the first data task through the producer node of the server cluster, and send the batch task message to the message queue; The batch task messages of the message queue are consumed in parallel by the consumer nodes of the server cluster.

5. The data processing method according to claim 4, characterized in that: The producer node of the server cluster generates a batch task message of the first data task according to the task identifier and batch task data of the first data task, including: According to the task identifier of the first data task, acquiring a preset number of target batch task data from the batch task data in batches; A batch task message of the first data task is generated according to the target batch task data and the task identifier of the first data task.

6. The data processing method according to claim 4, characterized in that: After executing the first data task of the task chain, the method further includes: Recording consumption information of the batch task data consumed by the consumer node, wherein the consumption information includes a consumption status of the batch task data; And / or, record execution information of the first data task, the execution information including at least one of the task execution start time, task completion time, task execution time and task execution status of the first data task.

7. The data processing method according to any one of claims 1 to 3, characterized in that: Before executing the first data task of the task chain, the method further includes: Determining whether the first data task is the first data task in the task chain; If the first data task is the first data task of the task chain, determining the pre-configured batch task data as the batch task data of the first data task; If the first data task is not the first data task of the task chain, the previous third data task of the first data task is determined according to the execution order of the data tasks, and the batch task data completed by the third data task is determined as the batch task data of the first data task.

8. The data processing method according to any one of claims 1 to 3, characterized in that: In response to the first data task not being the last data task of the task chain, the method further comprises: In response to the first data task being completed, it is determined that the first data task is not the last data task of the task chain.

9. A data processing device, characterized in that: include: A construction module, used to construct a task chain according to the task identifier of the data task and the execution order of the data task; an execution module, configured to execute the first data task of the task chain in response to an execution instruction of the task chain; The processing module is configured to execute a second data task according to an execution order of the data tasks in response to the first data task not being the last data task of the task chain.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the data processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed, the data processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed, the data processing method according to any one of claims 1 to 8 is implemented.

Images