Task processing method, task processing apparatus, and electronic device

Abstract

The application provides a task processing method, device and electronic equipment. The task processing method comprises the following steps: receiving a batch task sent by a task sender, splitting the batch task into a plurality of subtasks, and encapsulating each subtask in the plurality of subtasks into a subtask message, wherein the subtask message comprises a consumer name of each subtask; storing the encapsulated subtask message into a request task queue; and monitoring a request task list in real time, and if there is new data in the request task queue, selecting a suitable task consumer for task processing according to the consumer name contained in the subtask message. By adopting the embodiment of the application, the task processing of the task consumer can be standardized, and the tracking and query of task execution can be facilitated.

Description

Technical Field

[0001] This application relates to the field of batch data distributed computing technology, and in particular to a task processing method, task processing device and electronic device. Background Technology

[0002] For most application systems today, to ensure high availability, they are deployed in clusters, providing a load-balanced address similar to Nginx for users to access.

[0003] Many application systems involve batch data operations. For example, a scheduled task at 1 AM might retrieve outstanding payment contracts from the database yesterday, iterate through the contracts, and call other company systems (such as the accounting system) based on the contract number to update the status of the outstanding payment contracts (even if the outstanding payments have already been settled). Another example is when a user imports a batch of outstanding payment contracts from Excel, iterates through the contracts, and calls other company systems (such as the customer service system) based on the contract number to retrieve the corresponding customer information.

[0004] In scenarios involving batch data operations, the current common practice is to have a single node or device within the application system execute the operation. However, a single node has limited network bandwidth, computing power, and memory. If the processing is time-consuming, it severely impacts the performance of batch task processing. Therefore, distributed task processing has emerged, and how to effectively perform distributed tasks has become a hot research topic in various application systems. Summary of the Invention

[0005] This application provides a task processing method, a task processing device, and an electronic device.

[0006] Firstly, this application provides a task processing method applied to a distributed computing system. The distributed computing system performs distributed computing when invoked by any node in a big data cluster. The distributed computing system includes a producer management device and a consumer management device. The consumer management device manages the task execution of task consumers. The task processing method is executed by the producer management device and includes: when the distributed computing system is invoked by any node in the big data cluster to perform distributed computing, receiving at least one batch task sent by a task sender and splitting the at least one batch task into multiple subtasks; encapsulating the multiple subtasks into multiple subtask messages and storing them sequentially in a request task queue; wherein each subtask message includes a consumer name field; the request task queue is monitored in real time by the consumer management device. When the consumer management device detects a new subtask message in the request task queue, it routes the new subtask message to the corresponding task consumer based on the consumer name field in the new subtask message, so that the corresponding task consumer processes the new subtask to obtain a processing result.

[0007] Secondly, this application also provides a task processing method applied to a distributed computing system. When the distributed computing system is invoked by any node in a big data cluster, it performs distributed computing. The distributed computing system includes a producer management device and a consumer management device. The producer management device is used to split received batch tasks into multiple sub-task messages, each including a consumer name field, and store them in a request task queue. The task processing method is executed by the consumer management device. The task processing method includes: when the distributed computing system is invoked by any node in the big data cluster to perform distributed computing, monitoring the request task queue in real time; the request task... The task queue is used to store multiple subtask messages when they appear. These subtask messages are obtained by encapsulating multiple subtasks separately. The multiple subtasks are obtained by the producer management device from splitting at least one batch task. The at least one batch task is received from the task sender when the distributed computing system is called by any node in the big data cluster to perform distributed computing. When a new subtask message is detected in the request task queue, the task consumer corresponding to the consumer name field in the new subtask message is selected to process the subtask and obtain the processing result.

[0008] Thirdly, this application also provides a task processing device applied to a distributed computing system. When the distributed computing system is invoked by any node in a big data cluster to perform distributed computing, the distributed computing system includes a producer management device and a consumer management device. The consumer management device is used to manage the task execution of task consumers. The task processing device includes: a receiving unit, used to receive at least one batch task sent by a task sender when the distributed computing system is invoked by any node in the big data cluster to perform distributed computing; a splitting unit, used to split the at least one batch task into multiple subtasks; and a storage unit, used to encapsulate the multiple subtasks into multiple subtask messages and store them sequentially in a request task queue. Each subtask message includes a consumer name field. The request task queue is monitored in real time by the consumer management device. When the consumer management device detects a new subtask message in the request task queue, it routes the new subtask message to the corresponding task consumer based on the consumer name field in the new subtask message, so that the corresponding task consumer processes the new subtask to obtain a processing result.

[0009] Fourthly, this application also provides a task processing device applied to a distributed computing system. When the distributed computing system is invoked by any node in a big data cluster, it performs distributed computing. The distributed computing system includes a producer management device and a consumer management device. The producer management device is used to split received batch tasks into multiple sub-task messages, each including a consumer name field, and store them in a request task queue. The task processing device includes a monitoring unit, used to monitor the request task queue in real time when the distributed computing system is invoked by any node in the big data cluster to perform distributed computing. The request task queue is used to handle multiple sub-tasks when they occur. When a task message is sent, multiple subtask messages are stored. These multiple subtask messages are obtained by encapsulating multiple subtasks separately. These multiple subtasks are obtained by the producer management device from splitting at least one batch task. The at least one batch task is received from the task sender when the distributed computing system is called by any node in the big data cluster to perform distributed computing. The selection unit is used to select the task consumer corresponding to the consumer name field to process the subtask and obtain the processing result when a new subtask message is detected in the request task queue, based on the consumer name field contained in the new subtask message.

[0010] Fifthly, this application also provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores one or more computer programs executable by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the above-described task processing method.

[0011] Sixthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the above-described task processing method when executed by a processor / processor core.

[0012] In the embodiments provided in this application, for distributed computing of batch data, a centralized management approach is adopted in an application system including one or more nodes, such as a big data cluster, to perform distributed computing of batch data. A distributed computing system (also known as a distributed computing engine) is set up for the entire application system, including at least a global producer management device (also known as a producer processor) and a global consumer management device (also known as a consumer processor). The producer management device splits the batch tasks into encapsulated task messages with specific data structures. The encapsulated task messages include the consumer name field of multiple subtasks into which the batch data is split and are stored in the request task queue. This is a global splitting process to achieve global distributed scheduling of the entire application system. At the same time, a global consumer management device is set up for the entire application system to monitor whether there is new data in the request task queue in real time. When new data is found, the corresponding consumer is called to execute the task according to the relevant consumer name field stored in the encapsulated subtask message.

[0013] This invention provides a global task processing method that implements a globally distributed task processing approach, thereby standardizing the processing logic of each consumer and better tracking message consumption. Furthermore, this globally distributed processing approach improves application system operating efficiency, saves application system operating costs, facilitates standardized task processing by task consumers, and enhances task execution tracking and querying.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the present application and form part of the specification. They are used together with the embodiments of the present application to explain the application and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed example embodiments described with reference to the accompanying drawings, in which:

[0016] Figure 1 A schematic diagram of a distributed task execution process provided for related technologies;

[0017] Figure 2 A flowchart illustrating a task processing method provided in an embodiment of this application;

[0018] Figure 3 A flowchart illustrating another task processing method provided in an embodiment of this application;

[0019] Figure 4 This is a schematic diagram of a task processing system provided in an embodiment of this application;

[0020] Figure 5 This is a schematic diagram of another task processing system provided in an embodiment of this application;

[0021] Figure 6 A system schematic diagram of yet another task processing system provided in the embodiments of this application;

[0022] Figure 7 A system schematic diagram of another task processing system provided in the embodiments of this application;

[0023] Figure 8 A flowchart illustrating yet another task processing method provided in an embodiment of this application;

[0024] Figure 9 This is a schematic diagram of the structure of a task processing device provided in an embodiment of this application;

[0025] Figure 10 This is a schematic diagram of another task processing device provided in an embodiment of this application;

[0026] Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions of this application, exemplary embodiments of this application are described below in conjunction with the accompanying drawings, including various details of the embodiments of this application to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] Where there is no conflict, the various embodiments of this application and the features thereof may be combined with each other.

[0029] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, they specify the presence of features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Words such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect.

[0031] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this application, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0032] For ease of understanding, the following explanations are provided for some of the terms used in this invention:

[0033] Distributed computing: Distributed computing is a research direction in computer science that studies how to divide a problem that requires enormous computing power into many smaller parts, distribute these parts to multiple computers or processors for processing, and finally combine the results of these calculations to obtain the final result.

[0034] Batch data: One or more datasets with the same data structure are grouped together to form a batch of data.

[0035] A message queue is a container that holds messages during transmission. A message queue manager acts as an intermediary, relaying messages from their source to their destination. The primary purpose of a queue is to provide routing and guarantee message delivery; if the receiver is unavailable when a message is sent, the message queue retains the message until it can be successfully delivered.

[0036] As mentioned earlier, most batch data operations are currently executed by a single node in the application system. However, a single node has limited network bandwidth, computing power, and memory. If the processing is time-consuming, it severely impacts the performance of batch tasks. Assuming that processing one piece of data in a batch takes 500 milliseconds, and the batch size is 100,000 data points, then in a single-threaded scenario on the same node, the total processing time would be: 100,000 * 0.5 seconds = 50,000 seconds = 833.4 minutes. Even using multi-threaded processing on a single node, assuming 10 concurrent threads: 100,000 * 0.5 seconds / 10 = 5,000 seconds = 84 minutes. If all nodes in the cluster participate, assuming a cluster of 2 nodes, then 100,000 * 0.5 seconds / 10 / 2 = 2.5,000 seconds = 41 minutes. Clearly, the more nodes in the cluster, the lower the processing time.

[0037] To address the aforementioned issues, distributed task processing has become a hot research topic in various application systems. (See also...) Figure 1 This is a flowchart illustrating a distributed task processing approach in an application system, proposed by related technologies. For example, a user imports a batch of overdue payment contracts into an Excel spreadsheet. The system iterates through these contracts, retrieving the corresponding customer information from other company systems (such as the customer service system) based on the contract number. Currently, message queues are typically used to implement distributed computing. This involves splitting the batch data into individual data points, sending them to the message queue, and then all nodes in the cluster, including the task initiating node, acting as consumers, retrieving the data from the message queue, performing business logic processing based on the data, and saving the results to the database.

[0038] Specifically, such as Figure 1 As shown, the application system cluster includes multiple nodes ( Figure 1 The example only shows node 1 and node 2) and load balancer (such as Nginx). The application system also includes message queues.

[0039] Figure 1 The application system shown executes processes in the order illustrated.

[0040] 1. Two nodes (Node 1 and Node 2) in the application system actively subscribe to a specific topic in the message queue after startup. In existing technologies, when the application system starts up, each node in the application system subscribes to a specific topic in the message queue according to its configuration. This active subscription leads to the inability to effectively abstract and standardize the consumer's message consumption.

[0041] 2. The user triggers a batch operation request and sends it to the load balancer.

[0042] 3. A load balancer, such as Nginx, routes batch operation requests to node 1.

[0043] 4. Node 1 executes a batch operation request and splits the batch data into individual data items, sending them to a specific topic in the message queue.

[0044] 5. Since Node 1 and Node 2 have subscribed to a certain topic of the message queue, they become consumers of the corresponding topic. Therefore, when there is data in a certain topic, Node 1 and Node 2 will consume messages from the corresponding topic.

[0045] 6. Node 1 and Node 2 process the business logic based on the messages they consume.

[0046] Figure 1 The prior art shown has the following disadvantages:

[0047] a. If the application system has many similar batch operations, multiple topics need to be created for the application system, one topic per batch operation. Creating topics may require approval processes and will take time. Later, during version iterations, if new batch operations need to be added, the process of applying for topics will need to be repeated, resulting in duplicated work.

[0048] b. The logic processing of consumers in each topic cannot be effectively abstracted and standardized. Issues such as consumer concurrency settings, whether consumers consume individually or in batches, whether consumers report successful consumption to the message queue, and when they report successful consumption to the message queue are not addressed.

[0049] To address the aforementioned issues, this application provides a task processing method for batch data processing. In an application system comprising one or more nodes, a centralized global management approach is employed for distributed batch data computation. This method establishes producer management and consumer management devices for the entire application system. The producer management device breaks down batch tasks (batch data to be processed) into individual messages as subtasks. Each subtask is then encapsulated into a subtask message with a specific data structure. These encapsulated subtask messages are stored in a request task queue, which serves as a message queue. Each encapsulated subtask message includes at least a consumer name field indicating the consumer that processes the task within the subtask message. This provides a global batch data processing method, i.e., a globally distributed scheduling method for the entire application system. Simultaneously, a global consumer management device is established for the entire application system to monitor the request task queue in real time for new data. Upon detecting new data, the corresponding task consumer is invoked to execute the task based on the consumer name field stored in the encapsulated subtask message. This invention's global batch data processing method includes a global data splitting method. During data splitting, the producer management device specifies the task consumers corresponding to each subtask, thereby standardizing the processing logic of each subtask and better tracking message consumption. Furthermore, this globally distributed task processing method of the invention can improve the operating efficiency of application systems, save application system operating costs, facilitate the standardization of task consumer task processing, and facilitate task execution tracking and querying.

[0050] Please see Figure 2 This is a flowchart illustrating a task processing method provided in an embodiment of this application. Figure 2 The task processing method shown is applied to a distributed computing system, which performs distributed computing when invoked by any node in a big data cluster. The distributed system includes producer management devices and consumer management devices, with the consumer management devices used to manage the execution of task consumers. Figure 2 The task processing method shown is executed by a producer management device, which may include a terminal or a server. The terminal may include devices such as laptops, tablets, and smart interactive devices, while the server may include an independent physical server, a server cluster consisting of one or more servers, or a cloud server capable of cloud computing. Figure 2 The task processing method shown includes the following steps:

[0051] S301. When the distributed computing system is called by any node in the big data cluster to perform distributed computing, it receives at least one batch task sent by the task sender and splits the at least one batch task into multiple subtasks.

[0052] In one embodiment, the code logic for breaking down a batch task into multiple subtasks is provided by the business developers, while the producer management device only includes an interface for them to implement. For example, importing 100,000 overdue payment contracts via Excel constitutes a batch task. If each contract is broken down into a subtask, the batch task becomes 100,000 subtasks. These subtasks correspond to the same topic, and after sending each subtask to the message queue topic, the message queue topic contains 100,000 messages.

[0053] S302. Encapsulate multiple subtasks into multiple subtask messages and store them sequentially in the request task queue.

[0054] Optionally, the request task queue can reside in the memory of the target node, which refers to the node in the big data cluster that calls the distributed computing system; alternatively, the request task queue can also be deployed independently of the message queue middleware deployed on the target node. This application embodiment does not specifically limit the exact storage location of the request task queue and can set it according to actual needs.

[0055] In one embodiment, each subtask message may include multiple fields, a description, type, and remarks for each field. For example, the data structure of a subtask message is shown in Table 1 below:

[0056] Table 1

[0057]

[0058]

[0059] As shown in Table 1, a subtask message may include a batchNo field, which indicates the batch number of the task batch to which the subtask message belongs. A subtask message may also include a data field, which indicates the business data carried in the subtask message. A subtask message may also include a consumerName field, which indicates the consumer name of the subtask message.

[0060] It's important to note that existing batch tasks typically don't encapsulate data; at most, they convert the split data into JSON format and send the converted JSON to the request task queue. In a distributed application system, each node subscribes to different topics in the message queue based on its configuration. Therefore, multiple topics need to be created for the distributed application system, and the logic processing of consumers for each topic cannot be effectively abstracted and standardized. However, in this application, each subtask message is encapsulated and stored in the request task queue. Because each subtask message records several key fields, such as consumer name, producer name, and post-processor name, the subtask messages stored in the request task queue include a global data structure. For the "consumer name of batch data" field mentioned above, only one topic is needed. The messages in this topic can be of many different types of data, but these types of data will be encapsulated and abstracted. The subtask data is encapsulated and abstracted in the message producer management device, and the encapsulated and abstracted data becomes a message. That is, the required parameters mentioned in this table are initialized in the producer management device. The consumer name for batch data is set according to the configuration (or annotation). One producer corresponds to one task consumer and one batch data post-processor.

[0061] In this embodiment of the application, after multiple subtask messages are stored in the request task queue, the request task queue is monitored in real time by the consumer management device. When the consumer management device detects a new subtask message in the request task queue, it routes the new subtask message to the corresponding task consumer according to the consumer name field in the new subtask message, so that the corresponding task consumer can process the new subtask to obtain the processing result.

[0062] In one embodiment, the distributed computing system may further include a result log table for storing multiple subtask messages and the processing result of each subtask message. Furthermore, the distributed computing system may also include a result log table visualization and query module, which can be provided to users to facilitate querying log information related to the batch task stored in the result log table.

[0063] In the embodiments provided in this application, for distributed computing of batch data, a centralized management approach is adopted in an application system including one or more nodes, such as a big data cluster, to perform distributed computing of batch data. A distributed computing system (also known as a distributed computing engine) is set up for the entire application system, including at least a global producer management device (also known as a producer processor) and a global consumer management device (also known as a consumer processor). The producer management device splits the batch tasks into encapsulated task messages with specific data structures. The encapsulated task messages include the consumer name field of multiple subtasks into which the batch data is split and are stored in the request task queue. This is a global splitting process to achieve global distributed scheduling of the entire application system. At the same time, a global consumer management device is set up for the entire application system to monitor whether there is new data in the request task queue in real time. When new data is found, the corresponding consumer is called to execute the task according to the relevant consumer name field stored in the encapsulated subtask message.

[0064] The global data splitting method and corresponding routing method of this invention realize a global distributed task processing approach, thereby standardizing the processing logic of each consumer and better tracking message consumption. Furthermore, this global distributed processing approach of this invention can improve the operating efficiency of application systems, save application system operating costs, facilitate the standardization of task processing by task consumers, and facilitate the tracking and querying of task execution.

[0065] based on Figure 2 In addition to the task processing method described above, this application also provides another task processing method. See [link to application]. Figure 3 This application provides another task processing method, which can be executed by a consumer management device in a distributed computing system. The consumer management device may include a terminal or a server. The terminal may refer to a laptop, tablet, smart interactive device, etc., and the server may include a standalone physical server, a server cluster consisting of multiple servers, or a cloud server capable of cloud computing. Figure 3 The task processing method shown may include the following steps:

[0066] S501. When a distributed computing system is invoked by any node in a big data cluster to perform distributed computing, the request task queue is monitored in real time.

[0067] As mentioned above, the request task queue is used to store subtask messages. Multiple subtask messages are obtained by encapsulating multiple subtasks separately. The multiple subtasks are obtained by the producer management device splitting at least one batch task. The at least one batch task is received from the task sender when the distributed computing system is called by any node in the big data cluster to perform distributed computing.

[0068] S502. When a new subtask message is detected in the request task queue, select the task consumer corresponding to the consumer name field in the new subtask message to process the subtask and obtain the processing result.

[0069] In one embodiment, the distributed computing system further includes a result record table, which stores multiple subtask messages and the processing result corresponding to each subtask message. Each subtask message also includes a post-processor field corresponding to the subtask, such as the lerName field representing the post-processor name field in Table 1. The post-processor field corresponding to each subtask is used to indicate the post-processor matched by the batch task to which the corresponding subtask belongs. Based on this, the consumer management device processor also monitors in real time whether all subtasks in the batch task to which the corresponding subtask belongs in the request task queue have been executed; and if they have been executed, it retrieves the processing result of the batch task from the result record table and selects a suitable post-processor for data processing based on the post-processor name field.

[0070] In one embodiment, the distributed storage system further includes a remote interface router, the big data cluster includes an application system load balancer, and when the producer management device detects a new subtask message in the request task queue, it selects the task consumer corresponding to the consumer name field in the new subtask message to process the subtask and obtain a processing result. Specific implementations may include:

[0071] When a new subtask message is detected in the request task queue, the obtained subtask message is sent to the application system load balancer through the remote interface router. The application load balancer selects a node from at least one node in the big data cluster to call the remote interface router in the distributed computing system to call the consumer management device, so as to select the corresponding task consumer through the remote interface router to process the subtask according to the consumer name field in the subtask message and obtain the processing result.

[0072] In the embodiments provided in this application, for distributed computing of batch data, a centralized management approach is adopted in an application system including one or more nodes, such as a big data cluster, to perform distributed computing of batch data. A distributed computing system (also known as a distributed computing engine) is set up for the entire application system, including at least a global producer management device (also known as a producer processor) and a global consumer management device (also known as a consumer processor). The producer management device splits the batch tasks into encapsulated task messages with specific data structures. The encapsulated task messages include the consumer name field of multiple subtasks into which the batch data is split and are stored in the request task queue. This is a global splitting process to achieve global distributed scheduling of the entire application system. At the same time, a global consumer management device is set up for the entire application system to monitor whether there is new data in the request task queue in real time. When new data is found, the corresponding consumer is called to execute the task according to the relevant consumer name field stored in the encapsulated subtask message.

[0073] The global data splitting method and corresponding routing method of this invention realize a global distributed task processing approach, thereby standardizing the processing logic of each consumer and better tracking message consumption. Furthermore, this global distributed processing approach of this invention can improve the operating efficiency of application systems, save application system operating costs, facilitate the standardization of task processing by task consumers, and facilitate the tracking and querying of task execution.

[0074] based on Figure 2 and Figure 3 The aforementioned task processing method, in addition to providing a task processing system in this application embodiment, also includes a task processing system, see [link to relevant documentation]. Figure 4 This is a system schematic diagram of a task processing system provided in an embodiment of this application. Figure 4 The task processing system includes a distributed computing system, which performs distributed computing when invoked by any node in the big data cluster. Specifically, the distributed computing system may include producer management devices and consumer management devices. Optionally, the distributed computing system may also include a post-processor, a remote interface router, and a result record table visualization and query module.

[0075] The specific responsibilities of the producer management device, consumer management device, post-processing router, remote interface router, and result record table visualization query module are as follows:

[0076] Producer Management Device: The producer management device is a global producer management device for the entire application system. It manages the entire task processing system and is responsible for receiving at least one batch of tasks sent by the task sender, such as... Figure 4In the phrase "1. Send batch tasks to the engine," the engine refers to the distributed computing system. The producer management device within the distributed computing system receives at least one batch task; breaks down the batch task into individual subtasks; encapsulates each subtask into a subtask message in a message queue; and submits the subtask message to the request task queue. Figure 4 In section "3. Submit Messages to the Request Task Queue," the messages refer to the various subtask messages. The initial result record table for the encapsulated subtask messages essentially records each subtask message in the result record table. Each subtask message carries fields such as consumer name, producer name, and post-processor name. Subsequent components such as consumer management devices and post-processor routers will route the subtask messages to the corresponding task consumers and post-processors for processing based on the information in these fields.

[0077] Consumer management device: This consumer management device is a global consumer management device for the entire application system. It is responsible for retrieving messages from the request task queue, for example... Figure 4 In section "4. Computation Engine Retrieving Messages," the computation engine refers to the distributed computing system, specifically the consumer management device within that system. Retrieving messages means obtaining multiple subtask messages from the request task queue. The consumer management device then parses these subtask messages, identifies the corresponding task consumer based on the consumer name field within the message, and further calls the appropriate consumer to process the message. Specifically, as follows... Figure 4 In section 5, "The consumer management device routes messages to the task consumers for message processing," where "messages" refers to subtask messages; and updates the processing results in the result record table; and calculates whether the batch tasks have been completed, such as... Figure 4 The text continues with a section on "7. Updating the consumer management device's processing results and calculating whether the batch task has been completed based on the result record table." If the batch task has been completed, the text then notifies the post-processing router, such as... Figure 4 "8. Notice" in the middle.

[0078] Post-processing router: Upon receiving notification from the consumer management device, it obtains the batch processing results, performs routing calculations based on these results, and selects an appropriate post-processor for data processing. For example, Figure 4 The responsibilities of the post-processing router shown in the data flow of the block diagram of the distributed task processing device are: 9. Obtain the batch execution results; 10. Invoke the post-processor to process the batch execution results.

[0079] Remote interface router: This module is optional, when using the following Figure 5 and Figure 6The module only takes effect when the illustrated embodiment uses a distributed computing mode based on load balancing of the application system (it should be noted that, unless otherwise specified, this embodiment uses a big data cluster as an example of the application system). The remote interface router receives the request from the application system load balancer, performs routing calculations, finds a suitable task consumer, executes the task, and transmits the execution result to the consumer management device. For example, Figure 5 The responsibilities of the remote interface router in the data flow of the block diagram of the distributed task processing device shown are as follows: 5.1 The application system load balancer forwards requests to a certain application system node N according to its own routing policy, and the node N calls the distributed computing system. 5.2 Calculate the route and route the task request to the correct task consumer. 6. Return the results and make calls from the remote interface router to the consumer management device.

[0080] The results log visualization and query module allows for easy retrieval of log information related to batch task processing. This includes information such as when and which messages were sent to the request task queue, as well as which nodes in the big data cluster consumed the messages at what time, and the consumer processing results. This facilitates rapid troubleshooting when problems arise. For example, Figure 4 The block diagram of the distributed task processing device shows a data flow from the result record table to the result record table visualization query and a data flow from the result record table visualization query to the user's query batch task log information.

[0081] In the distributed computing system described above in this application, a series of abstract interfaces are provided for extending the distributed computing system, as specifically described below:

[0082] The request task queue interface is used to communicate with the request task queue. The implementation can be based on a local task queue within the application system process, or on a separately deployed message queue middleware such as Kafka. For example, if the distributed computing system only provides Kafka as the message queue, but the application system project uses RocketMQ, you only need to write an implementation class for the request task queue interface. Within the implementation class, you can handle message storage and retrieval from the RocketMQ queue and configure it to the default mode.

[0083] Both RocketMP and Kafka use the term "topic" to represent a collection of data. For example, if you need to send order messages to a message queue (MQ), you can group these types of messages into a topic, which is a collection of data containing order messages. When using an MQ, the first step should always be to create some topics to store different types of messages as data collections. In essence, this is the same as creating table structures first when using a database.

[0084] In the distributed computing system of this application, we directly use MQ. The first step is to create a topic in MQ. The number of topics to create depends on the actual needs.

[0085] The result record table interface is used to communicate with the result record table. The specific implementation is generally a database, which can be a relational database or an in-memory database, such as MySQL or Redis. For example, if a distributed computing system provides a MySQL relational database for storing the result record table, but the application system prefers to use a Redis in-memory database to store the result record table content, then only an implementation class needs to be written for the result record table interface. This implementation class should handle the storage and querying of the Redis database and be configured as the default method. In this invention, the application system's database result record table is not specifically limited; an appropriate storage method can be selected based on actual needs (e.g., batch data size).

[0086] User code interface: Provides users with an implementation interface for batch task distributed computing, such as communicating with the producer management device, consumer management device, post-processing router, remote interface router and result record table visualization query module of the distributed computing system. It needs to implement the interface's methods for task sending, task consumer, post-processor, task splitting, etc.

[0087] based on Figure 2 The task processing method and Figure 4 The task processing system described above is supplemented by another task processing system provided in this application embodiment. See also... Figure 5 This is a system architecture diagram of another task processing system provided in an embodiment of this application. Figure 4 Based on the task processing system shown, Figure 5 The key feature is that the request task queue, which stores messages for multiple subtasks, is located in the memory of the target node, which is used in the big data cluster to call the distributed computing system for distributed computing.

[0088] and Figure 4 The task processing system shown is the same. Figure 5The task processing system shown also includes producer management devices, consumer management devices, and post-processing routers. Figure 5 The target node can be node N in the big data cluster. This approach does not require the introduction of a separately deployed message queue middleware. Instead, a queue is allocated in the memory of node N as a request task queue. Node N can refer to any one of the multiple nodes in the big data cluster; this is just for illustration and is not specifically limited.

[0089] Suppose a user imports 100,000 outstanding payment contracts, and the cluster has three nodes: A, B, and C. If one of these nodes, such as node A, invokes the distributed computing system, it will allocate memory on node A to create a request task queue for this batch of tasks. The next time a user imports contracts, node B might invoke the distributed computing system, and it will also allocate memory on node B to create a request task queue for this batch of tasks.

[0090] exist Figure 5 In this scenario, where a request task queue is created in the memory of node N, the result record table is stored in the big data cluster database. Therefore, it can be seen that... Figure 5 In this embodiment, there is no need to introduce a separately deployed message queue middleware, and therefore no need for its operation and maintenance, thus reducing company costs. Regardless of which node the execution occurs on, the execution logs ultimately need to be recorded somewhere for later review. This can be recorded in a database, a local file system, or Redis. Generally, like other business data, the logs are recorded in the database corresponding to this big data cluster (e.g., ...). Figure 5 The application system database result record table shown is more suitable.

[0091] Furthermore, since there's no need to introduce a separately deployed message queue middleware, there's no need to create topics for the message queue or go through related processes, thus reducing project development time. Figure 5 The task processing system described above can standardize the processing of each batch of tasks and highly abstract the processing process. Developers only need to focus their main efforts on writing relevant business code, such as task senders, task consumers, and post-processors, which also reduces the development time and difficulty of the project.

[0092] based on Figure 2 The task processing method shown and Figure 4 The task processing system shown in this application provides yet another task processing system. See also... Figure 6 This is a schematic diagram of the structure of another task processing system provided in the embodiments of this application. Figure 4The task processing systems shown are different in that: Figure 6 The task processing system includes a distributed computing system, an application load balancer, and... Figure 5 The image shows multiple nodes of a big data cluster. (And...) Figure 5 The same, Figure 6 It is assumed that the request task queue used to store multiple subtask messages is located in the memory of the target node. Figure 6 The result record table in the application system database is used to store subtask messages and their corresponding processing nodes.

[0093] Both the application system load balancer and the application system database result record table communicate with multiple nodes. Compared to Figure 4 In the embodiment shown, the distributed computing system includes a remote interface router in addition to the producer management device, consumer management device, and back-end processing router. The remote interface router acts as an intermediary to enable communication between the consumer management device and the application system load balancer, and also acts as an intermediary to enable communication between the consumer management device and the task consumer.

[0094] For example, when three batch tasks A, B, and C need to be processed, and the entire application system cluster has five nodes, the task sender will send these batch tasks to one of the target nodes that invokes the distributed computing system, such as node A. Node A invokes the distributed computing system, using the producer management device to split these batch tasks into multiple sub-task messages, and submits the split sub-task messages to the request task queue. This request task queue can be a local task queue on node A. The consumer management device of the distributed computing system monitors the request task queue and sends the new sub-task messages obtained from the request task queue to the application system load balancer, such as... Figure 6 5. After receiving a request, the application system load balancer routes the request to a specific node in the cluster according to its routing strategy. Figure 6 In section 5.1 (e.g., node A, node B, or node C), the node calls the remote routing interface in the distributed computing system to calculate the task consumer for handling the request, and then sends the request to the corresponding task consumer for processing and returns the processing result. Figure 6 5.2.

[0095] In this application, the application system load balancer routes requests to a specific node in the big data cluster according to a routing policy, thereby activating that node to invoke the consumer management device of the distributed computing system. Since the batch data is split into multiple subtasks (encapsulated into multiple subtask messages), each subtask message sends a request to the application system load balancer via a remote interface router, requesting the load balancer to forward it to a backend node for distributed computing processing. This node then further invokes the distributed computing system for distributed computing. In this application, the load balancer selects a backend node based on its policy (e.g., round-robin), activates that node to invoke the distributed computing system, and further invokes the consumer management device via the remote interface router. The consumer management device, based on the "consumer name of batch data" field in the corresponding request message, selects a task consumer via the remote interface router to consume the subtask.

[0096] like Figure 6 As shown, the task processing method based on batch data distributed computing in this embodiment utilizes multiple nodes included in the application system to call the distributed computing system for execution. As described above, node A calls the distributed computing system, using the producer management device of the distributed computing system to split the batch task. The split sub-task messages are stored in the local request task queue on that node. The consumer management device of the distributed computing system monitors the request task queue. When a new sub-task message appears, it sends the sub-task message as a corresponding request to the application system load balancer via a remote interface router. The application system load balancer, according to its routing policy, forwards the request to a node of the application system, such as node B. Node B calls the distributed computing system, using its remote interface router to call the consumer management device. Based on the field representing the consumer name of the sub-task contained in the sub-task message, the remote interface router selects the task consumer corresponding to the consumer name field to process the sub-task and obtain the processing result. The processing result is then returned to the consumer management device via the remote interface router, and the consumer management device stores the processing result in the application system database result record table.

[0097] That is, in this embodiment, a remote interface router is introduced into the distributed computing system. It acts as an intermediary to realize communication between the consumer management device and the application system load balancer, and also as an intermediary to realize communication between the consumer management device and the task consumer, so that all computing is executed in multiple threads on different nodes.

[0098] pass Figure 5 and Figure 6 The system architecture diagram of the task processing system shown can be seen that, in Figure 5 and Figure 6 The request task queue used to store multiple subtask messages is located in the memory of the target node. This approach may have a problem: if the tasks in the task request queue have not been consumed, due to some objective reason (such as a power outage of the node), the subsequent unconsumed tasks will disappear, and the post-processor will not be able to run.

[0099] To address this problem, this application also proposes another implementation method: making the task request queue independent of the message middleware of the target node. Based on this, embodiments of this application provide yet another task processing system, see [link to relevant documentation]. Figure 7 The diagram shown is a schematic representation of the system architecture of another task processing system provided in this application embodiment. Assuming that in... Figure 7 In this context, the target node refers to node N in the big data cluster. Figure 7 and Figure 5 and Figure 6 The difference between the example shown and the example's local request task queue, which is based on the application system process, is that... Figure 7 The illustrated embodiment introduces a separately deployed message queue middleware, which can solve... Figure 5 and Figure 6 In this embodiment, if the tasks in the request task queue have not been consumed, due to some objective reason (such as a node power failure), the subsequent unconsumed tasks will disappear, and the post-processor will not be able to run.

[0100] Figure 7 The task processing system shown uses a distributed computing model based on a separately deployed message queue middleware. This requires the introduction of a separately deployed message queue middleware and its maintenance. However, it can ensure that during batch task execution, tasks will not fail to execute due to reasons such as node power failure. This solution can guarantee the reliability of task messages.

[0101] Although introducing a separately deployed message queue middleware requires creating a topic for the message queue, only one topic needs to be created. Even if a related process needs to be followed, it only needs to be followed once, regardless of how many batch tasks there are later, thereby reducing the project development time.

[0102] use Figure 7 The task processing system shown performs distributed computing, which can standardize the processing of each batch of tasks and highly abstract the processing process. Developers only need to focus their main efforts on writing relevant business code, such as task senders, task consumers and post-processors, which also reduces the development time and difficulty of the project.

[0103] Based on the aforementioned task processing method and system, this application provides yet another task processing method, see [link to relevant documentation]. Figure 8 This is a flowchart illustrating another task processing method provided in an embodiment of this application. Figure 8 The task processing method shown can be executed sequentially by the producer management device and the consumer management device. Both the producer management device and the consumer management device can be an electronic device, which can include a terminal or a server. The terminal can include tablet computers, laptops, smart wearable devices, vehicle terminals, etc., and the server can include an independent physical server, a server cluster consisting of one or more servers, or a cloud server capable of cloud computing. Figure 8 The task processing method shown may include the following steps S801 to S610:

[0104] S801. When the distributed computing system is invoked by any node in the big data cluster to perform distributed computing, the producer management device receives at least one batch task sent by the task sender.

[0105] The S802 distributed computing system's producer management device breaks down batch tasks into multiple subtasks and encapsulates each subtask into a subtask message.

[0106] S803, the producer management equipment initializes the result record table based on each subtask message.

[0107] The results record table records information about each subtask message, such as the consumer message field and the batch number of the batch task to which it belongs. For example, the content of a subtask message recorded in the results record table is shown in Table 2 below:

[0108] Table 2

[0109]

[0110]

[0111] S804, the producer management device of the distributed computing system sends subtask messages to the request task queue.

[0112] S805, the consumer management device monitors the request task queue, and if a new subtask message appears in the request message task queue, it retrieves the subtask message from the request task queue.

[0113] S806. The consumer management device of the distributed computing system parses the obtained subtask messages, obtains the corresponding consumers, and calls the corresponding task consumers for processing.

[0114] S807: The task consumer processes the task and finally returns the processing result to the consumer management device of the distributed computing system.

[0115] S808, the consumer management device of the distributed computing system updates the processing results to the result record table.

[0116] S809, the consumer management device of the distributed computing system calculates whether all tasks in the batch have been completed. If they have been completed, it notifies the post-processing router.

[0117] The S810 and post-processing router obtain the batch processing results, perform routing calculations based on the results, and select an appropriate post-processor for data processing.

[0118] The distributed computing system of this application mainly includes a producer management device and a consumer management device. The task processing method applied to the distributed computing system mainly includes a task processing method executed by the producer management device as a task processing device and a task processing method executed by the consumer management device as another task processing device. When the distributed computing system is invoked by any node in the big data cluster, the distributed computing task processing method is executed. The producer management device is used to split the received batch tasks into multiple sub-task messages, each including a consumer name field, and encapsulate and store them in the request task queue. The consumer management device is used to manage the execution of tasks by task consumers. When the producer management device stores the encapsulated new sub-task message in the request task queue, and a new sub-task message appears in the request task queue, the consumer management device obtains the new sub-task message by monitoring the request task queue in real time, and determines the task consumer to execute the task based on the consumer name field encapsulated in the sub-task message.

[0119] This invention's global data splitting method and corresponding routing method implement a global distributed task processing approach, which standardizes the processing logic of each consumer and better tracks task consumption. Furthermore, this global distributed processing approach improves application system operating efficiency, saves application system operating costs, helps standardize task processing for task consumers, and facilitates task execution tracking and querying.

[0120] Based on the aforementioned task processing method and system, this application provides a task processing apparatus, see [link to relevant documentation]. Figure 9 This is a schematic diagram of a task processing device provided in an embodiment of this application. This task processing device corresponds to the aforementioned producer management equipment and can be deployed within the producer management equipment. Figure 9 The schematic diagram of the task processing device shown may include the following units:

[0121] The receiving unit 901 is used to receive at least one batch task sent by the task sender when the distributed computing system is called by any node in the big data cluster to perform distributed computing.

[0122] The splitting unit 902 is used to split the at least one batch task into multiple subtasks;

[0123] Storage unit 903 is used to encapsulate the plurality of subtasks into a plurality of subtask messages and store them sequentially in a request task queue; wherein each subtask message includes a consumer name field; the request task queue is monitored in real time by the consumer management device, and when the consumer management device detects a new subtask message in the request task queue, it routes the new subtask message to the corresponding task consumer according to the consumer name field in the new subtask message, so that the corresponding task consumer processes the new subtask to obtain a processing result.

[0124] In one embodiment, each subtask message in the plurality of subtask messages further includes a post-processor field indicating the corresponding subtask. The post-processor field corresponding to each subtask is used to indicate the post-processor matched by the batch task to which the corresponding subtask belongs. The post-processor matched by each batch task is used to process the processing result of the corresponding batch task after all subtasks in the corresponding batch task have been executed.

[0125] In one embodiment, the distributed computing system further includes a result record table; the result record table is used to store the multiple subtask messages and the processing result corresponding to each subtask message; the distributed computing system further includes a result record table visualization query module; the result record table visualization query module is used to provide users with the ability to query log information related to the batch tasks stored in the result record table.

[0126] In one embodiment, the request task queue is located in the memory of the target node, which refers to the node in the large dataset cluster that calls the distributed computing system;

[0127] or,

[0128] The request task queue is a message queue middleware deployed independently of the target node.

[0129] In the embodiments provided in this application, for distributed computing of batch data, a centralized management approach is adopted in an application system including one or more nodes, such as a big data cluster, to perform distributed computing of batch data. A distributed computing system (also known as a distributed computing engine) is set up for the entire application system, including at least a global producer management device (also known as a producer processor) and a global consumer management device (also known as a consumer processor). The producer management device splits the batch tasks into encapsulated task messages with specific data structures. The encapsulated task messages include the consumer name field of multiple subtasks into which the batch data is split and are stored in the request task queue. This is a global splitting process to achieve global distributed scheduling of the entire application system. At the same time, a global consumer management device is set up for the entire application system to monitor whether there is new data in the request task queue in real time. When new data is found, the corresponding consumer is called to execute the task according to the relevant consumer name field stored in the encapsulated subtask message.

[0130] This invention provides a global task processing method that implements a globally distributed task processing approach, thereby standardizing the processing logic of each consumer and better tracking message consumption. Furthermore, this globally distributed processing approach improves application system operating efficiency, saves application system operating costs, facilitates standardized task processing by task consumers, and enhances task execution tracking and querying.

[0131] Based on the aforementioned task processing method and system, this application provides a task processing apparatus, see [link to relevant documentation]. Figure 10 This is a schematic diagram of another task processing device provided in an embodiment of this application. This task processing device corresponds to the aforementioned consumer management device and can be deployed in the consumer management device. Figure 10 The schematic diagram of the task processing device shown may include the following units:

[0132] The monitoring unit 1001 is used to monitor the request task queue in real time when the distributed computing system is called by any node in the big data cluster to perform distributed computing; the request task queue is used to store multiple sub-task messages when multiple sub-task messages occur, the multiple sub-task messages are obtained by encapsulating multiple sub-tasks separately, the multiple sub-tasks are obtained by the producer management device from splitting at least one batch task; the at least one batch task is received from the task sender when the distributed computing system is called by any node in the big data cluster to perform distributed computing;

[0133] Selection unit 1002 is used to select the task consumer corresponding to the consumer name field to process the subtask and obtain the processing result when a new subtask message appears in the request task queue, based on the field representing the consumer name of the subtask contained in the new subtask message.

[0134] In one embodiment, the distributed computing system further includes a result record table; the result record table is used to store the plurality of subtask messages and the processing result corresponding to each subtask message;

[0135] Each of the multiple subtask messages also includes a post-processor field indicating the corresponding subtask. The post-processor field for each subtask is used to indicate the post-processor matched by the batch task to which the corresponding subtask belongs. The monitoring unit 1001 is also used to monitor in real time whether all subtasks in the batch task to which the corresponding subtask belongs in the request task queue have been executed; and if they have been executed, to obtain the processing result of the batch task from the result record table, and to select a suitable post-processor for data processing according to the post-processor name field.

[0136] In one embodiment, when the monitoring unit 1001 detects a new subtask message in the request task queue, and selects the task consumer corresponding to the consumer name field in the new subtask message to process the subtask and obtain a processing result, the following steps are performed:

[0137] When a new subtask message is detected in the request task queue, the obtained subtask message is sent to the application system load balancer. The application load balancer selects a node from at least one node in the big data cluster to call the remote interface router to call the consumer management device, so as to select the corresponding task consumer through the remote interface router to process the subtask according to the consumer name field in the subtask message and obtain the processing result.

[0138] In the embodiments provided in this application, for distributed computing of batch data, a centralized management approach is adopted in an application system including one or more nodes, such as a big data cluster, to perform distributed computing of batch data. A distributed computing system (also known as a distributed computing engine) is set up for the entire application system, including at least a global producer management device (also known as a producer processor) and a global consumer management device (also known as a consumer processor). The producer management device splits the batch tasks into encapsulated task messages with specific data structures. The encapsulated task messages include the consumer name field of multiple subtasks into which the batch data is split and are stored in the request task queue. This is a global splitting process to achieve global distributed scheduling of the entire application system. At the same time, a global consumer management device is set up for the entire application system to monitor whether there is new data in the request task queue in real time. When new data is found, the corresponding consumer is called to execute the task according to the relevant consumer name field stored in the encapsulated subtask message.

[0139] This invention provides a global task processing method that implements a globally distributed task processing approach, thereby standardizing the processing logic of each consumer and better tracking message consumption. Furthermore, this globally distributed processing approach improves application system operating efficiency, saves application system operating costs, facilitates standardized task processing by task consumers, and enhances task execution tracking and querying.

[0140] Based on the aforementioned task processing method, task processing system, and task processing device, embodiments of this application provide an electronic device that can correspond to the aforementioned producer management device or consumer management device. See also... Figure 11 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes: at least one processor 1101; at least one memory 1102; and one or more I / O interfaces 1103 connected between the processor 1101 and the memory 1102; wherein the memory 1102 stores one or more computer programs executable by the at least one processor 1101, and the one or more computer programs are executed by the at least one processor 1101 to enable the at least one processor 1101 to execute:

[0141] When the distributed computing system is invoked by any node in the big data cluster to perform distributed computing, it receives at least one batch task sent by the task sender and splits the at least one batch task into multiple sub-tasks.

[0142] The multiple subtasks are encapsulated into multiple subtask messages and stored sequentially in a request task queue. Each subtask message includes a consumer name field. The request task queue is monitored in real time by the consumer management device. When the consumer management device detects a new subtask message in the request task queue, it routes the new subtask message to the corresponding task consumer based on the consumer name field in the new subtask message, so that the corresponding task consumer can process the new subtask to obtain the processing result.

[0143] Alternatively, one or more computer programs may be executed by at least one processor 1101 to cause at least one processor 1101 to perform:

[0144] When the distributed computing system is invoked by any node in the big data cluster to perform distributed computing, the request task queue is monitored in real time. The request task queue is used to store multiple subtask messages when multiple subtask messages occur. The multiple subtask messages are obtained by encapsulating multiple subtasks separately. The multiple subtasks are obtained by the producer management device by splitting at least one batch task. The at least one batch task is received from the task sender when the distributed computing system is invoked by any node in the big data cluster to perform distributed computing.

[0145] When a new subtask message is detected in the request task queue, the task consumer corresponding to the consumer name field of the new subtask message is selected to process the subtask and obtain the processing result.

[0146] In the embodiments provided in this application, for distributed computing of batch data, a centralized management approach is adopted in an application system including one or more nodes, such as a big data cluster, to perform distributed computing of batch data. A distributed computing system (also known as a distributed computing engine) is set up for the entire application system, including at least a global producer management device (also known as a producer processor) and a global consumer management device (also known as a consumer processor). The producer management device splits the batch tasks into encapsulated task messages with specific data structures. The encapsulated task messages include the consumer name field of multiple subtasks into which the batch data is split and are stored in the request task queue. This is a global splitting process to achieve global distributed scheduling of the entire application system. At the same time, a global consumer management device is set up for the entire application system to monitor whether there is new data in the request task queue in real time. When new data is found, the corresponding consumer is called to execute the task according to the relevant consumer name field stored in the encapsulated subtask message.

[0147] This invention provides a global task processing method that implements a globally distributed task processing approach, thereby standardizing the processing logic of each consumer and better tracking message consumption. Furthermore, this globally distributed processing approach improves application system operating efficiency, saves application system operating costs, facilitates standardized task processing by task consumers, and enhances task execution tracking and querying.

[0148] This application also provides a computer-readable storage medium storing a computer program thereon, wherein the computer program, when executed by a processor / processor core, implements the above-described task processing method. The computer-readable storage medium may be volatile or non-volatile.

[0149] This application also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in the processor of an electronic device, the processor in the electronic device executes the above-described processing method based on distributed computing of batch data.

[0150] Those skilled in the art will understand that all or some of the steps, systems, and apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software can be distributed on a computer-readable storage medium, which may include computer storage media (or non-transitory media) and communication media (or transient media).

[0151] As is known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable program instructions, data structures, program modules, or other data). Computer storage media includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), static random access memory (SRAM), flash memory or other memory technologies, portable compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible to a computer. Furthermore, it is known to those skilled in the art that communication media typically contain computer-readable program instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0152] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0153] The computer program instructions used to perform the operations of this application may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuits, such as programmable logic circuits, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), are personalized by utilizing the status information of the computer-readable program instructions. These electronic circuits can execute the computer-readable program instructions to implement various aspects of this application.

[0154] The computer program product described herein can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0155] Various aspects of this application are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0156] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0157] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0158] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction, which contains one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0159] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for general illustrative purposes only and should not be construed as limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in conjunction with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in conjunction with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this application as set forth by the appended claims.

Claims

1. A task processing method applied to a distributed computing system, wherein the distributed computing system performs distributed computing when invoked by any node in a big data cluster, the distributed computing system includes a producer management device and a consumer management device, the consumer management device being a global management device for the distributed computing system, the consumer management device being used to manage the task execution of task consumers; The task processing method is executed by the producer management equipment, and the task processing method includes: When the distributed computing system is invoked by any node in the big data cluster to perform distributed computing, it receives at least one batch task sent by the task sender and splits the at least one batch task into multiple sub-tasks. The multiple subtasks are encapsulated into multiple subtask messages and stored sequentially in the request task queue. Each subtask message includes a consumer name field and a post-processor field representing the corresponding subtask. The request task queue is monitored in real time by the consumer management device. When the consumer management device detects a new subtask message in the request task queue, it routes the new subtask message to the corresponding task consumer based on the consumer name field in the new subtask message, so that the corresponding task consumer can process the new subtask to obtain the processing result.

2. The task processing method according to claim 1, characterized in that, The post-processor field corresponding to each subtask is used to indicate the post-processor matched by the batch task to which the corresponding subtask belongs. The post-processor matched by each batch task is used to process the processing result of the corresponding batch task after all subtasks in the corresponding batch task have been executed.

3. The task processing method according to claim 2, characterized in that, The distributed computing system further includes a result record table; the result record table is used to store the multiple subtask messages and the processing result corresponding to each subtask message; the distributed computing system further includes a result record table visualization query module; The result record table visualization query module is used to provide users with the ability to query log information related to the batch task stored in the result record table.

4. The task processing method according to claim 1, characterized in that, The request task queue is located in the memory of the target node, which refers to the node in the large dataset cluster that calls the distributed computing system. or, The request task queue is a message queue middleware deployed independently of the target node.

5. A task processing method applied to a distributed computing system, wherein the distributed computing system performs distributed computing when invoked by any node in a big data cluster, the distributed computing system includes a producer management device and a consumer management device, the consumer management device being a global management device for the distributed computing system, and the producer management device being used to split received batch tasks into multiple subtask messages, each including a consumer name field and a post-processor field representing the corresponding subtask, and store them in a request task queue; The task processing method is executed by the consumer management device, and the task processing method includes: When the distributed computing system is invoked by any node in the big data cluster to perform distributed computing, the request task queue is monitored in real time. The request task queue is used to store multiple subtask messages when multiple subtask messages occur. The multiple subtask messages are obtained by encapsulating multiple subtasks separately. The multiple subtasks are obtained by the producer management device by splitting at least one batch task. The at least one batch task is received from the task sender when the distributed computing system is invoked by any node in the big data cluster to perform distributed computing. When a new subtask message is detected in the request task queue, the task consumer corresponding to the consumer name field of the new subtask message is selected to process the subtask and obtain the processing result.

6. The task processing method according to claim 5, characterized in that, The distributed computing system also includes a result record table; the result record table is used to store the multiple subtask messages and the processing result corresponding to each subtask message; The post-processor field corresponding to each subtask is used to indicate the post-processor matched to the batch task to which the corresponding subtask belongs. The task processing method further includes: Real-time monitoring of whether all subtasks in the batch task to which the corresponding subtask in the request task queue belongs have been completed; as well as If the execution is complete, the processing result of the batch task is obtained from the result record table, and a suitable post-processor is selected for data processing based on the post-processor name field.

7. The task processing method according to claim 6, characterized in that, The distributed computing system also includes a remote interface router; the big data cluster includes an application system load balancer; and When a new subtask message is detected in the request task queue, the process involves selecting the task consumer corresponding to the consumer name field in the new subtask message to process the subtask and obtain a processing result, including: When a new subtask message is detected in the request task queue, the obtained subtask message is sent to the application system load balancer. The application system load balancer selects a node from at least one node in the big data cluster to call the remote interface router to call the consumer management device, so as to select the corresponding task consumer through the remote interface router to process the subtask according to the consumer name field in the subtask message and obtain the processing result.

8. A task processing device applied to a distributed computing system, wherein the distributed computing system performs distributed computing when invoked by any node in a big data cluster, the distributed computing system including a producer management device and a consumer management device, the consumer management device being a global management device for the distributed computing system, the consumer management device being used to manage the task execution of task consumers; the task processing device includes: The receiving unit is used to receive at least one batch task sent by the task sender when the distributed computing system is called by any node in the big data cluster to perform distributed computing. A splitting unit is used to split the at least one batch task into multiple subtasks; A storage unit is used to encapsulate the multiple subtasks into multiple subtask messages and store them sequentially in a request task queue; wherein each subtask message includes a consumer name field and a post-processor field indicating the corresponding subtask. The request task queue is monitored in real time by the consumer management device. When the consumer management device detects a new subtask message in the request task queue, it routes the new subtask message to the corresponding task consumer according to the consumer name field in the new subtask message, so that the corresponding task consumer can process the new subtask to obtain the processing result.

9. A task processing device applied to a distributed computing system, wherein the distributed computing system performs distributed computing when invoked by any node in a big data cluster, the distributed computing system includes a producer management device and a consumer management device, the consumer management device being a global management device for the distributed computing system, and the producer management device being used to split received batch tasks into multiple subtask messages, each including a consumer name field and a post-processor field representing the corresponding subtask, and store them in a request task queue; The task processing device includes: The monitoring unit is used to monitor the request task queue in real time when the distributed computing system is invoked by any node in the big data cluster to perform distributed computing. The request task queue is used to store multiple subtask messages when multiple subtask messages occur. The multiple subtask messages are obtained by encapsulating multiple subtasks separately. The multiple subtasks are obtained by the producer management device from splitting at least one batch task. The at least one batch task is received from the task sender when the distributed computing system is invoked by any node in the big data cluster to perform distributed computing. The selection unit is used to select the task consumer corresponding to the consumer name field to process the subtask and obtain the processing result when a new subtask message appears in the request task queue, based on the consumer name field containing the new subtask message.

10. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores one or more computer programs that can be executed by the at least one processor, and the one or more computer programs are executed by the at least one processor to enable the at least one processor to perform the task processing method according to any one of claims 1-4; Alternatively, the task processing method according to any one of claims 5-7 may be performed.

Images