Task processing method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a task processing method and device, equipment, a storage medium and a program product, and relates to the technical field of data processing. According to the method, tasks of a to-be-processed batch are obtained, batch task files corresponding to the tasks of the to-be-processed batch are generated, then a distributed task processing cluster is scheduled to process the tasks based on the batch task files in a file mode, and the task processing efficiency is improved under the condition that target tasks failing to be processed exist. The distributed task processing cluster is dispatched to process the target task in the streaming mode, the technical effect of dynamically regulating and controlling the processing mode of the task according to the execution result of the task is achieved, the target task is processed in the streaming mode, and the problem that the task processing efficiency is reduced due to the fact that the target task continues to be processed in a file mode is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a task processing method and device, equipment, a storage medium and a program product. BACKGROUND

[0002] In the financial field, the financial system often faces large-scale data processing scenarios. For example, end-of-day clearing business, and scanning and identifying of a large number of transaction data by a risk control system. In order to improve the data processing efficiency, the prior art mainly adopts a centralized batch processing scheme and a batch processing scheme relying on a distributed architecture to process batch tasks.

[0003] Among them, the centralized batch processing scheme adopts single-node sequential processing of batch tasks, controls the number of concurrent threads through a fixed-size thread pool, and all task requests are executed in queue order. The system architecture adopted by this method is simple, but when encountering a high-concurrency scenario, the centralized architecture cannot be dynamically expanded, which easily leads to system crashes. The distributed architecture has expansion performance. The batch processing technology relying on the distributed architecture improves the data processing efficiency by distributing task data to multiple nodes for parallel processing, but the method relies on a pre-set processing mode to process multiple tasks.

[0004] Therefore, the prior art cannot dynamically regulate the processing logic of batch tasks according to the execution state of the tasks, thereby reducing the processing efficiency of the batch tasks. SUMMARY

[0005] The present application provides a task processing method, device, equipment, storage medium and program product to solve the technical problem that the prior art cannot dynamically regulate the processing logic of batch tasks according to the execution state of the tasks, thereby reducing the processing efficiency of the batch tasks.

[0006] In a first aspect, the present application provides a task processing method, comprising:

[0007] obtaining tasks of a batch to be processed;

[0008] generating a batch task file corresponding to the tasks of the batch to be processed;

[0009] scheduling a distributed task processing cluster to process the tasks based on the batch task file in a file mode;

[0010] in the case that there is a target task that fails to be processed, scheduling the distributed task processing cluster to process the target task in a streaming mode.

[0011] In a second aspect, the present application provides a task processing device, comprising:

[0012] an obtaining module configured to obtain tasks of a batch to be processed;

[0013] The generating module is configured to generate a batch task file corresponding to the tasks of the batch to be processed;

[0014] The processing module is configured to schedule the distributed task processing cluster to process the tasks in a file mode based on the batch task file.

[0015] The processing module is further configured to, in a case where there is a target task that fails to be processed, schedule the distributed task processing cluster to process the target task in a streaming mode.

[0016] In a third aspect, an electronic device is provided, including a memory and a processor.

[0017] The memory stores computer-executable instructions.

[0018] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the first aspect and / or various possible implementation manners of the first aspect.

[0019] In a fourth aspect, a computer-readable storage medium is provided, which stores computer-executable instructions. The computer-executable instructions are executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0020] In a fifth aspect, a computer program product is provided, which includes a computer program. The computer program is executed by a processor to implement the first aspect and / or various possible implementation manners of the first aspect.

[0021] The task processing method provided by the present application acquires tasks of a batch to be processed, generates a batch task file corresponding to the tasks of the batch to be processed, and then schedules a distributed task processing cluster to process the tasks in a file mode based on the batch task file. In a case where there is a target task that fails to be processed, the distributed task processing cluster is scheduled to process the target task in a streaming mode. The technical effect of dynamically regulating the processing mode of the tasks according to the execution result of the tasks is achieved. The target task is processed in the streaming mode, which avoids continuing to process the target task in the file mode, thereby reducing the problem of low task processing efficiency. BRIEF DESCRIPTION OF DRAWINGS

[0022] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and serve to explain the principles of the present application together with the specification.

[0023] Figure 1 A flowchart of a task processing method provided by an embodiment of the present application Figure 1 ;

[0024] Figure 2 A flowchart of a task processing method provided for an embodiment of the present application Figure 2 ;

[0025] Figure 3 A flowchart of a task processing method provided for an embodiment of the present application Figure 3 ;

[0026] Figure 4 A flowchart of a task processing method provided for an embodiment of the present application Figure 4 ;

[0027] Figure 5 A structural diagram of a task processing device provided for the present application

[0028] Figure 6 A structural diagram of an electronic device provided for the present application

[0029] The specific embodiments of the present application have been shown through the above-described drawings, and will be described in more detail hereinafter. These drawings and the written description are not intended to restrict the scope of the present application in any way, but to illustrate the concept of the present application for those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0030] The exemplary embodiments will be described in detail herein with reference to the attached drawings. The following description is made with reference to the accompanying drawings in which like reference numerals refer to like elements, unless the context of use indicates otherwise. The following description of exemplary embodiments is not representative of all embodiments consistent with the present application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present application as detailed in the appended claims.

[0031] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of related data comply with relevant laws, regulations and standards of relevant countries and regions, necessary security measures are taken, public order and good customs are not violated, and appropriate operation portals are provided for users to choose authorization or refusal.

[0032] And the present application relates to the big data analysis of user information (including but not limited to personal biological characteristics, identity data, consumption data, asset data, electronic terminal operation data, etc.), and adopts artificial intelligence technology to make automatic decision, makes technical scheme based on automatic decision result that has important influence on personal rights and interests decision, provides corresponding operation entrance for user, for user to choose to agree or refuse automatic decision result;If the user chooses to refuse, enter the expert decision process.

[0033] It should be noted that the task processing method, device, equipment, storage medium and program product provided by the present application can be used in the field of financial technology, and can also be used in any field other than financial technology. The application field of the task processing method, device, equipment, storage medium and program product in the present application is not limited.

[0034] First, the terms involved in the present application are explained:

[0035] Distributed batch processing: refers to splitting large-scale transaction tasks into multiple sub-tasks, and executing them in parallel through distributed cluster nodes to achieve high-throughput transaction processing mode.

[0036] TPS (Transactions Per Second, Transactions Per Second): is a core indicator to measure the concurrent processing capacity of the system.

[0037] File mode: read data in batches through pre-generated files and execute transactions, reduce real-time database interaction, and improve processing efficiency.

[0038] Streaming mode: real-time streaming data from the database to execute transactions, high flexibility.

[0039] With the development of big data technology, the fields of finance, Internet of Things, traffic flow scheduling, etc. are facing the processing demand of massive data. For example, in the financial field, the daily clearing business, and the scanning and identification of a large number of transaction data by the risk control system. In order to realize batch processing of data and improve the efficiency of data processing, the existing technology adopts centralized batch processing scheme and batch processing scheme relying on distributed architecture to process large-scale task data in batches.

[0040] Among them, the centralized batch processing scheme adopts single node sequential processing of batch tasks, controls the number of concurrent threads through a fixed size thread pool, and all task requests are executed in queue order. The system architecture adopted by this method is simple, which can meet the data processing condition of small data volume and low number of concurrent requests. However, in the case of high concurrency, the centralized architecture cannot be dynamically expanded to cope with the sudden increase in a large number of task requests, which can easily lead to system crash and task backlog.

[0041] The distributed architecture has scalability. Batch processing technology relying on the distributed architecture improves data processing efficiency by distributing task data to multiple nodes for parallel processing. However, this method relies on a pre-configured processing mode for task processing, for example, a file mode and a streaming mode.

[0042] That is, the prior art cannot dynamically regulate the processing logic of batch tasks according to the execution state of the tasks, thereby reducing the processing efficiency of the batch tasks.

[0043] The task processing method provided by the present application processes the tasks based on batch task files by scheduling the distributed task processing cluster in the file mode, and processes the target tasks in the streaming mode by scheduling the distributed task processing cluster in the streaming mode in the case that there are target tasks with processing failures. This method achieves the technical effect of dynamically regulating the processing mode of the tasks according to the execution result of the tasks, and avoids the problem of reducing the task processing efficiency due to the task file reloading and other operations by processing the target tasks with processing failures in the streaming mode.

[0044] The technical solutions of the present application and how the technical solutions of the present application solve the above technical problems will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes can not be described again in some embodiments. The embodiments of the present application will be described below with reference to the drawings.

[0045] Figure 1 Flowchart of a task processing method provided by an embodiment of the present application Figure 1 As shown in Figure 1 , the execution subject of the method can be, for example, a task scheduling system. The method comprises the following steps:

[0046] S101, obtaining tasks in a to-be-processed batch.

[0047] A large number of tasks to be executed are stored in a business system. Specifically, all the tasks to be executed are loaded from the business system, and the all the tasks to be executed are split into multiple batches of tasks, and a to-be-processed batch is determined from the multiple batches. For example, any of the multiple batches is selected as the to-be-processed batch, and the number of the to-be-processed batches can be determined based on the number of batch nodes of the distributed cluster, for example. For example, the number of the to-be-processed batches is consistent with the number of the batch nodes.

[0048] Optionally, at least one division parameter is determined from the data of the to-be-executed task according to a preset rule, wherein the preset rule comprises: determining at least one key data of the to-be-executed task as the division parameter, and the number of the division parameters is not less than a preset number; the at least one division parameter of the to-be-executed task is combined to obtain a target division parameter, and a hash value of the target division parameter is calculated, and then the hash value is taken modulo to obtain a corresponding modulus value, so that the to-be-executed task is divided into a sub-batch corresponding to the modulus value. That is, if the modulus value is 1, the corresponding to-be-executed task is divided into the first sub-batch.

[0049] For example, the total to-be-executed task, for example, 10 million pension opening tasks, is divided into 200 to-be-processed batch tasks according to a preset rule, and each to-be-processed batch includes multiple pension opening tasks. The preset rule is, for example, batch division according to the geographical position corresponding to the 10 million pension opening tasks.

[0050] S102, a batch task file corresponding to the to-be-processed batch is generated.

[0051] For the multiple tasks of the to-be-processed batch, the task data corresponding to each task is obtained, and a batch task file is generated based on the task data corresponding to the multiple tasks respectively. For example, multiple customers reserve a pension opening service in a business reservation system, and for the to-be-processed batch of opening services, the task scheduling system loads the customer information in the reservation record of the to-be-processed batch as the batch task file.

[0052] S103, scheduling the distributed task processing cluster to process the tasks in the file mode based on the batch task file.

[0053] The distributed task processing cluster is scheduled to respectively distribute the batch processing files corresponding to the multiple to-be-processed batches to multiple batch nodes of the distributed task processing cluster. For any batch node in the multiple batch nodes, after the batch node receives the corresponding batch task file, the batch node obtains multiple task data from the batch task file, generates a task processing request based on the multiple task data, and sends the task processing request to an online node, so that the online node executes the corresponding multiple tasks based on the task processing request, and feeds back the execution result to the batch node. The number of online nodes is determined by the execution state of the multiple tasks.

[0054] It can be understood that the plurality of batch nodes generate and send the task requests concurrently. The number of concurrent is determined based on an execution index of the task, which can be an execution rate for example. In a case where the execution rate is less than a preset execution rate, the number of concurrent is reduced by a first parameter, and in a case where the execution rate is not less than the preset execution rate, the number of concurrent is increased by a second parameter. The first parameter and the second parameter are both positive numbers, and are set according to data processing requirements, which are not limited in the present application.

[0055] S104, in a case where there is a target task with processing failure, scheduling the distributed task processing cluster to process the target task in a streaming mode.

[0056] The task execution result fed back by the online node is obtained from the plurality of batch nodes, and the task execution result is used to indicate task processing success or processing failure. In a case where at least one task execution result in the plurality of task execution results indicates processing failure, the corresponding task is determined as a target task, and the distributed task processing cluster is scheduled to be in a streaming mode.

[0057] The task execution result corresponding to the target task includes but is not limited to a task identifier. Based on the task execution results of the plurality of target tasks, a target task queue is generated. The target task queue includes at least one target task respectively corresponding to a to-be-retried message. The to-be-retried message is generated based on the task execution result of the corresponding target task, and the to-be-retried message includes the task identifier of the corresponding target task.

[0058] The to-be-retried message of each target task is obtained from the target task queue concurrently by a stream processing module deployed in the distributed task processing cluster, and real-time task data corresponding to the target task is obtained from the business database based on the task identifier in the to-be-retried message. The stream processing module generates a task request corresponding to each target task based on at least one real-time task data, and distributes at least one task request to at least one online node respectively, so that the online node executes the corresponding target task and feeds back the execution result to the stream processing module. The task request includes the real-time task data of the target task.

[0059] The task processing method provided in the embodiment achieves the technical effect of dynamically regulating the processing mode of the task according to the execution result of the task, and avoids continuously processing the target task in the file mode by processing the target task in the streaming mode, thereby reducing the problem of low task processing efficiency.

[0060] Figure 2 A flowchart of a task processing method provided by an embodiment of the present application Figure 2 As shown in Figure 1 The embodiment provides detailed description of a switching strategy of a file mode and a streaming mode on the basis of the embodiment. The method comprises the following steps. Figure 3

[0061] S201, monitoring a processing failure rate of the batch of tasks in a process of processing the tasks in the file mode.

[0062] Specifically, a first quantity of processed tasks corresponding to a current time is determined, and task execution results corresponding to all the processed tasks are obtained from the plurality of batch nodes, the task execution results comprising a first result identifier and a second result identifier. The first result identifier is used to indicate that the corresponding task is executed successfully, and the second result identifier is used to indicate that the corresponding task is executed unsuccessfully. Further, a second quantity of task execution results carrying the second result identifier is determined, and the current processing failure rate of the batch of tasks is determined based on the second quantity and the first quantity.

[0063] It can be understood that the processing failure rate of the batch of tasks is determined according to a preset period, or the processing failure rate of the batch of tasks is determined in real time. The present application does not limit this.

[0064] S202, in a case where the processing failure rate is greater than or equal to a preset threshold, controlling the distributed task processing cluster to switch from the file mode to the streaming mode.

[0065] It is determined whether the processing failure rate obtained in the step S201 is greater than or equal to a preset threshold. If yes, it indicates that there is a batch task file error and the like in the file mode, and the distributed task processing cluster is controlled to switch from the file mode to the streaming mode, so as to improve the processing efficiency of the tasks through the streaming mode.

[0066] For example, in a case where the data processing performance of an external system decreases, the streaming mode maintains the stable operation of the external system by reducing the sending rate of the task request, and is beneficial to ensuring the successful execution of the tasks compared with the file mode, so as to improve the execution efficiency of the tasks. The external system is, for example, an information authentication system interface that needs to be called for the opening of a pension account. If the interface is down or the response time increases from 200 ms to 10 s, all the concurrent requests in the file mode will be timed out or failed. The information authentication system is the external system.

[0067] S203, scheduling the distributed task processing cluster to process target tasks in the processed tasks and the remaining unprocessed tasks in the streaming mode.

[0068] ​The processed task is determined based on the task execution result fed back by the online node, and the task corresponding to the task execution result of the batch node at the current time is the processed task.

[0069] Based on the second result identifier in the task execution result, the corresponding target task is determined. For example, the second result identifier corresponding to the task execution result further includes a processing failure reason of the processed task. In the case where the processing failure reason belongs to a non-user side reason, the corresponding processed task is determined as the target task, the target task is processed in the streaming mode, and the remaining unprocessed tasks are processed.

[0070] It should be noted that the user side reason may be, for example, that the task is in a completed state, task data is missing, and the non-user side reason may be, for example, system busy, network exception, etc.

[0071] S204, if the processing failure rate continues to be less than the preset threshold, the distributed task processing cluster is switched from the file mode to the streaming mode after the last task is processed in the file mode.

[0072] In the case where the processing failure rate obtained in the step S201 is less than the preset threshold, it is indicated that the processing effect of the task in the file mode is good, and the distributed task processing cluster is switched from the file mode to the streaming mode after the last task is processed in the file mode, so as to execute the step S205 to quickly process the target task in the streaming mode, thereby improving the processing efficiency of the task.

[0073] S205, the distributed task processing cluster is scheduled to process the target task in the streaming mode.

[0074] The step S205 is similar to the explanation of the step S104, and will not be described here.

[0075] The task processing method provided in the embodiment monitors the processing failure rate of the batch task during processing of the task in the file mode, and controls the distributed task processing cluster to be switched from the file mode to the streaming mode in the case where the processing failure rate is greater than or equal to the preset threshold, and then schedules the distributed task processing cluster to process the target task in the streaming mode. The processed task and the remaining unprocessed tasks are processed in the streaming mode, so as to switch to the streaming mode in time in the case where a large number of tasks are executed in the file mode, which helps to improve the processing efficiency of the batch task.

[0076] Furthermore, if the failure rate remains below a preset threshold, after processing the last task in file mode, the distributed task processing cluster is switched from file mode to streaming mode, and the distributed task processing cluster is scheduled to process the target task in streaming mode, thereby improving task processing efficiency based on the efficiency of streaming mode.

[0077] Figure 3 A flowchart illustrating a task processing method provided in this application embodiment. Figure 1-2 ,like Figure 4 As shown, in this embodiment... Figure 4 Based on the embodiments, a possible task processing method is described in detail. The method includes:

[0078] S301. Obtain the tasks in the batch to be processed; generate the batch task file corresponding to the tasks in the batch to be processed.

[0079] The explanation of step S301 is similar to that of steps S101 and S102 above, and will not be repeated here.

[0080] S302. Schedule the distributed task processing cluster in file mode to process tasks based on batch task files.

[0081] Optionally, during the process of the distributed task processing cluster processing the batch of tasks, the distributed task processing cluster may be scaled down or expanded based on the number of nodes that can process tasks and the number of tasks to be processed in the batch.

[0082] This method dynamically scales up or down nodes based on the number of nodes that can handle tasks and the number of tasks to be processed in the batch, thereby achieving dynamic control of data processing resources. This avoids the waste of idle data processing resources and ensures the rational allocation of resources, thus helping to improve data processing efficiency.

[0083] For example, the difference between the number of nodes capable of processing tasks and the number of tasks to be processed in the batch is calculated. If this difference is greater than a first threshold, the distributed task processing cluster is scaled down. The first threshold is a positive number and is set based on the actual needs of task processing; this application does not impose any restrictions on it.

[0084] Furthermore, if the aforementioned difference is less than the second threshold, the distributed task processing cluster undergoes node expansion. The second threshold is a negative number and is set based on the actual needs of task processing; this application does not impose any restrictions on it.

[0085] It can be understood that the node expansion processing can reduce the first preset number of batch nodes, and the node reduction processing can increase the second preset number of processing nodes. The first preset number and the second preset number are set according to user demand, and the application does not limit this.

[0086] S303, monitoring the processing failure reason of the target task with processing failure.

[0087] In the case where the task execution result indicates processing failure, the processing failure reason of the corresponding target task is further determined. The processing failure reason includes a user-side reason and a non-user-side reason. The user-side reason and the non-user-side reason are as shown in the above step S203, and will not be repeated here.

[0088] S304, adding a retry identifier to the target task in the case where the processing failure reason belongs to the non-user-side reason.

[0089] The retry identifier is used to indicate retrying the target task, and the retry identifier can be, for example, "1".

[0090] The target task with a user-side reason for failure cannot be successfully processed by retrying, and the method adds a retry identifier to the target task with a non-user-side reason for failure, avoiding retrying the target task with a user-side reason for failure, thereby helping to improve the processing efficiency of batch tasks.

[0091] S305, scheduling the distributed task processing cluster to process the target task with the retry identifier in a streaming mode.

[0092] Optionally, if the number of target tasks with retry identifiers is multiple, the target tasks with retry identifiers are sorted, and the distributed task processing cluster is scheduled to process the target tasks with retry identifiers in a streaming mode based on the sorting order. This method helps to ensure the smooth execution of high-priority tasks, thereby meeting the urgent needs of the business.

[0093] It can be understood that the target tasks with retry identifiers are sorted in descending order of processing priority. The processing priority of the target task is determined based on the task creation time, and / or the number of task retries, and / or the task urgency.

[0094] The task processing method provided in this embodiment obtains a batch of tasks to be processed and generates a batch task file corresponding to the batch of tasks. Then, it schedules a distributed task processing cluster to process the tasks in file mode based on the batch task file. During task processing in file mode, it monitors the reasons for processing failures of target tasks. If the reason for failure is not user-side, it adds a retry flag to the target task. Then, it schedules the distributed task processing cluster to process the target task with the retry flag in streaming mode. This method improves the processing efficiency of batch tasks by adding a retry flag to target tasks whose failure reasons are not user-side, thus avoiding retries for target tasks whose failure reasons are user-side.

[0095] Figure 1-3 A flowchart illustrating a task processing method provided in this application embodiment. Figure 5 ,like Figure 5 As shown, in this embodiment... Figure 6 Based on the embodiments, a method for dynamically adjusting the task processing volume based on task processing latency is described in detail during task processing. This method includes:

[0096] S401. During the process of the batch of tasks being processed by the distributed task processing cluster, obtain the concurrent task processing volume of the distributed task processing cluster, as well as the task processing response time.

[0097] This function retrieves the concurrent task processing volume of the distributed task processing cluster at the current moment. Specifically, each batch node has a corresponding counting module, which counts the tasks processed by that batch node. For example, when the first batch node starts processing a new task, the value in the counting module is incremented by 1. When the task is completed, regardless of whether the execution result is success or failure, the value in the counting module is decremented by 1, thus ensuring that the value in the counting module accurately reflects the instantaneous load of the corresponding batch node.

[0098] In addition, during the task processing, the start time and completion time of each task are recorded. Based on the start time and completion time, the first response time of the corresponding task is determined. This first response time can be stored in a monitoring database, thereby obtaining multiple first response times corresponding to the batch of tasks at the current time. Based on the multiple first response times, the corresponding average response time is determined and used as the task processing response time.

[0099] S402. Based on the number of concurrent tasks, the task processing response time, and the target throughput, obtain the task processing latency.

[0100] The target throughput is the maximum number of tasks processed per second. Based on the number of concurrent tasks, the task processing response time, and the target throughput, the task processing delay is calculated. Optionally, the task processing delay is determined using the following formula:

[0101]

[0102] wherein, the task processing delay, the number of concurrent tasks, the target throughput, the task processing response time.

[0103] It can be understood that in the above formula, based on the number of concurrent tasks and the target throughput, the target processing time corresponding to the number of concurrent tasks is determined. If the target processing time is reached, it means that the number of concurrent tasks being processed is close to the maximum number of tasks that the system can handle in a stable running state.

[0104] S403, based on the task processing delay, adjust the number of concurrent tasks to make the number of concurrent tasks equal to or close to the target throughput.

[0105] If the task processing delay is positive, it means that the actual task processing time consumed during task processing is higher than the target processing time corresponding to the number of concurrent tasks, i.e., the system response is slow, then reduce the number of concurrent tasks in the next batch to make the number of concurrent tasks equal to or close to the target throughput, thereby helping to improve the efficiency of task processing.

[0106] If the task processing delay is negative, it means that the actual task processing time consumed is higher than the target processing time corresponding to the number of concurrent tasks, i.e., the data processing resources are sufficient, then increase the number of concurrent tasks in the next batch, which does not exceed the target throughput, to ensure that the system is in the best running state.

[0107] The task processing method provided in the embodiment, in the process of processing the batch of tasks by the distributed task processing cluster, acquires the concurrent task processing amount of the distributed task processing cluster and the task processing response time length, and acquires the task processing time delay based on the concurrent task processing amount, the task processing response time length, and the target throughput, and then adjusts the concurrent task processing amount based on the task processing time delay, so that the concurrent task processing amount is equal to or close to the target throughput. The method achieves the technical effect of reducing the concurrent task processing amount in the case of a long system response time length, so that the concurrent task processing amount is equal to or close to the target throughput, thereby improving the processing efficiency of the batch of tasks. In addition, the method increases the concurrent task processing amount in the case of sufficient data processing resources, so that the concurrent task processing amount is equal to or close to the target throughput, thereby increasing the processing efficiency of the batch of tasks while ensuring that the system is in the best operating state.

[0108] Figure 6 The structural schematic diagram of the task processing device provided in the present application is shown in ​ The task processing device 50 provided in the embodiment includes:

[0109] The acquisition module 501 is configured to acquire a batch of tasks to be processed.

[0110] The generation module 502 is configured to generate a batch task file corresponding to the batch of tasks to be processed.

[0111] The processing module 503 is configured to schedule the distributed task processing cluster to process the above-mentioned tasks based on the batch task file in the file mode.

[0112] The processing module 503 is further configured to schedule the distributed task processing cluster to process the target task in the stream mode in the case where there is a target task with processing failure.

[0113] In a possible implementation manner, the device further includes a monitoring module 504 and a switching module 505.

[0114] The monitoring module 504 is configured to monitor the processing failure rate of the batch of tasks in the process of processing the tasks in the file mode.

[0115] The switching module 505 is configured to control the distributed task processing cluster to switch from the file mode to the stream mode in the case where the processing failure rate is greater than or equal to a preset threshold.

[0116] The processing module 503 is specifically configured to schedule the distributed task processing cluster to process the target task in the stream mode and the remaining unprocessed tasks.

[0117] In a possible implementation, the switching module 505 is further configured to, if the processing failure rate continues to be less than the preset threshold, control the distributed task processing cluster to switch from the file mode to the streaming mode after processing of the last task in the file mode is completed.

[0118] The processing module 503 is specifically configured to schedule the distributed task processing cluster to process the target task in the streaming mode.

[0119] In a possible implementation, the apparatus further includes an adding module 506.

[0120] The monitoring module 504 is further configured to monitor a processing failure reason of the target task that fails in processing during processing of the task in the file mode.

[0121] The adding module 506 is configured to add a retry identifier to the target task in a case where the processing failure reason belongs to a non-user side reason.

[0122] The processing module 503 is specifically configured to schedule the distributed task processing cluster to process the target task with the retry identifier in the streaming mode.

[0123] In a possible implementation, the apparatus further includes a sorting module 507.

[0124] The sorting module 507 is configured to sort the target tasks with the retry identifier in a case where the number of the target tasks with the retry identifier is more than one.

[0125] The processing module 503 is specifically configured to schedule the distributed task processing cluster to process the target tasks with the retry identifier in the streaming mode based on the sorting order.

[0126] In a possible implementation, the sorting module 507 is specifically configured to sort the target tasks with the retry identifier in descending order of processing priorities of the target tasks with the retry identifier.

[0127] In a possible implementation, the apparatus further includes an adjusting module 508.

[0128] The obtaining module 501 is further configured to obtain a concurrent task processing amount and a task processing response time length of the distributed task processing cluster during processing of the batch of tasks by the distributed task processing cluster.

[0129] The obtaining module 501 is further configured to obtain a task processing time delay based on the concurrent task processing amount, the task processing response time length, and a target throughput.

[0130] The adjusting module 508 is configured to adjust the concurrent task processing amount based on the task processing time delay, so that the concurrent task processing amount is equal to or close to the target throughput.

[0131] In a possible implementation, the processing module 503 is further configured to, during processing of the batch of tasks by the distributed task processing cluster, perform node scaling-in or scaling-out processing on the distributed task processing cluster based on the number of nodes capable of processing tasks and the number of tasks in the batch to be processed.

[0132] The task processing apparatus provided in this embodiment can execute the method provided in the method embodiments, and has similar implementation principles and technical effects. Details are not described herein again.

[0133] ​ A structural schematic diagram of an electronic device provided in this application is shown in FIG. 6. As shown in the figure, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, the memory 602 and the communication component 603 are connected through a bus 604. ​

[0134] In the implementation process, the at least one processor 601 executes the computer-executable instructions stored in the memory 602, so that the at least one processor 601 executes the method described above.

[0135] The specific implementation process of the processor 601 can refer to the method embodiments described above, and has similar implementation principles and technical effects. Details are not described herein again.

[0136] In the above embodiments, it should be understood that the processor can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. The steps of the method disclosed in the application can be directly embodied as execution completed by a hardware processor, or executed by a combination of hardware and software modules in the processor.

[0137] The memory can include a random access memory (RAM), and can also include a non-volatile memory (NVM), for example, at least one disk memory.

[0138] ​The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the convenience of representation, the bus in the drawings of the present application does not limit to only one bus or one type of bus.

[0139] The present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the method described above.

[0140] The present application also provides a computer readable storage medium, which stores computer execution instructions, and when a processor executes the computer execution instructions, the method described above is implemented.

[0141] The readable storage medium described above can be realized by any type of volatile or non-volatile storage device or their combination, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special purpose computer.

[0142] An exemplary readable storage medium is coupled to the processor, so that the processor can read information from the readable storage medium, and can write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (ASIC). Of course, the processor and the readable storage medium can also exist as discrete components in the device.

[0143] The division of units is only a logical functional division, and in actual implementation, there can be another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0144] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment of the present application.

[0145] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit.

[0146] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the part of the present application that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method of each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0147] Those skilled in the art can understand that all or part of the steps of the above-mentioned method embodiments can be completed by program instruction related hardware. The aforementioned program can be stored in a computer readable storage medium. The program executes the steps of the above-mentioned method embodiments when executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk, and various program code storage media.

[0148] It should be noted that for the above-mentioned method embodiments, in order to simply describe, they are all expressed as a combination of a series of actions, but those skilled in the art should know that the present application is not limited by the order of the described actions, because according to the present application, certain steps can be performed in other order or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily necessary for the present application.

[0149] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0150] It should be understood that the above-described device embodiments are merely illustrative, and the device of this application can also be implemented in other ways. For example, the division of units / modules in the above embodiments is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units, modules, or components may be combined, or integrated into another system, or some features may be ignored or not executed.

[0151] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0152] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0153] If the integrated units / modules are implemented in the form of software program modules and sold or used as independent products, they can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory and includes a number of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the method of the present application. The aforementioned memory includes: a U disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0154] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments. The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described, however, as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0155] Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. The application is intended to cover any variations, uses or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice in the art to which the application pertains or can relate. The specification and examples are to be regarded as exemplary only, and the true scope and spirit of the application are indicated by the following claims.

[0156] It should be understood that the application is not limited to the precise construction that has been described above and illustrated in the accompanying drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the application is limited only by the claims that follow.

Claims

1. A task processing method, characterized in that, include: Retrieve tasks in the pending batch; Generate batch task files corresponding to the batch of tasks to be processed; The distributed task processing cluster is scheduled to process the tasks based on the batch task files in file mode; In the event that a target task fails to be processed, the distributed task processing cluster is scheduled to process the target task in streaming mode.

2. The method according to claim 1, characterized in that, The method further includes: During the processing of the task using the file mode, monitor the processing failure rate of the batch of tasks; If the processing failure rate is greater than or equal to a preset threshold, the distributed task processing cluster is controlled to switch from the file mode to the streaming mode. The process of scheduling the distributed task processing cluster to process the target task in streaming mode includes: The distributed task processing cluster is scheduled to process the target task among the processed tasks, as well as the remaining unprocessed tasks, in the streaming mode.

3. The method according to claim 2, characterized in that, The method further includes: If the processing failure rate is consistently less than a preset threshold, then after processing the last task in the file mode, the distributed task processing cluster is controlled to switch from the file mode to the streaming mode. The distributed task processing cluster is scheduled to process the target task in the streaming mode.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: During the processing of the task using the file mode, monitor the reasons for the failure of the target task. If the reason for the processing failure is not due to a user-side reason, a retry flag is added to the target task; The process of scheduling the distributed task processing cluster to process the target task in streaming mode includes: The distributed task processing cluster is scheduled to process target tasks with retry flags in streaming mode.

5. The method according to claim 4, characterized in that, The scheduling of the distributed task processing cluster in streaming mode to process target tasks with retry flags includes: If there are multiple target tasks with retry flags, then sort the target tasks with retry flags. The distributed task processing cluster is scheduled to process target tasks with retry flags in streaming mode based on the sorting order.

6. The method according to claim 5, characterized in that, The sorting of target tasks with retry flags includes: The target tasks with retry flags are sorted in descending order of their processing priority.

7. The method according to any one of claims 1-3, characterized in that, The method further includes: During the process of the batch of tasks being processed by the distributed task processing cluster, the concurrent task processing volume of the distributed task processing cluster and the task processing response time are obtained. Based on the number of concurrent tasks, the task processing response time, and the target throughput, the task processing latency is obtained. Based on the task processing latency, the concurrent task processing volume is adjusted so that the concurrent task processing volume is equal to or close to the target throughput.

8. The method according to any one of claims 1-3, characterized in that, The method further includes: During the process of the distributed task processing cluster processing the batch of tasks, the distributed task processing cluster is scaled down or expanded based on the number of nodes that can process tasks and the number of tasks to be processed in the batch.

9. A task processing device, characterized in that, include: The acquisition module is used to acquire tasks in batches to be processed. The generation module is used to generate batch task files corresponding to the batch of tasks to be processed; The processing module is used to schedule the distributed task processing cluster to process the tasks based on the batch task files in file mode; The processing module is also used to schedule the distributed task processing cluster to process the target task in streaming mode when there is a target task that has failed to be processed.

10. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 8.

12. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 8.

Images