Task processing method and device, equipment, storage medium and program product

Abstract

The invention provides a task processing method and device, equipment, a storage medium and a program product. The method comprises the steps that in response to a task processing request for multiple to-be-processed tasks, a target first working process carrying a first lock identifier is determined from a first working process set, and the first lock identifier indicates a data downloading lock used for providing a data downloading permission; obtaining a task data set based on the plurality of to-be-processed tasks through the target first working process; for each to-be-processed task, a target second working process carrying a second lock identifier is determined from a second working process set, and the second lock identifier indicates a task processing lock used for providing task processing permission; and processing the task data of the to-be-processed task through the target second working process. By means of the task processing method and device, the problems of resource scheduling and resource preemption caused by concurrent processing of the multiple to-be-processed tasks can be effectively solved, and therefore the task processing efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of task processing, and in particular to a task processing method, device, equipment, storage medium and program product. Background Art

[0002] In various business systems, task processing usually refers to batch task processing, i.e. running a batch. Batch task processing is a processing mode in which a series of pending tasks are executed by a computer program based on batch input task data without human intervention. The essence of batch task processing is that within a specified time, a computer program performs business processing on the task data to be processed to generate target data required by the business. Batch task processing is suitable for scenarios with large data volumes and low requirements for real-time and interactivity. For example, data reconciliation and batch issuance of funds by financial enterprises, as well as batch backup logs in log systems, etc.

[0003] However, in the task processing implementation of related technologies, a task serial processing mode on a single machine is usually adopted. Due to the limitation of single machine resources, such as disk space, CPU performance, memory capacity and number of network connections, the maximum concurrent processing capacity that a single machine can support is limited. In other words, related technologies have the problems of low order of magnitude of running batch tasks, low concurrency, and performance depending on a single machine, which cannot effectively solve the resource scheduling and preemption problems caused by multi-task concurrency. Summary of the invention

[0004] The embodiments of the present application provide a task processing method, apparatus, device, storage medium and program product, which can effectively solve the problems of resource scheduling and resource preemption caused by the concurrent processing of multiple tasks to be processed.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] An embodiment of the present application provides a task processing method, the method comprising: in response to a task processing request for multiple tasks to be processed, determining a target first work process carrying a first lock identifier from a first work process set, the first lock identifier indicating a data download lock used to provide data download permission; obtaining a task data set based on the multiple tasks to be processed through the target first work process, the task data set including task data respectively indicated by the multiple tasks to be processed; for each of the multiple tasks to be processed, determining a target second work process carrying a second lock identifier from a second work process set, the second lock identifier indicating a task processing lock used to provide task processing permission; and processing the task data of the tasks to be processed through the target second work process.

[0007] An embodiment of the present application provides a task processing device, including: a first determination module, used to determine, in response to a task processing request for multiple tasks to be processed, a target first work process carrying a first lock identifier from a first work process set, wherein the first lock identifier indicates a data download lock used to provide data download permission; an acquisition module, used to obtain a task data set based on the multiple tasks to be processed through the target first work process, wherein the task data set includes task data respectively indicated by the multiple tasks to be processed; a second determination module, used to determine, for each of the multiple tasks to be processed, a target second work process carrying a second lock identifier from a second work process set, wherein the second lock identifier indicates a task processing lock used to provide task processing permission; a task processing module, used to process the task data of the tasks to be processed through the target second work process.

[0008] In the above scheme, the device also includes a third determination module, which is used to: when receiving the task processing request, assign the data download lock to the work process in the first work process set; when the first work process in the first work process set is assigned to the data download lock, associate the first lock identifier of the data download lock with the first work process; and determine the first work process associated with the first lock identifier as the target first work process.

[0009] In the above scheme, the third determination module is also used to: determine the first cycle of each work process in the first work process set; obtain the first delay time preset for each of the work processes; use the first delay time to update the first cycle to obtain the second cycle of each of the work processes; based on the second cycle, allocate the data download lock to the work process in the first work process set.

[0010] In the above scheme, the third determination module is also used to: start timing from the time each work process in the first work process set is started; in response to the timing of any of the work processes being equal to the duration corresponding to the second period, determine any of the work processes as the first work process, and allocate the data download lock to the first work process.

[0011] In the above scheme, the device also includes a fourth determination module, which is used to: assign the task processing lock to the work process in the second work process set; when the second work process in the second work process set is assigned to the task processing lock, associate the second lock identifier of the task processing lock with the second work process; and determine the second work process associated with the second lock identifier as the target second work process.

[0012] In the above scheme, the fourth determination module is also used to: determine the third period of each work process in the second work process set; obtain the second delay time preset for each of the work processes; use the second delay time to update the third period to obtain the fourth period of each of the work processes; based on the fourth period, allocate the task processing lock to the work processes in the second work process set.

[0013] In the above scheme, the fourth determination module is also used to: start timing from the time each work process in the second work process set is started; in response to the timing of any of the work processes being equal to the duration corresponding to the fourth period, determine any of the work processes as the second work process, and allocate the task processing lock to the second work process.

[0014] In the above scheme, the device further includes a creation module, and the creation module is used to: create a first list of pending tasks; the first list of pending tasks includes pending tasks in an unprocessed state among the multiple pending tasks.

[0015] In the above scheme, the fourth determination module is also used to: obtain the first pending task from the first pending task list; detect the task status of the first pending task; and when the detection result indicates that the first pending task is still in the unprocessed state, allocate the task processing lock to the work process in the second work process set for the first pending task.

[0016] In the above solution, the device further includes an updating module, and the updating module is used to update the task state of the first to-be-processed task from the unprocessed state to the processed state.

[0017] In the above scheme, the device also includes an allocation module, which is used to: when the detection result indicates that the first pending task is in a processing state, delete the first pending task from the first pending task list to obtain a second pending task list; obtain the second pending task in the unprocessed state from the second pending task list, and for the second pending task, allocate the task processing lock to the work process in the second work process set.

[0018] In the above scheme, the device also includes a recording module, which is used to: create a progress list for the pending tasks in the first pending task list; in the process of processing the task data of the pending tasks through the target second working process, record the task processing progress of the pending tasks into the progress list according to the progress status change time of the pending tasks.

[0019] In the above scheme, the device also includes an interruption recovery module, which is used to: in response to an interruption in the task processing process, record the current task processing progress and task interruption identifier of the task to be processed into the progress list; in response to the task processing process recovering from an interrupted state to a normal processing state, obtain the current task processing progress of the task to be processed from the progress list based on the task interruption identifier; and continue to process the task data of the task to be processed based on the current task processing progress.

[0020] In the above scheme, the device also includes a deletion module, which is used to: obtain the task execution cycle of the pending task; when receiving the task processing request, start timing the task execution duration of the pending task; in response to the task execution duration being greater than the duration corresponding to the task execution cycle, and the current task processing progress of the pending task being in an unfinished state, delete the pending task from the first pending task list.

[0021] In the above scheme, the device also includes a generation module, which is used to: obtain the number of first work processes in the first work process set, the logical concurrency of the task processing system, and the number of copies of the task processing system; determine the number of second work processes based on the number of tasks to be processed, the number of first work processes, the logical concurrency and the number of copies; generate the second work process set according to the number of second work processes.

[0022] In the above scheme, the generation module is also used to: determine the product of the number of copies, the number of the first working processes and the logical concurrency as the actual concurrency of the task processing system at the current moment; obtain the call time and preset adjustment parameters of the target task in the task processing system; based on the actual concurrency, the call time and the preset adjustment parameters, determine the number of query requests of the task processing system at the current moment; based on the number of query requests, determine the number of the second working processes.

[0023] An embodiment of the present application provides an electronic device, comprising: a memory for storing computer-executable instructions; and a processor for implementing the task processing method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0024] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the task processing method provided in the embodiment of the present application when executed by a processor.

[0025] An embodiment of the present application provides a computer program product, which includes computer executable instructions, and the computer executable instructions are stored in a computer-readable storage medium; wherein, when a processor of an electronic device reads the computer executable instructions from the computer-readable storage medium and executes the computer executable instructions, the task processing method provided in the embodiment of the present application is implemented.

[0026] The embodiments of the present application have the following beneficial effects:

[0027] When performing task processing, a target first work process carrying a first lock identifier is determined from the first work process set, and the first lock identifier indicates a data download lock for providing data download permissions. In this way, when obtaining a task data set based on multiple tasks to be processed, since the target first work process in the first work process set carries the first lock identifier, the data download lock indicated by the first lock identifier can provide data download permissions for the target first work process, thereby allowing the target first work process in the first work process set to obtain the task data set, while other first work processes in the first work process set cannot obtain the task data set. By downloading task data through the target first work process carrying the first lock identifier in the first work process set, thereby obtaining the task data set, it is possible to avoid concurrent processing problems caused by multiple first work processes downloading task data from memory during the data download stage, thereby improving the efficiency and accuracy of data downloading.

[0028] And, after obtaining the task data set, for each of the multiple pending tasks, a target second work process carrying a second lock identifier is determined from the second work process set, and the second lock identifier indicates a task processing lock for providing task processing authority. In this way, when processing task data, since the target second work process in the second work process set carries the second lock identifier, the task processing lock indicated by the second lock identifier can provide the target second work process with the task processing authority of the pending task, so the target second work process in the second work process set is allowed to process the task data of the pending task, which can avoid the problem of concurrent processing caused by multiple second work processes simultaneously processing the task data of the same pending task in the task data processing stage, thereby ensuring the accuracy and stability of the task data processing process, and improving the efficiency of task processing. In summary, determining the target first work process based on the data download lock, and determining the target second work process based on the task processing lock can effectively solve the problems of resource scheduling and resource preemption caused by the concurrent processing of multiple pending tasks, thereby improving the efficiency of task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a structural diagram of the task processing system architecture provided by an embodiment of the present application;

[0030] Figure 2 is a structural diagram of a task processing device provided in an embodiment of the present application;

[0031] Figure 3 This is an optional flowchart of the task processing method provided in the embodiment of the present application;

[0032] Figure 4 is another optional flowchart of the task processing method provided in the embodiment of the present application;

[0033] Figure 5 It is a schematic diagram of the implementation process of determining the first working process of the target provided in the embodiment of the present application;

[0034] Figure 6 It is a schematic diagram of the implementation process of allocating a data download lock provided in an embodiment of the present application;

[0035] Figure 7 It is a schematic diagram of an implementation flow of determining a second work process set provided in an embodiment of the present application;

[0036] Figure 8 It is a schematic diagram of the implementation process of determining the second working process of the target provided in the embodiment of the present application;

[0037] Fig. 9 This is an optional implementation flow diagram of allocating task processing locks provided in an embodiment of the present application;

[0038] Fig.10 This is another optional implementation flow diagram of allocating task processing locks provided in an embodiment of the present application;

[0039] Fig.11 It is a flowchart of a task interruption recovery method provided in an embodiment of the present application;

[0040] Fig.12 It is a schematic diagram of implementing task clearing of pending tasks that have timed out and have not been completed, provided in an embodiment of the present application;

[0041] Fig.13 It is a schematic diagram of the overall process of the high-performance processing guarantee engine for massive data provided by the embodiment of the present application;

[0042] Fig.14 It is a schematic diagram of the complete design and execution flow of the massive data high-performance processing guarantee engine provided by the embodiment of the present application;

[0043] Fig.15 It is a schematic diagram of triggering a new batch running task provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0045] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0046] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0047] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0048] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0049] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0050] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0051] 1) High concurrency: refers to the ability of a system to process a large number of requests in parallel within a unit of time in the design of Internet distributed system architecture. High concurrency is an important performance indicator, especially in Internet applications with a huge number of users, where high concurrency is one of the key factors in measuring system performance.

[0052] 2) Hadoop Distributed File System (HDFS): It is one of the core components of the Hadoop framework and is a distributed file system designed to run on general-purpose hardware. HDFS provides large-scale data storage and management capabilities with high reliability, high availability, and high throughput.

[0053] 3) Cloud File Storage (CFS) is a secure and scalable file storage service. CFS can provide unlimited scalability, high reliability, and globally shared file storage capabilities for computing resources such as virtual machines and containers on the cloud through standard file access protocols. CFS can be used in business scenarios such as content management and Web applications, big data analysis, pan-entertainment, and media applications.

[0054] In the task processing implementation of the related technology, a task serial processing mode on a single machine is adopted. Due to the limitation of single machine resources, such as disk space, CPU performance, memory capacity and number of network connections, the maximum concurrent processing capacity that a single machine can support is limited, and the serial execution of tasks further limits the concurrent potential of the system. In order to improve the concurrent processing capability, the related technology divides resources through sequential scheduling and step size strategy, or uses hashing and other means, and allocates the divided resources to different work processes for processing. However, the scalability of this method is limited, and the effect of improving the concurrent capability by increasing the number of copies is not significant, and it cannot effectively solve the resource scheduling and preemption problems caused by multi-task concurrency.

[0055] In response to at least one of the above related technologies, an embodiment of the present application proposes a high-performance processing method for massive data based on scalable concurrent capabilities, multi-task concurrent scheduling control and task interruption recovery mechanism. The method adopts multi-machine deployment (i.e., realizing multiple copies of data), task parallel processing (i.e., downloading N task data and processing N task data in parallel at the same time) and logical concurrent execution (i.e., using multi-process, multi-thread or multi-coroutine technology). In theory, this task concurrent execution method can achieve unlimited expansion, only limited by the processing bottleneck of the called downstream system. In addition, the embodiment of the present application adopts a preemptive scheduling strategy, allowing idle work processes to actively preempt idle tasks and realize exclusive processing. This strategy has good scalability because the concurrent processing capability of the system can be directly improved by increasing the number of copies.

[0056] The following describes an exemplary application of the task processing device (i.e., electronic device) provided in the embodiment of the present application. The task processing device provided in the embodiment of the present application can be implemented as a laptop, tablet computer, desktop computer, set-top box, mobile device (e.g., mobile phone, portable music player, personal digital assistant, dedicated messaging device, portable gaming device), smart phone, smart speaker, smart watch, smart TV, vehicle-mounted terminal, and other types of user terminals capable of performing task processing, and can also be implemented as a server. The following describes an exemplary application when the task processing device is implemented as a server.

[0057] See also Figure 1 , Figure 1 It is a structural diagram of the task processing system 100 architecture provided in an embodiment of the present application. To support a task processing application, the terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0058] The terminal 400 is used to send a task processing request to the server 200. The server 200 constitutes a task processing device of an embodiment of the present application. The server 200 is used to respond to the task processing request for multiple tasks to be processed, determine the target first work process carrying the first lock identifier from the first work process set; then, through the target first work process, obtain a task data set based on multiple tasks to be processed; the task data set includes task data respectively indicated by multiple tasks to be processed; then, for each task to be processed, determine the target second work process carrying the second lock identifier from the second work process set; finally, through the target second work process, process the task data of the task to be processed to obtain a task processing result. The task processing result is returned to the terminal 400, so that the task processing result is output at the terminal 400 or the terminal 400 continues to perform the next business processing based on the task processing result.

[0059] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal 400 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, a car terminal, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0060] See also Figure 2 , Figure 2 is a schematic diagram of the structure of an electronic device 40 provided in an embodiment of the present application, Figure 2 The electronic device 40 shown may be a task processing device, which includes: at least one processor 410, a memory 450, at least one network interface 420, and a user interface 430. The various components in the task processing device are coupled together via a bus system 440. It is understood that the bus system 440 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 440 is not described in detail. Figure 2 Various buses are labeled as bus system 440 .

[0061] The processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] The user interface 430 includes one or more output devices 431 that enable presentation of media content, including one or more speakers and one or more visual display screens. The user interface 430 also includes one or more input devices 432, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices that are physically remote from the processor 410.

[0064] The memory 450 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 450 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, memory 450 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0066] Operating system 451, including system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0067] A network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420. Exemplary network interfaces 420 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.; a presentation module 453 is used to enable presentation of information (e.g., a user interface for operating peripheral devices and displaying content and information) via one or more output devices 431 (e.g., display screens, speakers, etc.) associated with a user interface 430; an input processing module 454 is used to detect one or more user inputs or interactions from one of the one or more input devices 432 and translate the detected inputs or interactions.

[0068] In some embodiments, the device provided in the embodiments of the present application can be implemented in software. Figure 2 The task processing device 455 stored in the memory 450 is shown, which can be software in the form of a program and a plug-in, etc., and includes the following software modules: a first determination module 4551, an acquisition module 4552, a second determination module 4553 and a task processing module 4554. These modules are logical, and therefore can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0069] In some further embodiments, the apparatus provided in the embodiments of the present application may be implemented in hardware. As an example, the apparatus provided in the embodiments of the present application may be a processor in the form of a hardware decoding processor, which is programmed to execute the task processing method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor may be one or more application specific integrated circuits (ASIC), digital signal processors (DSP), programmable logic devices (PLD), complex programmable logic devices (CPLD), field programmable gate arrays (FPGA) or other electronic components.

[0070] In some embodiments, the terminal or server can implement the task processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, or it can be a small program that can be embedded in any APP, that is, a program that can be run only by obtaining it in the browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0071] The task processing method provided in each embodiment of the present application can be executed by an electronic device, wherein the electronic device can be a server or a terminal, that is, the task processing method provided in each embodiment of the present application can be executed by a server, can be executed by a terminal, or can be executed through interaction between a server and a terminal.

[0072] See also Figure 3 , Figure 3 is an optional flowchart of the task processing method provided in the embodiment of the present application, which will be combined with Figure 3 The steps shown are described in detail, and the execution subject of the task processing method is a server as an example. The method includes the following steps S101 to S104:

[0073] Step S101, in response to a task processing request for a plurality of tasks to be processed, determining a target first working process carrying a first lock identifier from a set of first working processes.

[0074] In the embodiment of the present application, pending tasks refer to task items queued for processing in the task processing system, and the number of pending tasks is multiple. For example, pending tasks may be transfer tasks in payment scenarios, batch backup log tasks and file downloads in log systems, etc. Task processing requests are used to request the task processing system to perform task processing on multiple pending tasks at the same time, for example, task processing requests may be transfer requests, batch backup log requests and file download requests. The task processing system here is a distributed task scheduling system for task processing, which can be used to coordinate and manage the mechanism of task processing in distributed systems. The task processing system can handle large-scale, high-concurrency tasks and provide high availability and scalability. The first worker process set refers to a set of a group of worker processes, and the worker process is usually used to perform specific tasks, for example, the above-mentioned multiple pending tasks. The number of worker processes in the first worker process set is multiple, and each worker process may be a thread or process in the task processing system. In addition, the first worker process set includes a target first worker process carrying a first lock identifier of a data download lock.

[0075] The first lock identifier indicates a data download lock for providing data download permission. The data download lock is an exclusive lock, and the data download lock can also be a synchronization mechanism. The data download lock can be used to provide data download permission for the work process, that is, to control the work process's access to shared resources (for example, data in memory, where the memory is used to store task data in the task data set), and ensure that only one work process with data download permission is allowed to perform data download operations on the shared resource at a certain moment. That is to say, when a work process in the first work process set carries the first lock identifier, the work process has data download permission. Other work processes in the first work process set cannot obtain the data download lock, that is, other work processes do not have data download permission until the data download lock is released. The first lock identifier is a mark or identifier used to uniquely identify the work process in the first work process set that carries the first lock identifier. The identifier can be the ID, name, or any other attribute that can uniquely identify the work process. The target first working process refers to the working process selected from the first working process set and currently carrying the first lock identifier. The target first working process has the data download permission provided by the data download lock, that is, the data download operation can be performed through the target first working process.

[0076] In an embodiment of the present application, when determining the target first work process carrying the first lock identifier of the data download lock from the first work process set, the associated data of all the first work processes in the first work process set can be judged in turn. If the associated data of a first work process contains the first lock identifier, the first work process is determined as the target first work process.

[0077] Step S102: obtaining a task data set based on a plurality of tasks to be processed through a target first working process.

[0078] In the embodiment of the present application, the task data set includes task data respectively indicated by multiple tasks to be processed.

[0079] In some embodiments, when a task processing request for multiple pending tasks is received, if the task processing request encapsulates the acquisition path of the task data set corresponding to the multiple pending tasks, and only the task data set is stored under the acquisition path, the acquisition path of the task data set can be directly parsed. Then, by acquiring the target first working process of the data download lock, according to the acquisition path of the task data set, the task data set is searched from the acquisition path, and the task data set is downloaded to obtain the task data set, that is, the acquisition of the task data set is realized.

[0080] In other embodiments, a task identifier can also be assigned to each of the multiple pending tasks, so that the multiple pending tasks correspond to a task identifier respectively, forming a task identifier set. When a task processing request for multiple pending tasks is received, if the task processing request simultaneously encapsulates the acquisition path of the task data set corresponding to the multiple pending tasks and the task identifier set of the multiple pending tasks, and the acquisition path stores other data in addition to the task data set, then the acquisition path of the task data set and the task identifier set can be directly parsed. Then, by acquiring the target first working process of the data download lock, according to the acquisition path of the task data set and the task identifier set, the data set corresponding to the task identifier set is searched from the acquisition path, and the data set is used as the task data set. Finally, the task data set is downloaded, that is, the acquisition of the task data set is realized.

[0081] In some embodiments, the task data set can be pre-stored in a database, and the target first working process can directly query the database to obtain the task data set based on the acquisition path and the task identification set of multiple tasks to be processed. The task data set can also be pre-cached in a memory or a distributed cache system (e.g., HDFS), and the target first working process can obtain the task data set from the cache based on the acquisition path and the task identification set of multiple tasks to be processed. The acquisition path of the task data set is not specifically limited here.

[0082] Step S103: for each of the multiple tasks to be processed, determine a target second work process carrying a second lock identifier from the set of second work processes.

[0083] In the embodiment of the present application, the number of work processes in the second work process set is multiple, and each work process can be a thread or process in the task processing system. In addition, the second work process set includes a target second work process carrying a second lock identifier. The second lock identifier indicates a task processing lock for providing task processing authority, which is another exclusive lock. The task processing lock is also a synchronization mechanism. The task processing lock can be used to provide task processing authority for the work process, that is, to control the processing of shared resources (for example, task data sets), and to ensure that only one work process with task processing authority is allowed at a certain moment, and task processing operations can be performed for the shared resources. In other words, when a work process in the second work process set carries the second lock identifier, the work process has task processing authority. Other work processes in the second work process set cannot obtain the task processing lock, that is, other work processes do not have task processing authority until the task processing lock is released. The second lock identifier is a mark or identifier for uniquely identifying the work process carrying the second lock identifier in the second work process set. The identifier can be the ID, name, or any other attribute that can uniquely identify the work process. The target second work process refers to the work process selected from the second work process set and currently carrying the second lock identifier. The target second work process has the data download permission provided by the task processing lock, that is, the task processing operation can be executed through the target second work process, and the task data of the task to be processed can be processed.

[0084] In an embodiment of the present application, when determining the target second work process with the second lock identifier carrying the task processing lock from the second work process set, the associated data of all the first work processes in the first work process set can be judged in turn. If the associated data of a second work process contains the second lock identifier, the second work process is determined as the target second work process.

[0085] For each pending task, a target second work process carrying the second lock identifier of the task processing lock is determined from the second work process set, that is, each of the multiple pending tasks corresponds to a target second work process carrying the second lock identifier of the task processing lock, and the target second work process is used to process the task data of the pending task. For different pending tasks, the target second work process used to process the task data of the pending task is different.

[0086] Step S104: Processing the task data of the task to be processed through the target second working process.

[0087] In an embodiment of the present application, when the target second work process that can currently perform the task processing operation is determined from the second work process set, the task data of the task to be processed is processed through the target second work process to obtain the task processing result. Processing refers to a series of operations on the task data of the task to be processed. For example, the processing can be data conversion, data analysis, data calculation or data integration of the task data, etc., through the processing of the task data to obtain the desired result or output, that is, the task processing result. The task processing result refers to the output result obtained after the target second work process processes the task data. The task processing result can be whether the task to be processed is successfully processed, whether the task processing is completed, or whether an error occurs during the task processing process. The task processing result can also be data generated after the task processing, such as calculation results, report files or log information.

[0088] The task processing method provided by the embodiment of the present application determines the target first work process carrying the first lock identifier from the first work process set when performing task processing, and the first lock identifier indicates a data download lock used to provide data download permissions. In this way, when obtaining a task data set based on multiple tasks to be processed, since the target first work process in the first work process set carries the first lock identifier, the data download lock indicated by the first lock identifier provides data download permissions for the target first work process, thereby allowing the target first work process in the first work process set to obtain the task data set, while other first work processes in the first work process set cannot obtain the task data set. By downloading task data through the target first work process carrying the first lock identifier in the first work process set, thereby obtaining the task data set, it is possible to avoid concurrent processing problems caused by multiple first work processes downloading task data from the memory during the data download stage, thereby improving the efficiency and accuracy of data downloading.

[0089] And, after obtaining the task data set, for each of the multiple pending tasks, a target second work process carrying a second lock identifier is determined from the second work process set, and the second lock identifier indicates a task processing lock for providing task processing authority. In this way, when processing task data, since the target second work process in the second work process set carries the second lock identifier, the task processing lock indicated by the second lock identifier provides the target second work process with the task processing authority of the pending task, so the target second work process in the second work process set is allowed to process the task data of the pending task, which can avoid the problem of concurrent processing caused by multiple second work processes simultaneously processing the task data of the same pending task in the task data processing stage, thereby ensuring the accuracy and stability of the task data processing process and improving the efficiency of task processing. In summary, determining the target first work process based on the data download lock and determining the target second work process based on the task processing lock can effectively solve the problems of resource scheduling and resource preemption caused by the concurrent processing of multiple pending tasks, thereby improving the efficiency of task processing.

[0090] The following will describe the task processing method in the embodiment of the present application in conjunction with the interaction between the terminal and the server in the task processing system. It should be noted that the task processing method here is a task processing method implemented by the interaction between the terminal and the server, which is substantially the same as the task processing method executed by the server in the above embodiment. The only difference is that the embodiment of the present application also describes the actions performed by the terminal during the execution of the task processing method, and some steps can be executed by both the terminal and the server. Therefore, for the steps in this embodiment that are the same as those in the above embodiment but have different execution subjects, this embodiment is only an exemplary description. In the process of implementation, it can be executed by any execution subject, and the embodiment of the present application does not limit this.

[0091] Figure 4 is another optional flow chart of the task processing method provided in the embodiment of the present application, such as Figure 4 As shown, the method includes the following steps S201 to S209:

[0092] Step S201: the terminal receives a task processing operation input by a user for a plurality of tasks to be processed.

[0093] In one implementation, a task processing application may be run on the terminal, and the user may input task processing operations for multiple pending tasks on the client of the task processing application. In the task processing application, a task processing function may be provided, and the user may input task processing operations for multiple pending tasks on the task processing function page to trigger task processing requests for multiple pending tasks. In another implementation, there may be multiple terminals, each of which has a task processing application run on it, and the user of each terminal may input task processing operations for pending tasks on the client of the task processing application, thereby forming multiple pending tasks, and the task processing requests for multiple pending tasks may be triggered by the task processing operations performed on each terminal.

[0094] In some embodiments, when a user inputs a task processing operation for multiple tasks to be processed, the user can also input the acquisition path of the task data set at the same time. When the terminal receives the acquisition path, a confirmation task processing window will pop up on the task processing function page. After the terminal detects that the user clicks the confirmation task processing button, further processing is performed based on the acquisition path to implement task processing. Alternatively, in other embodiments, the user can directly input the acquisition path of the task data set on the task processing function page. When the terminal receives the acquisition path, it can directly trigger the task processing function and perform further processing based on the acquisition path to implement task processing.

[0095] Step S202 : the terminal generates a task processing request for a plurality of tasks to be processed in response to a task processing operation for a plurality of tasks to be processed.

[0096] In an embodiment of the present application, the data input by the user can be encapsulated into the task processing request. For example, in the display interface of the task processing application, the acquisition path of the task data set is displayed, and the user can select or confirm in the display interface according to actual needs, and then the acquisition path of the task data set input by the user can be encapsulated into the task processing request.

[0097] Step S203: the terminal sends task processing requests for the multiple tasks to be processed to the server.

[0098] Step S204: In response to the task processing request for the plurality of tasks to be processed, the server determines a target first working process carrying a first lock identifier from the first working process set.

[0099] In an embodiment of the present application, in response to a task processing request for multiple tasks to be processed, a target first work process carrying a first lock identifier is determined from a set of first work processes. The specific meaning and implementation methods of this step can be found in the description of step S101 above and will not be repeated here.

[0100] In some embodiments, see Figure 5 , Figure 5 It shows that before the server determines the target first working process carrying the first lock identifier of the data download lock from the first working process set, the following steps S301 to S303 are also included:

[0101] Step S301, when a task processing request is received, a data download lock is allocated to a work process in a first work process set.

[0102] In the embodiment of the present application, the process of allocating a data download lock to a work process in the first work process set is actually the process of multiple work processes in the first work process set trying to obtain the data download lock. Each work process in the first work process set will try to obtain the data download lock, but due to the exclusivity of the data download lock, only one work process can successfully obtain the data download lock, that is, only one of the work processes in the first work process set is allocated a data download lock, and the other work processes must wait until the data download lock is released.

[0103] For example, assume that there are five worker processes in the first worker process set, named Worker1, Worker2, Worker3, Worker4, and Worker5. In response to task processing requests for multiple pending tasks, the five worker processes may simultaneously attempt to acquire the data download lock. When Worker2 successfully acquires the data download lock, the data download lock is allocated to Worker2, and the other worker processes will wait until Worker2 releases the data download lock before attempting to acquire the data download lock to perform the data download operation for the task data of the new pending task.

[0104] In some embodiments, see Figure 6 , Figure 6 It is shown that in step S301, the server allocates a data download lock to a work process in the first work process set, which can be achieved by the following steps S3011 to S3014:

[0105] Step S3011, determining the first cycle of each work process in the first work process set.

[0106] In an embodiment of the present application, the first cycle refers to the process cycle of each work process in the first work process set, and the process cycle refers to the cycle of the work process to execute tasks regularly according to a predetermined schedule. The cycle length of the first cycle can be fixed or dynamically adjusted according to specific conditions. The first cycle of each work process in the first work process set can be different, and there may also be multiple work processes with the same first cycle. For example, following the example in step S301 above, Worker1 can perform a data synchronization task every 10 minutes, Worker2 can perform a data synchronization task every 15 minutes, and Worker3, Worker4 and Worker5 can perform a data download task every hour.

[0107] In some embodiments, determining the first period of each work process in the first work process set can be achieved in the following ways: Method 1, reviewing the configuration file. When the periodic task of the work process is set by the configuration file, the first period can be determined by directly checking the configuration file. For example, check the configuration settings of the Windows task planner or the application. Method 2, analyze the log file. The work process usually records its activities in the log file, including the start and end time of the task execution. By analyzing the log file, the execution mode and period of the work process can be determined. Method 3, use system detection tools (for example, Linux's top, ps commands, or professional detection software, etc.) to detect the status and timestamp of the work process in real time, and analyze these data to determine the first period. Method 4, customize the detection script, write a script to regularly check the status or output of the work process, record the relevant time information, and analyze these data to determine the first period. Method 5, use a performance analysis tool (for example, Linux's perf, Windows's performance detector) to determine the CPU and memory usage of the work process, thereby inferring the first period of the work process. The method for determining the first period is not specifically limited here.

[0108] Step S3012, obtaining a first delay time preset for each work process.

[0109] In the embodiment of the present application, the first delay time is a random time period preset on the basis of the first cycle of the work process, that is, the first delay time is a random value, which can be a positive value or a negative value. The first delay time is used to introduce a certain degree of randomness into the fixed first cycle of the work process, so that the start time point of each work process is deviated to reduce the probability of multiple work processes executing data download tasks at the same time, thereby avoiding resource conflicts or load peaks caused by multiple work processes triggering data download tasks at the same time. The first delay time is configurable, and users (for example, system administrators or developers, etc.) can adjust the first delay time according to workload and resource utilization.

[0110] Step S3013: Use the first delay time to update the first cycle to obtain the second cycle of each working process.

[0111] In the embodiment of the present application, the process of updating the first cycle is actually the process of adding the first delay time to the first cycle of each work process in the first work process set. The second cycle refers to the process cycle obtained by adding the first delay time to the first cycle of each work process, and the second cycle is equal to the sum of the first cycle and the first delay time.

[0112] For example, following the example in step S3011 above, Worker3, Worker4 and Worker5 can perform a data download task once every hour, and the first period is 1 hour. Assuming that the preset first delay times are 1 minute, 3 minutes and 5 minutes respectively. After the first period is updated with the first delay time, the second periods of Worker3, Worker4 and Worker5 are 1 hour and 1 minute, 1 hour and 3 minutes and 1 hour and 5 minutes respectively.

[0113] Step S3014: Based on the second cycle, a data download lock is allocated to the work process in the first work process set.

[0114] In the embodiment of the present application, after determining the second cycle of each work process in the first work process set, a data download lock is allocated to the work process in the first work process set according to the second cycle of each work process. At this time, the number of work processes simultaneously attempting to obtain the data download lock is less than the number of work processes before updating the first cycle.

[0115] Here, by updating the first cycle of each work process in the first work process set by a preset first delay time, the probability of multiple work processes executing data download tasks at the same time can be greatly reduced, thereby avoiding resource conflicts or load peaks caused by multiple work processes triggering data download tasks at the same time.

[0116] In some embodiments, based on the second period, a data download lock is assigned to a work process in the first work process set, which can be achieved in the following way: start timing from the start of each work process in the first work process set; in response to the timing of any work process being equal to the duration corresponding to the second period, determine any work process as the first work process, and assign a data download lock to the first work process.

[0117] That is to say, when each work process is started, a timer is started for each work process to record how long the work process has been running. When the timing of any work process reaches the duration corresponding to the second cycle, the number of work processes whose timing reaches the duration corresponding to the second cycle is at least one. When the number of work processes whose timing reaches the duration corresponding to the second cycle is one, the work process is determined as the first work process, and a data download lock is allocated to the first work process. When the number of work processes whose timing reaches the duration corresponding to the second cycle is multiple, one of the work processes can be selected as the first work process according to a preset strategy (for example, first-come, first-served, priority or polling strategies), and a data download lock is allocated to the first work process. The preset strategy can be pre-set by the user or randomly selected by the system, and there is no specific limitation on this.

[0118] Here, determining the first working process for allocating the data download lock by timing can ensure that the data download task will not conflict due to the simultaneous execution of multiple working processes, and the load of the working process and the allocation of resources can be controlled by adjusting the second cycle duration.

[0119] Step S302: when a first working process in the first working process set is assigned to a data download lock, a first lock identifier of the data download lock is associated with the first working process.

[0120] Step S303: determine the first working process associated with the first lock identifier as the target first working process.

[0121] In an embodiment of the present application, when the first work process in the first work process set is assigned to the data download lock, first, the first lock identifier of the data download lock is obtained; then, the first lock identifier is associated with the first work process assigned to the data download lock; finally, the first work process associated with the first lock identifier is determined as the target first work process, that is, the target first work process is the work process in the first work process set that carries the first lock identifier of the data download lock.

[0122] In some embodiments, the process of associating the first lock identifier of the data download lock with the first working process may be a process of establishing a mapping (e.g., a dictionary), and a dictionary may be created to store the first lock identifier and the corresponding first working process, with the first lock identifier as a key and the first working process as a value. The corresponding first working process may be determined by the first lock identifier, and the first working process may be determined as the target first working process.

[0123] Here, based on the first lock identifier, the target first working process for the data download task can be determined from the first working process set, ensuring that only the target first working process can execute the data download task, avoiding concurrent processing problems caused by multiple working processes downloading the same data file at the same time, and improving the efficiency and accuracy of data downloading.

[0124] In some embodiments, after the target first work process completes the data download task based on the data download lock, the target first work process will release the data download lock. At this time, the association between the first lock identifier and the target first work process is ended, so that the target first work process can be used to execute other tasks.

[0125] Step S205: The server obtains a task data set based on multiple tasks to be processed through the target first working process.

[0126] In the embodiment of the present application, the task data set includes the task data of each of the multiple pending tasks. For the specific meaning and implementation method of obtaining the task data set based on the multiple pending tasks through the target first work process, please refer to the description of step S102 above, which will not be repeated here.

[0127] Step S206: For each of the multiple tasks to be processed, the server determines a target second working process carrying a second lock identifier from the set of second working processes.

[0128] In an embodiment of the present application, for each pending task among multiple pending tasks, a target second work process carrying a second lock identifier is determined from a set of second work processes. The specific meaning and implementation methods of this step can be found in the description of step S103 above and will not be repeated here.

[0129] In some embodiments, see Figure 7 , Figure 7 The process of determining the second work process set is shown. The second work process set can be determined by the following steps S401 to S403:

[0130] Step S401, obtaining the number of first working processes in the first working process set, the number of logical concurrencies of the task processing system, and the number of replicas of the task processing system.

[0131] In the embodiment of the present application, the number of first working processes in the first working process set refers to the number of processes of all first working processes in the first working process set. The number of first working processes can be obtained by viewing the system configuration file, querying the database or using the system detection tool. The logical concurrency of the task processing system refers to the maximum number of tasks that can be processed simultaneously in the task processing system. The logical concurrency is usually determined by the resource limitations of the task processing system (for example, the number of CPU cores and the memory size, etc.) and the nature of the task (for example, I / O intensive or CPU intensive). The logical concurrency can be the number of physical cores or the number of concurrent threads allowed by the task processing system. In a multi-threaded or multi-process environment, the logical concurrency can be much larger than the number of physical cores. The logical concurrency can be obtained by commands provided by the task processing system (for example, top, ps, tasklist and other commands), and can also be queried by libraries provided by programming languages. The number of copies of the task processing system refers to the number of replicated instances of the task processing system. In a distributed system, copies are used for load balancing and fault tolerance to ensure the high availability of the system. Copies can be multiple instances of the same task processing system, which may be deployed on different servers to distribute the load and provide redundancy. The number of replicas is specified in the configuration files of the task processing system. You can obtain information by viewing these configuration files, or by querying them through the application programming interface (API) or command line tools.

[0132] Step S402, determining the number of second working processes based on the number of tasks to be processed, the number of first working processes, the number of logical concurrency and the number of copies.

[0133] In the embodiment of the present application, the number of the second work process can be determined according to the number of tasks to be processed, the number of the first work process, the number of logical concurrency and the number of copies. According to the actual number of tasks to be processed, the number of the second work process is dynamically adjusted. The number of the second work process may be equal to the number of the first work process, or may not be equal to the number of the first work process.

[0134] In some embodiments, determining the number of second work processes based on the number of tasks to be processed, the number of first work processes, the logical concurrency and the number of copies can be achieved in the following way: determining the product of the number of copies, the number of first work processes and the logical concurrency as the actual concurrency of the task processing system at the current moment; obtaining the call time and preset adjustment parameters of the target task in the task processing system; determining the number of query requests of the task processing system at the current moment based on the actual concurrency, the call time and the preset adjustment parameters; determining the number of second work processes based on the number of query requests.

[0135] That is to say, first, by multiplying the number of copies, the number of first working processes and the number of logical concurrency, the actual number of concurrency of the task processing system at the current moment is calculated. The actual number of concurrency is used to represent the number of tasks that the task processing system can theoretically process simultaneously. Next, the call time of the target task in the task processing system is obtained. The call time refers to the time required for each target task from receiving the task processing request to completing the task processing, that is, the time required to process a target task. At the same time, it is also necessary to obtain a preset adjustment parameter, which is the number of seconds that the working process rests before waiting for the next task after processing a task. The preset adjustment parameter can be used to control the working rhythm of the working process to prevent the working process from over-using resources or generating too much load. The number of query requests of the task processing system at the current moment refers to the number of query requests that the task processing system can process per second at the current moment, that is, the total number of tasks. The number of query requests can be obtained by dividing the actual number of concurrency by the product between the call time and the preset adjustment parameter. Finally, according to the number of query requests, the number of second working processes can be determined. This step needs to take into account the resource limitations and task processing capabilities of the task processing system to ensure that the number of second work processes can meet the needs of task processing without over-consuming system resources. If the number of query requests exceeds the current system processing capacity, the number of second work processes is increased. If the system resources are sufficient but the number of query requests is small, the number of second work processes is not increased, or the number of second work processes can be reduced to save resources.

[0136] Here, by comprehensively considering factors such as the number of first working processes, the number of logical concurrency, the number of copies, the calling time of the target task and preset adjustment parameters, the number of second working processes can be dynamically adjusted to maintain the optimal performance and efficiency of the task processing system.

[0137] Step S403: Generate a second work process set according to the number of second work processes.

[0138] In an embodiment of the present application, after determining the number of second work processes, the second work processes are configured and initialized according to the number of second work processes, necessary parameters and resources are configured for each second work process, and finally a second work process set including multiple second work processes is created.

[0139] Here, determining the number of second working processes is a dynamic adjustment process, and the number of second working processes can be adjusted according to real-time performance detection data of the task processing system to maintain the optimal performance and efficiency of the task processing system.

[0140] In some embodiments, see Figure 8 , Figure 8 It is shown that for each task to be processed, before the server determines the target second working process carrying the second lock identifier from the second working process set, the following steps S501 to S503 are also included:

[0141] Step S501: assigning a task processing lock to a work process in the second work process set.

[0142] In the embodiment of the present application, for each task to be processed, the process of allocating a task processing lock to a work process in the second work process set is actually the process of multiple work processes in the second work process set trying to obtain the task processing lock. Each work process in the second work process set will try to obtain the task processing lock, but due to the exclusivity of the task processing lock, only one work process can successfully obtain the task processing lock, that is, only one of the work processes in the second work process set is allocated a task processing lock, and the other work processes must wait until the task processing lock is released.

[0143] For example, suppose there are five worker processes in the second worker process set, named Worker1, Worker2, Worker3, Worker4, and Worker5. For each pending task, the five worker processes can simultaneously attempt to acquire the task processing lock. When Worker3 successfully acquires the task processing lock, the task processing lock is assigned to Worker3, and the other worker processes will wait until Worker3 releases the task processing lock before attempting to acquire the task processing lock to perform the task processing operation for the new pending task.

[0144] In some embodiments, see Fig. 9 , Fig. 9 It is shown that in step S501, the server allocates a task processing lock to a work process in the second work process set, which can be implemented by the following steps S5011 to S5014:

[0145] Step S5011, determining the third period of each work process in the second work process set.

[0146] In an embodiment of the present application, the third cycle refers to the process cycle of each work process in the second work process set, and the process cycle refers to the cycle of the work process to execute tasks regularly according to a predetermined schedule. The cycle length of the third cycle can be fixed or dynamically adjusted according to specific conditions. The third cycle of each work process in the second work process set can be different, and there may also be multiple work processes with the same third cycle. For example, following the example in step S501 above, Worker1 can perform a data synchronization task every 5 minutes, Worker2 can perform a data synchronization task every 10 minutes, and Worker3, Worker4 and Worker5 can perform a task processing operation every 20 minutes.

[0147] In some embodiments, determining the third period of each work process in the second work process set can be achieved in the following ways: Method 1, reviewing the configuration file. When the periodic task of the work process is set by the configuration file, the third period can be determined by directly checking the configuration file. For example, check the configuration settings of the Windows task planner or the application. Method 2, analyze the log file. The work process usually records its activities in the log file, including the start and end time of the task execution. By analyzing the log file, the execution mode and period of the work process can be determined. Method 3, use system detection tools (for example, Linux's top, ps commands, or professional detection software, etc.) to detect the status and timestamp of the work process in real time, and analyze these data to determine the third period. Method 4, customize the detection script, write a script to regularly check the status or output of the work process, record the relevant time information, and analyze these data to determine the third period. Method 5, use performance analysis tools (for example, Linux's perf, Windows's performance detector) to determine the CPU and memory usage of the work process, thereby inferring the third period of the work process. The method for determining the third period is not specifically limited here.

[0148] Step S5012, obtaining a second delay time preset for each work process.

[0149] In the embodiment of the present application, the second delay time is a random time period preset on the basis of the third period of the work process, that is, the second delay time is a random value, which can be a positive value or a negative value. The second delay time is used to introduce a certain randomness into the fixed third period of the work process, so that the start time point of each work process is deviated to reduce the probability of multiple work processes performing task processing operations at the same time, thereby avoiding resource conflicts or load peaks caused by multiple work processes triggering task processing operations at the same time. The second delay time is configurable, and users (for example, system administrators or developers, etc.) can adjust the second delay time according to workload and resource utilization.

[0150] Step S5013: Use the second delay time to update the third period to obtain the fourth period of each working process.

[0151] In the embodiment of the present application, the process of updating the third period is actually the process of adding the second delay time to the third period of each work process in the second work process set. The fourth period refers to the process period obtained by adding the second delay time to the third period of each work process, and the fourth period is equal to the sum of the third period and the second delay time.

[0152] For example, following the example in step S5011 above, Worker3, Worker4 and Worker5 can perform a task processing operation once every 20 minutes, and the third period is 20 minutes. Assuming that the preset second delay time is 1 minute, 2 minutes and 3 minutes respectively. After the third period is updated with the second delay time, the fourth periods of Worker3, Worker4 and Worker5 are 21 minutes, 22 minutes and 23 minutes respectively.

[0153] Step S5014: Based on the fourth cycle, a task processing lock is allocated to the work process in the second work process set.

[0154] In the embodiment of the present application, after determining the fourth cycle of each work process in the second work process set, a task processing lock is allocated to the work process in the second work process set according to the fourth cycle of each work process. At this time, the number of work processes simultaneously attempting to obtain the task processing lock is less than the number of work processes before updating the third cycle.

[0155] Here, by updating the third period of each work process in the second work process set by the preset second delay time, the probability of multiple work processes executing task processing operations at the same time can be greatly reduced, thereby avoiding resource conflicts or load peaks caused by multiple work processes triggering task processing operations at the same time.

[0156] In some embodiments, before allocating the task processing lock to the work process in the second work process set, a first pending task list may be created. The first pending task list includes pending tasks in an unprocessed state among the plurality of pending tasks.

[0157] In some embodiments, the task status of the pending task includes an unprocessed state and a processed state. For each pending task, before the task processing lock is assigned to the work process in the second work process set, the task status of the pending task is an unprocessed state, that is, the pending task is in an unprocessed state; after the task processing lock is assigned to the work process in the second work process set, the task status of the pending task is updated from the unprocessed state to the processed state, and at this time, the pending task is in the processed state.

[0158] Here, by updating the task status of the pending task, it is possible to intuitively determine whether the first pending task is assigned to the work process that has obtained the task processing lock, and quickly assign the work process that has obtained the task processing lock to the pending task whose task status is unprocessed.

[0159] In some embodiments, a first list of pending tasks can be created based on a task data set, the task data in the task data set is traversed, the task status of the pending tasks corresponding to the task data is detected, and the pending tasks in an unprocessed state among multiple pending tasks are combined into a first list of pending tasks.

[0160] In some embodiments, the created first to-be-processed task list may be stored in a storage space. For example, the storage space may be a database, a distributed cache system, or a shared file storage service (eg, a CFS cloud disk).

[0161] In some embodiments, see Fig.10 , Fig.10 It is shown that in step S501, the server allocates a task processing lock to the work process in the second work process set, which can also be implemented by the following steps S5015 to S5019:

[0162] Step S5015: Obtain the first task to be processed from the first task list to be processed.

[0163] In the embodiment of the present application, the server may randomly obtain a task to be processed from the first list of tasks to be processed, and determine the task to be processed as the first task to be processed.

[0164] Step S5016: Detect the task status of the first task to be processed.

[0165] In an embodiment of the present application, the task status of a randomly acquired first task to be processed is detected to obtain a detection result of the first task to be processed, and the detection result is used to characterize that the first task to be processed is still in an unprocessed state, or that the first task to be processed is in a processed state, that is, the task status of the first task to be processed has been updated from an unprocessed state to a processed state.

[0166] In some embodiments, the task status of the first task to be processed can be periodically detected through a preset detection module, a timed task or a polling mechanism. For example, the preset detection module can be set or written by the user, or it can be a detection module in the storage space for storing the first list of tasks to be processed, which is not specifically limited here.

[0167] Step S5017: When the detection result indicates that the first to-be-processed task is still in an unprocessed state, a task processing lock is allocated to a work process in the second work process set for the first to-be-processed task.

[0168] In an embodiment of the present application, when the detection result indicates that the first pending task is still in an unprocessed state, it means that during the acquisition and task state detection process for the first pending task, the task state of the first pending task has not changed. That is, for the first pending task, the task processing lock has not been allocated to the work process in the second work process set. At this time, the work process in the second work process set can try to obtain the task processing lock for the first pending task, and when the work process in the second work process set obtains the task processing lock, the process of allocating the task processing lock to the work process in the second work process set is implemented.

[0169] In some embodiments, after allocating a task processing lock to a work process in the second work process set for the first task to be processed, the task state of the first task to be processed may also be updated from an unprocessed state to a processed state.

[0170] Here, according to the task status of the first task to be processed, it can be quickly and accurately determined whether to allocate a task processing lock to a work process in the second work process set for the first task to be processed.

[0171] Step S5018: When the detection result indicates that the first task to be processed is in a processing state, the first task to be processed is deleted from the first task list to be processed, and a second task list to be processed is obtained.

[0172] In an embodiment of the present application, when the detection result indicates that the first pending task is in a processing state, it means that during the acquisition and task state detection process for the first pending task, the task state of the first pending task has changed. That is, for the first pending task, a task processing lock has been assigned to the work process in the second work process set. At this time, the first pending task is deleted from the first pending task list to obtain a second pending task list, so that the first pending task is not included in the second pending task list. In other words, if it is detected that there is a pending task with a task state of processing in the pending task list, the pending task with a task state of processing is deleted from the pending task list to obtain a new pending task list, i.e., the second pending task list, so that there is no pending task with a task state of processing in the new pending task list as much as possible.

[0173] Step S5019: Obtain the second pending task in an unprocessed state from the second pending task list, and allocate a task processing lock to the work process in the second work process set for the second pending task.

[0174] In an embodiment of the present application, a pending task is randomly obtained from the second pending task list, and the task status of the pending task is continuously detected. If the detection result indicates that there is a second pending task in an unprocessed state in the second pending task list, the second pending task is obtained from the second pending task list. The work process in the second work process set can attempt to obtain a task processing lock for the second pending task. When the work process in the second work process set obtains the task processing lock, the process of allocating the task processing lock to the work process in the second work process set is implemented.

[0175] Here, by randomly obtaining one of the pending tasks in an unprocessed state from the pending list, and allocating a task processing lock to the work process in the second work process set for the pending task, it can be ensured that only the pending tasks in an unprocessed state are allocated a task processing lock to each work process in the second work process set, thereby avoiding conflicts caused by multiple work processes executing the same pending task at the same time.

[0176] In some embodiments, based on the fourth period, a task processing lock is assigned to a work process in the second work process set, which can be achieved in the following manner: start timing from the time each work process in the second work process set is started; in response to the timing of any work process being equal to the duration corresponding to the fourth period, determine any work process as a second work process, and assign a task processing lock to the second work process.

[0177] That is to say, when each work process in the second work process is started, a timer is started for each work process to record how long the work process has been running. When the timing of any work process reaches the duration corresponding to the fourth cycle, the number of work processes whose timing reaches the duration corresponding to the fourth cycle is at least one. When the number of work processes whose timing reaches the duration corresponding to the fourth cycle is one, the work process is determined as the second work process, and a task processing lock is allocated to the second work process. When the number of work processes whose timing reaches the duration corresponding to the fourth cycle is multiple, one of the work processes can be selected according to a preset strategy (for example, first-come, first-served, priority or polling strategies) to be determined as the second work process, and a task processing lock is allocated to the second work process. The preset strategy can be pre-set by the user or randomly selected by the system, and is not specifically limited to this.

[0178] Here, determining the second working process for allocating the task processing lock by timing can ensure that the task processing operation will not conflict due to the simultaneous execution of multiple working processes, and the load of the working process and the allocation of resources can be controlled by adjusting the fourth cycle duration.

[0179] Step S502: When a second work process in the second work process set is assigned to a task processing lock, a second lock identifier of the task processing lock is associated with the second work process.

[0180] Step S503: determine the second working process associated with the second lock identifier as the target second working process.

[0181] In an embodiment of the present application, when the second work process in the second work process set is assigned to the task processing lock, first, the second lock identifier of the task processing lock is obtained; then, the second lock identifier is associated with the second work process assigned to the task processing lock; finally, the second work process associated with the second lock identifier is determined as the target second work process, that is, the target second work process is the work process in the second work process set that carries the second lock identifier of the task processing lock.

[0182] In some embodiments, the process of associating the second lock identifier of the task processing lock with the second work process may be a process of establishing a mapping (e.g., a dictionary), and a dictionary may be created to store the second lock identifier and the corresponding second work process, with the second lock identifier as a key and the second work process as a value. The corresponding second work process may be determined by the second lock identifier, and the second work process may be determined as the target second work process.

[0183] Here, based on the second lock identifier, the target second work process used for the task processing operation can be determined from the second work process set, ensuring that only the target second work process can perform the task processing operation, avoiding the concurrent processing problem caused by multiple work processes simultaneously processing the task data of the same task to be processed, thereby ensuring the accuracy and stability of the task data processing process and improving the efficiency of task processing.

[0184] In some embodiments, after the target second work process completes the task processing operation based on the task processing lock, the target second work process releases the task processing lock. At this time, the association between the second lock identifier and the target second work process is ended, so that the target second work process can be used to execute other tasks.

[0185] Step S207: The server processes the task data of the task to be processed through the target second working process to obtain the task processing result.

[0186] In the embodiment of the present application, for processing the task data of the task to be processed through the target second work process to obtain the task processing result, the specific meaning explanation and specific implementation method involved in this step can be found in the description of step S104 above, and will not be repeated here.

[0187] Step S208: the server sends the task processing result to the terminal.

[0188] Step S209: the terminal outputs the task processing result.

[0189] In an embodiment of the present application, when performing task processing, a target first work process carrying a first lock identifier of a data download lock is determined from a first work process set, and a task data set is obtained through the target first work process. In this way, when obtaining a task data set, only the target first work process carrying a first lock identifier of a data download lock in the first work process set is allowed to obtain the task data set, and other work processes other than the target first work process in the first work process set are not allowed to obtain the task data set. In addition, after obtaining the task data set, for each of a plurality of pending tasks, a target second work process carrying a second lock identifier of a task processing lock is determined from a plurality of work processes, and task processing is performed on the pending task through the target second work process. In this way, when performing task processing on each pending task, only the target second work process carrying a second lock identifier of a task processing lock in the second work process set is allowed to perform task processing on the pending task, and other work processes other than the target second work process in the second work process set are not allowed to perform task processing on the pending task. In summary, based on the data download lock and the task processing lock, the problems of resource scheduling and resource preemption caused by the concurrent processing of multiple tasks to be processed can be effectively solved.

[0190] The present application embodiment can also be applied to task interruption recovery scenarios, see Fig.11 , Fig.11 A task interruption recovery method provided by an embodiment of the present application is shown, and the method can be implemented by following steps S601 to S603:

[0191] Step S601: In response to an interruption in the task processing process, the server records the current task processing progress and the task interruption identifier of the task to be processed into a progress list.

[0192] In the embodiment of the present application, the current task processing progress refers to the proportion or status of the part of the pending task that has been completed at the current moment to the entire pending task. For example, the current task processing progress can be waiting for resources, task completion, task failure, or task interruption. The task interruption flag is a sign or signal, and the task interruption flag is used to indicate whether the pending task is interrupted during normal execution. When an interruption is detected in the task processing process of the pending task, the current task processing progress and task interruption flag of the pending task are obtained, and the current task processing progress and task interruption flag are recorded in the progress list.

[0193] In some embodiments, the progress list can be determined in the following manner: creating a progress list for the pending tasks in the first pending task list; in the process of calling the target second working process to process the pending tasks, recording the task processing progress of the pending tasks in the progress list according to the progress status change time of the pending tasks.

[0194] In the embodiment of the present application, a progress list is created for each pending task in the first pending task list, and the progress list can be used to store the task processing progress of the pending task. In the process of calling the target second work process to process the pending task, the task processing progress of the pending task will change. Every time the progress status of the pending task changes, there is a specific time point corresponding to the task processing progress. The time point can be the duration since the start of the pending task, or it can be an absolute timestamp.

[0195] Here, the time point and the corresponding task processing progress can be recorded in the progress list, so as to facilitate the subsequent rapid acquisition of the task processing progress of the task to be processed from the progress list.

[0196] Step S602, in response to the task processing process being restored from an interrupted state to a normal processing state, the server obtains the current task processing progress of the to-be-processed task from a progress list based on the task interruption flag.

[0197] In an embodiment of the present application, when the task processing process of the pending task is restored from an interrupted state to a normal processing state, the task interruption identifier when the task processing process is interrupted is obtained from the progress list, and based on the correspondence between the task interruption identifier and the current task processing progress, the current task processing progress of the pending task when the task processing process is interrupted is obtained from the progress list.

[0198] Step S603: The server continues to process the task data of the task to be processed based on the current task processing progress.

[0199] In the embodiment of the present application, when the current task processing progress of the task to be processed is obtained when the task processing process is interrupted, task processing of the task to be processed continues based on the task data of the task to be processed.

[0200] Here, by recording the task processing progress of the pending tasks in the progress list, when the task processing process is interrupted, the task progress at the time of interruption can be quickly recorded in the progress list, and when the task processing process is interrupted and resumed, the task processing process can be quickly and effectively resumed according to the task progress at the time of interruption.

[0201] In some embodiments, for a task that has timed out and has not been completed among multiple tasks to be processed, see Fig.12 The following steps S701 to S703 can be used to clean up pending tasks that have timed out and have not been completed:

[0202] Step S701, obtaining the task execution period of the task to be processed.

[0203] In an embodiment of the present application, the task execution cycle of a pending task refers to the time period required for the pending task to be completed from the start of execution to the expected completion. The task execution cycle of the pending task can be obtained from the task configuration. Each pending task has a related task configuration including a task execution cycle when it is created, and the task execution cycle of the pending task can be read directly from the task configuration. The task execution cycle of the pending task can also be obtained by querying a database. The task information of each pending task can be stored in a database, and the task execution cycle of the pending task can be obtained by querying the database. The method for obtaining the task execution cycle of the pending task is not specifically limited here.

[0204] Step S702, when a task processing request is received, the task execution time of the task to be processed begins to be counted.

[0205] In the embodiment of the present application, the task execution time refers to the time consumed from the start of execution of the pending task to the current moment. The task execution time can be calculated by recording the difference between the timestamp of the start of the pending task and the current timestamp. When a task processing request for the pending task is received, the task execution time of the pending task is started by a timer.

[0206] Step S703, in response to the task execution time being greater than the time corresponding to the task execution cycle, and the current task processing progress of the task to be processed is in an unfinished state, deleting the task to be processed from the first task to be processed list.

[0207] In the embodiment of the present application, the task execution duration of the pending task is compared with the duration corresponding to the task execution cycle, and the current task processing progress of the pending task is detected. The current task processing progress of the pending task can be a completed state or an uncompleted state. When the task execution duration is greater than the duration corresponding to the task execution cycle, and the current task processing progress of the pending task is an uncompleted state, it means that the pending task is a task that has timed out and has not been completed. At this time, the pending task is deleted from the first pending task list.

[0208] Here, by comparing the task execution time of the pending tasks and the time corresponding to the task execution cycle, the pending tasks that have timed out and not been completed among multiple pending tasks can be quickly determined, and the pending tasks that have timed out and not been completed can be quickly cleared, effectively avoiding waste of resources, thereby improving the overall efficiency of the task processing system.

[0209] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0210] The embodiment of the present application provides a high-performance processing guarantee engine for massive data based on scalable concurrency, multi-task concurrent scheduling control, and task interruption recovery mechanism, which is used to solve the problems of low order of magnitude of batch running tasks, low concurrency, and performance depending on a single machine in the related technology. The overall flow diagram of the embodiment of the present application is as follows Fig.13 As shown, the data source file (i.e., the above-mentioned task data set) is obtained from the Hadoop Distributed File System (HDFS) cluster 131, and the data source file is traversed line by line and the task is executed in the batch tool 132. During the task execution process, a call to the downstream service 133 is initiated for each task, thereby achieving offline batch running.

[0211] The complete design execution flow diagram of this solution is as follows Fig.14As shown, in the cloud computing platform, the data download task is automatically triggered by a preset timer job to obtain the data source file from the HDFS cluster 131. The work processes in multiple copies compete for exclusive locks to obtain the task execution right. Once a work process successfully obtains the lock, the work process will exclusively occupy the data source file for traversal processing. After the traversal is completed, the work process will execute the relevant tasks. After the task is successfully executed, the work process will record the progress cursor of the task so that it can resume execution from the last progress point after the task is interrupted.

[0212] First, trigger the batch task through the application programming interface (API) or user interface (UI). Fig.15 As shown, in the UI management console, users can create new tasks or execute manually to trigger new batch running tasks. In addition, new batch running tasks can be triggered by creating new tasks or executing manually in the UI management console.

[0213] Secondly, the task enters the dispatcher 142 (dispatcher) in the process logic. After the user triggers the batch task, the task will be sent to the dispatcher 142 in the process logic. The dispatcher 142 is mainly composed of minute-level scheduled tasks. Through minute-level scheduled tasks, data can be polled every minute to pull data and assign tasks to the working process. Among them, the data capture module 143 (fetcher) is responsible for pulling new data that needs to be batch-run from the HDFS cluster 131. The data is downloaded through the working process, and the tasks are executed in rotation, that is, after the current task is completed, the next task will be executed.

[0214] Then, the data capture module 143 follows the download lock mechanism of the scheduler 142 when pulling data. Before downloading data from HDFS, the data capture module must first obtain a data download lock 144 to avoid repeated downloading of the same data by other tasks. This process is called "grabbing the lock before downloading". Therefore, when the data capture module 143 pulls new data that needs to be batch-run from the HDFS cluster 131, the data is downloaded while the data download lock is obtained.

[0215] Next, the task list is managed by the scheduler 142. In this step, the scheduler 142 will add the new task to the task list through the task list module 145 to update the task list, and perform a "pre-update lock grabbing" operation before updating to ensure data consistency. After completing the lock grabbing of the task processing lock 146, the scheduler 142 adds the newly acquired data to the task list, which is responsible for maintaining the batch running tasks of all tasks to be executed.

[0216] Then, the specific task is triggered to execute through the scheduler 142. The tasks corresponding to the data source file are split, and the split specific tasks are respectively assigned to each work process, and the progress status of each specific task is updated. Each work process will store the progress information of each specific task according to its processing situation. The scheduler 142 will call the background business remote procedure call (RPC) interface corresponding to each specific task to perform the actual call operation.

[0217] Finally, the clearing plan module 147 (cleanSchedule) is used to regularly clear the tasks that have not been executed and record the updated status. The tasks that have not been executed and have expired are marked for clearing and the status is recorded in the task list.

[0218] Based on the above scheme, the following functional components as shown in Table 1 are designed.

[0219] Table 1 Functional components

[0220]

[0221] The complete design and execution process of this solution also involves the following types of data storage as shown in Table 2.

[0222] Table 2 Data storage format

[0223]

[0224]

[0225]

[0226] According to the content in Table 2, based on the considerations of lightness, convenience and low cost, this solution directly uses the cloud file storage CFS cloud disk as a unified data storage, taking into account the storage of source data files and the storage of status data during operation. Since a global lock is required for concurrent preemption (low-frequency writing), a cloud key value service (Cloud KeyValue, CKV) (for example, Redis) is introduced as a global exclusive lock to solve the data storage problem of batch running tasks.

[0227] In this solution, only two scenarios require locking operations. When downloading data source files, only one worker process needs to be arranged for serial processing to realize the download of data source files. The worker process needs to grab a download lock first. After the lock is successfully grabbed, the worker process will download all data source files to CFS.

[0228] When updating the preemption status and requiring multiple worker processes to simultaneously preempt the task data corresponding to each specific task in the data source file, it is necessary to first control one of the multiple worker processes to obtain the preemption lock to prevent multiple worker processes from preempting the same task data at the same time.

[0229] In the above two scenarios, the work process is driven by the scheduled task to achieve task data preemption. Usually, if all work processes are started at the same time, the first preemption cycle of each work process is the same. At this time, the preemption cycle of each work process can be randomized to reduce lock conflicts.

[0230] When preempting each task data, a list of all unpreempted task data can be obtained. The worker process can preempt by randomly selecting an unpreempted task data in the task data list. If a task data in the task data list has been preempted, the worker process will randomly select again until it successfully preempts an unpreempted task data to preempt, thereby reducing the conflict of multiple worker processes preempting the same task data at the same time.

[0231] The core goal of this solution is to make the number of concurrent connections controllable. The actual number of concurrent connections can be controlled by adjusting the task parameters and the number of working resources. Therefore, the following parameters are designed as shown in Table 3 to control the actual number of concurrent connections.

[0232] Table 3 Adjustable parameters for controlling the actual number of concurrent connections

[0233]

[0234]

[0235] Therefore, the theoretical calculation formula for the actual number of concurrent requests is: actual number of concurrent requests = R × P × C, and the calculation formula for the number of requests processed per second (Query Per Second, QPS) (i.e. the number of query requests mentioned above) is: QPS = R × P × C / (T (i.e. the call time of the downstream task) + S). From the above formula, it can be seen that unlimited concurrency can be obtained by increasing resources. At the same time, the actual QPS can be reduced by increasing the interval sleep seconds (i.e. the above preset adjustment parameter).

[0236] The method provided in the embodiment of the present application is based on the considerations of light weight, convenience and low cost. It directly uses the CFS cloud disk as a unified data storage, taking into account the storage of source data files and the storage of status data during operation. Since a global lock (low-frequency write) is required for concurrent preemption, a CKV (for example, Redis) is introduced as a global exclusive lock to solve the data storage problem of batch tasks. By setting two exclusive locks, one for downloading data source files and the other for updating the preemption status of tasks, the resource scheduling and preemption problems caused by multi-task concurrency are solved. In addition, by adjusting the task parameters and the number of working resources, the actual number of concurrencies can be flexibly controlled to achieve unlimited expansion of the number of concurrencies, and the overall scheduling management can be completed through carefully designed logic.

[0237] It can be understood that in the embodiments of the present application, the content involving user information, such as information such as tasks to be processed, task data sets and task data, if it involves data related to user information or corporate information, when the embodiments of the present application are applied to specific products or technologies, it is necessary to obtain user permission or consent, or to blur this information to eliminate the correspondence between this information and the user; and the relevant data collection and processing should be strictly in accordance with the requirements of relevant national laws and regulations when applied in examples, obtain the informed consent or separate consent of the subject of personal information, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the subject of personal information.

[0238] The following is a description of an exemplary structure of the task processing device 455 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules stored in the task processing device 455 of the memory 450 may include: a first determination module 4551, used to determine, in response to a task processing request for multiple tasks to be processed, a target first work process carrying a first lock identifier from a first work process set, wherein the first lock identifier indicates a data download lock used to provide data download permission; an acquisition module 4552, used to obtain a task data set based on the multiple tasks to be processed through the target first work process, wherein the task data set includes task data respectively indicated by the multiple tasks to be processed; a second determination module 4553, used to determine, for each of the multiple tasks to be processed, a target second work process carrying a second lock identifier from a second work process set, wherein the second lock identifier indicates a task processing lock used to provide task processing permission; a task processing module 4554, used to process the task data of the tasks to be processed through the target second work process.

[0239] In some embodiments, the device 455 also includes a third determination module, which is used to: assign the data download lock to the work process in the first work process set; when the first work process in the first work process set is assigned to the data download lock, associate the first lock identifier of the data download lock with the first work process; and determine the first work process associated with the first lock identifier as the target first work process.

[0240] In some embodiments, the third determination module is also used to: determine the first cycle of each work process in the first work process set; obtain the first delay time preset for each of the work processes; use the first delay time to update the first cycle to obtain the second cycle of each of the work processes; based on the second cycle, allocate the data download lock to the work process in the first work process set.

[0241] In some embodiments, the third determination module is also used to: start timing from the time each work process in the first work process set is started; in response to the timing of any of the work processes being equal to the duration corresponding to the second period, determine any of the work processes as the first work process, and assign the data download lock to the first work process.

[0242] In some embodiments, the device 455 also includes a fourth determination module, which is used to: assign the task processing lock to the work process in the second work process set; when the second work process in the second work process set is assigned to the task processing lock, associate the second lock identifier of the task processing lock with the second work process; and determine the second work process associated with the second lock identifier as the target second work process.

[0243] In some embodiments, the fourth determination module is also used to: determine the third period of each work process in the second work process set; obtain the second delay time preset for each of the work processes; use the second delay time to update the third period to obtain the fourth period of each of the work processes; based on the fourth period, allocate the task processing lock to the work processes in the second work process set.

[0244] In some embodiments, the fourth determination module is also used to: start timing from the time each work process in the second work process set is started; in response to the timing of any of the work processes being equal to the duration corresponding to the fourth period, determine any of the work processes as the second work process, and allocate the task processing lock to the second work process.

[0245] In some embodiments, the device 455 further includes a creation module, and the creation module is used to: create a first list of pending tasks; the first list of pending tasks includes pending tasks in an unprocessed state among the multiple pending tasks.

[0246] In some embodiments, the fourth determination module is also used to: obtain a first pending task from the first pending task list; detect the task status of the first pending task; and when the detection result indicates that the first pending task is still in the unprocessed state, allocate the task processing lock to the work process in the second work process set for the first pending task.

[0247] In some embodiments, the device 455 further includes an updating module, and the updating module is used to update the task status of the first to-be-processed task from the unprocessed status to the processed status.

[0248] In some embodiments, the device 455 also includes an allocation module, which is used to: when the detection result indicates that the first pending task is in a processing state, delete the first pending task from the first pending task list to obtain a second pending task list; obtain the second pending task in the unprocessed state from the second pending task list, and for the second pending task, allocate the task processing lock to the work process in the second work process set.

[0249] In some embodiments, the device 455 also includes a recording module, which is used to: create a progress list for the pending tasks in the first pending task list; in the process of processing the task data of the pending tasks through the target second working process, record the task processing progress of the pending tasks into the progress list according to the progress status change time of the pending tasks.

[0250] In some embodiments, the device 455 also includes an interruption recovery module, which is used to: in response to an interruption in the task processing process, record the current task processing progress and task interruption identifier of the task to be processed into the progress list; in response to the task processing process recovering from an interrupted state to a normal processing state, based on the task interruption identifier, obtain the current task processing progress of the task to be processed from the progress list; based on the current task processing progress, continue to process the task data of the task to be processed.

[0251] In some embodiments, the device 455 also includes a deletion module, which is used to: obtain the task execution cycle of the pending task; when receiving the task processing request, start timing the task execution duration of the pending task; in response to the task execution duration being greater than the duration corresponding to the task execution cycle, and the current task processing progress of the pending task being in an unfinished state, delete the pending task from the first pending task list.

[0252] In some embodiments, the device 455 also includes a generation module, which is used to: obtain the number of first work processes in the first work process set, the logical concurrency of the task processing system, and the number of copies of the task processing system; determine the number of second work processes based on the number of tasks to be processed, the number of first work processes, the logical concurrency and the number of copies; generate the second work process set according to the number of second work processes.

[0253] In some embodiments, the generation module is also used to: determine the product of the number of copies, the number of the first work processes and the logical concurrency as the actual concurrency of the task processing system at the current moment; obtain the call time and preset adjustment parameters of the target task in the task processing system; based on the actual concurrency, the call time and the preset adjustment parameters, determine the number of query requests of the task processing system at the current moment; based on the number of query requests, determine the number of the second work processes.

[0254] It should be noted that the description of the device of the embodiment of the present application is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment, so it is not repeated. For technical details not disclosed in the embodiment of the device, please refer to the description of the method embodiment of the present application for understanding.

[0255] The present application embodiment provides a computer-readable storage medium in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the task processing method provided by the present application embodiment, for example, Figure 3 The task processing method shown.

[0256] The embodiment of the present application provides a computer program product, which includes computer executable instructions, which are stored in a computer-readable storage medium. A processor of an electronic device reads the computer executable instructions from the computer-readable storage medium, and the processor executes the computer executable instructions, so that the electronic device performs the task processing method described above in the embodiment of the present application.

[0257] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0258] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0259] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0260] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0261] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A task processing method, characterized in that: The method comprises: In response to a task processing request for a plurality of pending tasks, determining a target first working process carrying a first lock identifier from a set of first working processes, wherein the first lock identifier indicates a data download lock for providing data download permission; Obtaining a task data set based on the plurality of tasks to be processed through the target first working process, the task data set including task data respectively indicated by the plurality of tasks to be processed; For each of the plurality of tasks to be processed, determining a target second work process carrying a second lock identifier from the set of second work processes, where the second lock identifier indicates a task processing lock for providing task processing authority; The task data of the task to be processed is processed through the target second working process.

2. The method according to claim 1, characterized in that Before determining the target first working process carrying the first lock identifier from the first working process set, the method further includes: When receiving the task processing request, allocating the data download lock to the work process in the first work process set; When a first working process in the first working process set is assigned to the data download lock, associating the first lock identifier of the data download lock with the first working process; The first working process associated with the first lock identifier is determined as the target first working process.

3. The method according to claim 2, characterized in that The allocating the data download lock to the work process in the first work process set includes: Determine a first cycle of each work process in the first set of work processes; Obtaining a first delay time preset for each of the working processes; Using the first delay time to update the first cycle, to obtain a second cycle of each of the working processes; Based on the second period, the data download lock is allocated to the work process in the first work process set.

4. The method according to claim 3, characterized in that The allocating the data download lock to the work process in the first work process set based on the second cycle includes: The timing starts from when each work process in the first work process set is started; In response to the timing of any of the work processes being equal to the duration corresponding to the second period, any of the work processes is determined as the first work process, and the data download lock is allocated to the first work process.

5. The method according to any one of claims 1 to 4, characterized in that: Before determining the target second working process carrying the second lock identifier from the set of second working processes, the method further includes: Allocating the task processing lock to the work process in the second work process set; When a second work process in the second work process set is assigned to the task processing lock, associating the second lock identifier of the task processing lock with the second work process; The second working process associated with the second lock identifier is determined as the target second working process.

6. The method according to claim 5, characterized in that The allocating the task processing lock to the work process in the second work process set includes: Determining a third period of each work process in the second set of work processes; Obtaining a second delay time preset for each of the working processes; Using the second delay time to update the third period to obtain a fourth period of each working process; Based on the fourth cycle, the task processing lock is allocated to the work process in the second work process set.

7. The method according to claim 6, characterized in that The allocating the task processing lock to the work process in the second work process set based on the fourth cycle includes: The timing starts from when each work process in the second work process set is started; In response to the timing of any of the work processes being equal to the duration corresponding to the fourth period, any of the work processes is determined as the second work process, and the task processing lock is allocated to the second work process.

8. The method according to claim 5, characterized in that Before allocating the task processing lock to the work process in the second work process set, the method further includes: Creating a first pending task list; the first pending task list includes pending tasks in an unprocessed state among the plurality of pending tasks; The allocating the task processing lock to the work process in the second work process set includes: Acquire a first task to be processed from the first list of tasks to be processed; Detecting the task status of the first task to be processed; When the detection result indicates that the first to-be-processed task is still in the unprocessed state, the task processing lock is allocated to the work process in the second work process set for the first to-be-processed task.

9. The method according to claim 8, characterized in that After allocating the task processing lock to the work process in the second work process set for the first to-be-processed task, the method further includes: The task status of the first to-be-processed task is updated from the unprocessed status to the processed status.

10. The method according to claim 8 or 9, characterized in that: The method further comprises: If the detection result indicates that the first to-be-processed task is in a processing state, deleting the first to-be-processed task from the first to-be-processed task list to obtain a second to-be-processed task list; The second to-be-processed task in the unprocessed state is obtained from the second to-be-processed task list, and the task processing lock is allocated to the work process in the second work process set for the second to-be-processed task.

11. The method according to any one of claims 1 to 10, characterized in that: The method further comprises: Creating a progress list for the tasks to be processed in the first list of tasks to be processed; In the process of processing the task data of the task to be processed by the target second working process, the task processing progress of the task to be processed is recorded in the progress list according to the progress status change time of the task to be processed.

12. The method according to any one of claims 1 to 11, characterized in that: The method further comprises: In response to an interruption in the task processing process, recording the current task processing progress and the task interruption identifier of the task to be processed into the progress list; In response to the task processing process being restored from an interrupted state to a normal processing state, based on the task interruption identifier, acquiring the current task processing progress of the to-be-processed task from the progress list; Based on the current task processing progress, continue to process the task data of the task to be processed.

13. The method according to any one of claims 1 to 12, characterized in that: The method further comprises: Obtaining the task execution period of the task to be processed; When the task processing request is received, the task execution time of the task to be processed starts to be counted; In response to the task execution duration being greater than the duration corresponding to the task execution cycle, and the current task processing progress of the to-be-processed task being in an unfinished state, the to-be-processed task is deleted from the first to-be-processed task list.

14. The method according to any one of claims 1 to 13, characterized in that: After obtaining a task data set based on the plurality of tasks to be processed through the target first working process, the method further includes: Obtaining the number of first working processes in the first working process set, the logical concurrency of the task processing system, and the number of replicas of the task processing system; Determine the number of second working processes based on the number of tasks to be processed, the number of the first working processes, the number of logical concurrency and the number of replicas; Generate the second work process set according to the number of the second work processes.

15. The method according to claim 14, characterized in that The determining the number of second working processes based on the number of tasks to be processed, the number of the first working processes, the number of logical concurrency, and the number of replicas includes: The product of the number of replicas, the number of the first working processes and the logical concurrency is determined as the actual concurrency of the task processing system at the current moment; Obtaining the calling time and preset adjustment parameters of the target task in the task processing system; Determine the number of query requests of the task processing system at the current moment based on the actual number of concurrent requests, the call duration and the preset adjustment parameter; Based on the number of query requests, determine the number of the second working processes.

16. A task processing device, characterized in that: include: A first determination module, configured to determine, in response to a task processing request for a plurality of pending tasks, a target first working process carrying a first lock identifier from a first working process set, wherein the first lock identifier indicates a data download lock for providing data download permission; An acquisition module, configured to acquire a task data set based on the plurality of tasks to be processed through the target first working process, wherein the task data set includes task data respectively indicated by the plurality of tasks to be processed; A second determination module is used to determine, for each of the plurality of pending tasks, a target second work process carrying a second lock identifier from the second work process set, where the second lock identifier indicates a task processing lock for providing task processing authority; The task processing module is used to process the task data of the task to be processed through the target second working process.

17. An electronic device, characterized in that: include: A memory for storing computer executable instructions; A processor, configured to implement the task processing method according to any one of claims 1 to 15 when executing the computer executable instructions stored in the memory.

18. A computer-readable storage medium, characterized in that: Computer executable instructions are stored, and when the computer executable instructions are executed by a processor, the task processing method according to any one of claims 1 to 15 is implemented.

19. A computer program product, characterized in that The computer program product includes computer executable instructions stored in a computer readable storage medium; Wherein, when the processor of the electronic device reads the computer executable instructions from the computer readable storage medium and executes the computer executable instructions, the task processing method described in any one of claims 1 to 15 is implemented.

Images