Task processing method and apparatus

Abstract

This invention discloses a task processing method and apparatus, relating to the field of computer technology. One specific embodiment of the method includes: establishing a data structure containing multiple pluggable nodes; the pluggable nodes are components of the data structure, entering a working state based on an insertion operation and a stopped state based on a removal operation; determining multiple pluggable nodes for each of a set of multiple storage spaces, storing the task data to be executed in each storage space into the corresponding pluggable node, and performing an insertion operation on the determined pluggable nodes; allocating the pluggable nodes in the working state to multiple set of computing devices for processing; and, in response to any computing device completing the processing of an allocated pluggable node, reassigning an allocated pluggable node that has not yet been processed on another computing device to the first computing device for processing. This embodiment can balance the workload among different computing devices to improve overall task processing efficiency.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a task processing method and apparatus. Background Technology

[0002] When performing various tasks such as financial data calculation and logistics order calculation, the differences in data quality, network, configuration and other aspects between different computing devices (such as servers) result in different task execution times for different computing devices, leading to a long tail effect. That is, some computing devices complete quickly and are idle, while other computing devices take a long time to complete, affecting the overall task processing efficiency. Summary of the Invention

[0003] In view of this, embodiments of the present invention provide a task processing method and apparatus that can balance the workload among different computing devices to improve overall task processing efficiency.

[0004] To achieve the above objectives, according to one aspect of the present invention, a task processing method is provided.

[0005] The task processing method of this invention includes: establishing a data structure containing multiple pluggable nodes; the pluggable nodes are components of the data structure, entering a working state based on an insertion operation and entering a stopped state based on a removal operation; determining multiple pluggable nodes for each of a set of multiple storage spaces, storing the task data to be executed in each storage space into the corresponding pluggable node, and performing an insertion operation on the determined pluggable nodes; allocating the pluggable nodes in the working state to multiple set computing devices for processing; and in response to any computing device completing the processing of an allocated pluggable node, reassigning an allocated pluggable node that has not been processed by another computing device to any computing device for processing.

[0006] Optionally, multiple pluggable nodes are defined for each of the multiple pre-defined storage spaces, including: defining an equal number of multiple pluggable nodes for each storage space.

[0007] Optionally, each storage space stores an equal amount of task data to be executed; and storing the task data to be executed in each storage space into the corresponding pluggable node includes: dividing the task data to be executed in each storage space into equal amounts and storing it into the corresponding pluggable node, so that each pluggable node of the data structure stores an equal amount of task data to be executed.

[0008] Optionally, the pluggable nodes in operation are distributed to multiple pre-configured computing devices for processing, including: allocating an equal number of pluggable nodes in operation to each computing device.

[0009] Optionally, in response to any computing device completing the processing of an assigned pluggable node, reassigning an assigned pluggable node that has not been processed by another computing device to any computing device for processing includes: in response to any computing device completing the processing of an assigned pluggable node, determining the computing device with the largest number of assigned pluggable nodes that have not been processed, and reassigning at least one of the assigned pluggable nodes that has not been processed by that computing device to any computing device for processing.

[0010] Optionally, the task processing method further includes: in response to the completion of processing of any pluggable node, performing a pluggable node unplugging operation.

[0011] Optionally, the task processing method further includes: for source data in multiple database tables with varying amounts of data, performing hash processing on the identifier of the source data, and performing a modulo operation on the resulting hash value; storing the identifier of the source data in the storage space according to the result of the modulo operation; the task data to be executed in the storage space includes the identifier of the source data.

[0012] Optionally, the data structure is a circular linked list, a circular buffer, or a circular queue; the computing device processes tasks based on source data determined from a database table using source data identifiers stored in allocated pluggable nodes.

[0013] To achieve the above objectives, according to another aspect of the present invention, a task processing apparatus is provided.

[0014] The task processing apparatus of this invention includes: a data structure establishment unit, used to establish a data structure containing multiple pluggable nodes; the pluggable nodes are constituent elements of the data structure, entering a working state based on an insertion operation and entering a stopped state based on a removal operation; a node determination unit, used to determine multiple pluggable nodes for each of a preset multiple storage spaces, store the task data to be executed in each storage space into the corresponding pluggable node, and perform an insertion operation on the determined pluggable nodes; and a node allocation unit, used to allocate the pluggable nodes in the working state to multiple preset computing devices for processing; and in response to any computing device completing the processing of an allocated pluggable node, reassigning an allocated pluggable node that has not been processed by another computing device to any computing device for processing.

[0015] Optionally, the node determination unit is also used to: determine an equal number of multiple pluggable nodes for each storage space.

[0016] Optionally, each storage space stores an equal amount of task data to be executed; and the node determination unit is further configured to: divide the task data to be executed in each storage space into equal amounts and store them in the corresponding pluggable nodes, so that each pluggable node of the data structure stores an equal amount of task data to be executed.

[0017] Optionally, the node allocation unit is also used to allocate an equal number of pluggable nodes in operation to each computing device.

[0018] Optionally, the node allocation unit is further configured to: in response to any computing device completing the processing of an allocated pluggable node, determine the computing device with the largest number of allocated pluggable nodes that have not yet been processed, and reassign at least one of the allocated pluggable nodes of that computing device that has not yet been processed to any computing device for processing.

[0019] Optionally, the node allocation unit is further configured to: in response to the completion of processing of any pluggable node, perform a pluggable node unplugging operation.

[0020] Optionally, the task processing device further includes a data balancing unit, used for: performing hash processing on the identifier of the source data in multiple database tables with unequal data volumes, performing a modulo operation on the obtained hash value; storing the identifier of the source data in the storage space according to the result of the modulo operation; and including the identifier of the source data in the task data to be executed in the storage space.

[0021] Optionally, the data structure is a circular linked list, a circular buffer, or a circular queue; the computing device processes tasks based on source data determined from a database table using source data identifiers stored in allocated pluggable nodes.

[0022] To achieve the above objectives, according to another aspect of the present invention, an electronic device is provided.

[0023] An electronic device according to the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the task processing method provided by the present invention.

[0024] To achieve the above objectives, according to another aspect of the present invention, a computer-readable storage medium is provided.

[0025] The present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the task processing method provided by the present invention.

[0026] To achieve the above objectives, according to another aspect of the present invention, a computer program product is provided.

[0027] One computer program product of the present invention includes a computer program that, when executed by a processor, implements the task processing method provided by the present invention.

[0028] According to the technical solution of the present invention, the embodiments described above have the following advantages or beneficial effects:

[0029] First, a data structure containing multiple pluggable nodes is established. These pluggable nodes enter a working state based on insertion operations and a stopped state based on removal operations. Next, multiple pluggable nodes are determined for each of the pre-defined storage spaces. The task data to be executed in each storage space is stored in the corresponding pluggable node, and an insertion operation is performed on the determined pluggable nodes. Then, the pluggable nodes in the working state are allocated to multiple pre-defined computing devices for processing. If any computing device finishes processing an assigned pluggable node, the unprocessed assigned pluggable nodes from another computing device are reassigned to any of the above computing devices for processing. Through these steps, the tasks in each storage space can be broken down into refined tasks based on pluggable nodes, which are then executed by the computing devices. Task transfer and balancing between computing devices are performed on a pluggable node basis, thereby improving the overall task processing efficiency and avoiding the long-tail effect.

[0030] The further effects of the aforementioned unconventional alternative methods will be explained below in conjunction with specific implementation methods. Attached Figure Description

[0031] The accompanying drawings are provided to better understand the invention and are not intended to unduly limit the scope of the invention. Wherein: Figure 1 This is a schematic diagram of the main steps of the task processing method in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the specific execution steps of the task processing method in this embodiment of the invention; Figure 3 This is a schematic diagram of the first system architecture of the task processing method in an embodiment of the present invention; Figure 4 This is a schematic diagram of the second system architecture of the task processing method in an embodiment of the present invention; Figure 5 This is a schematic diagram of the components of the task processing device in an embodiment of the present invention; Figure 6 This is an exemplary system architecture diagram that can be applied thereto according to embodiments of the present invention; Figure 7 This is a schematic diagram of the electronic device structure used to implement the task processing method in the embodiments of the present invention. Detailed Implementation

[0032] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of the present invention, including various details to aid understanding. These details should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the invention. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0033] It should be noted that, unless otherwise specified, the embodiments of the present invention and the technical features thereof can be combined with each other.

[0034] Figure 1 This is a schematic diagram of the main steps of the task processing method in an embodiment of the present invention.

[0035] like Figure 1 As shown, the task processing method of this embodiment of the invention can be executed by the main device (e.g., a scheduling server or a monitoring center server), and the specific execution steps are as follows: Step S101: The main device establishes a data structure containing multiple pluggable nodes; the pluggable nodes are the constituent elements of the data structure, and enter the working state based on the insertion operation and enter the stop state based on the removal operation.

[0036] Step S102: The main device determines multiple pluggable nodes for each of the multiple preset storage spaces, stores the task data to be executed in each storage space into the corresponding pluggable node, and performs an insertion operation on the determined pluggable node.

[0037] Step S103: The main device assigns the pluggable nodes that are in working state to multiple preset computing devices for processing; in response to any computing device completing the processing of the assigned pluggable nodes, the assigned pluggable nodes that have not been processed by another computing device are reassigned to any computing device for processing.

[0038] Through the above steps, tasks in each storage space can be broken down into fine-grained tasks executed by computing devices on a pluggable node basis. Task transfer and balancing are performed between computing devices on a pluggable node basis, thereby improving the overall processing efficiency of tasks and avoiding the occurrence of long-tail effects.

[0039] Figure 2 This is a schematic diagram illustrating the specific execution steps of the task processing method in an embodiment of the present invention, such as... Figure 2 As shown, the task processing method includes the following execution steps.

[0040] Step S201: The main device acquires source data from the database tables. This step and step S202 are preferred steps to address the problem of uneven data processing volume among different computing devices. In related technologies, business data tables are processed through database sharding to form source data in database tables. Due to certain objective reasons, the number of data in each database table varies greatly, resulting in a serious data skew problem. If these database tables are directly distributed to computing devices for computation, the computation time of each computing device will differ significantly, leading to uneven resource utilization and affecting the overall task execution efficiency. As a preferred solution, this embodiment of the invention can address this issue through steps S201 and S202. It is understood that in this embodiment of the invention, steps S203 to S209 can also be executed directly without executing steps S201 and S202, which can also solve the problem of uneven task execution time among computing devices to a certain extent. The computing devices mentioned above can be servers or other devices used to execute tasks. In this step, the main device first acquires source data from multiple database tables, and the amount of data in these database tables may vary significantly in actual applications.

[0041] Step S202: The main device maps the source data to multiple storage spaces. In this step, the source data can be mapped to multiple storage spaces with equal data volume. In this document, "equal data volume" and "equal amount" mean that the difference in data volume is within a pre-set error range, for example, the relative error is within 1%. Preferably, the following steps can be performed to achieve the above data mapping. For source data in multiple database tables with unequal data volumes, the main device first performs hash processing on the identifier of the source data, and then performs a modulo operation on the resulting hash value; subsequently, based on the modulo operation result, the identifiers of the source data are stored in equal amounts in the storage spaces. The data records in the storage spaces can be called task data to be executed. The task data to be executed can include only the identifier of the source data to save storage medium space, or a new identifier can be established for the source data and the mapping relationship between the new identifier and the original identifier can be stored. Other fields can also be included as needed. In specific applications, the number of storage spaces can be determined according to the number of computing devices. For example, if there are 64 computing devices, the number of storage spaces can be determined to be 128, allowing one computing device to process data from two storage spaces.

[0042] Step S203: The main device determines the pluggable nodes in the data structure. In this step, the main device first establishes a data structure containing multiple pluggable nodes. Exemplarily, the data structure can be a circular linked list, a circular buffer, or a circular queue, with slots (i.e., storage locations) for inserting nodes. A pluggable node refers to a constituent element of the aforementioned data structure; for example, in the case of a circular linked list, a pluggable node is an element within the circular linked list. Each pluggable node includes a data field and a pointer field. The data field stores data, and the pointer field stores the storage address of the next node. It can be understood that the position of each pluggable node within the data structure can be determined through the pointer field information.

[0043] Pluggable nodes can enter a working state based on an insertion operation and a stopped state based on a removal operation. The working state refers to the state where the computing device can perform data processing and calculations, while the stopped state refers to the inoperable state. The implementation logic of the insertion and removal operations of pluggable nodes is related to the internal logic of the data structure. Taking a circular linked list as an example, the insertion operation of a pluggable node is as follows: link the pointer field of the node preceding the insertion position to the data field of the pluggable node, and link the pointer field of the pluggable node to the data field of the node following the insertion position. The removal operation of a pluggable node is as follows: link the pointer field of the node preceding the pluggable node directly to the data field of the node following the insertion position, bypassing the pluggable node itself.

[0044] Pluggable nodes can be inserted into slots in data structures when task processing is required, thus being in a detectable working state. When task processing is completed, they can be pulled out of the slot, thus being detectable as being in a stopped state. This pluggable method avoids the scheduling device from repeatedly polling and scheduling completed tasks ineffectively, thereby improving task execution efficiency.

[0045] The preferred structures for the above data structures are circular linked lists, circular buffers, and circular queues. For circular linked lists, circular buffers, and circular queues, the first and last storage locations are connected. Therefore, a preset number of pluggable nodes in adjacent storage locations can be selected to perform allocation (e.g., allocating pluggable nodes to computing devices) and to obtain data to be processed (e.g., reading data of tasks to be executed from storage space). There is no problem of the first and last storage locations being isolated from each other.

[0046] After establishing the data structure, the host device can also assign multiple pluggable nodes to each storage space. In practical applications, the host device can determine the number of pluggable nodes based on the number of storage spaces, ensuring that each storage space corresponds to an equal number (an integer greater than 1) of pluggable nodes. Clearly, the number of pluggable nodes is an integer multiple of the number of storage spaces. Continuing the previous example, if the number of computing devices is 64 and the number of storage spaces is 128, then the number of pluggable nodes can be determined to be 256, so that one computing device corresponds to 4 pluggable nodes and one storage space corresponds to two pluggable nodes.

[0047] Step S204: The host device determines the correspondence between storage spaces and pluggable nodes. In this step, the host device can determine the correspondence between each storage space and multiple pluggable nodes, and save the determined correspondence. This process can be called pluggable node registration.

[0048] Step S205: The main device stores the task data to be executed in the storage space to the pluggable nodes. In this step, the main device can equally divide the task data to be executed in each storage space and store it in the corresponding pluggable nodes, so that each pluggable node of the data structure stores an equal amount of task data to be executed. Through this step, fine-grained task data to be executed can be achieved at the unit of pluggable nodes, which is beneficial for subsequent task transfer and balancing based on fine-grained data.

[0049] Step S206: The main device inserts a pluggable node into the data structure. In this step, the main device inserts the previously determined pluggable node into the slot of the data structure to put the pluggable node into a working state. This step can be performed before, during, or after step S205.

[0050] Step S207: The main device allocates the pluggable nodes of the data structure to computing devices. In this step, the main device can allocate the pluggable nodes in the working state to multiple pre-set computing devices for processing. Preferably, an equal number of pluggable nodes in the working state can be allocated to each computing device. The computing device can perform calculations on the task data to be executed stored in the allocated pluggable nodes to complete the relevant task. Preferably, when the data structure is a circular linked list, circular buffer, or circular queue, multiple adjacent pluggable nodes can be allocated to the same computing device in a fixed direction (e.g., clockwise), and the next adjacent node can be allocated to the next computing device in sequence. The same pluggable node can only be allocated to one computing device. In one embodiment, when the computing device performs a computing task, it can first determine the source data from the database table using the source data identifier stored in the allocated pluggable nodes, and then perform calculations on the determined source data.

[0051] Step S208: The master device performs task transfer and balancing between computing devices on a per-pluggable-node basis. As a preferred solution, the master device monitors the status of computing devices and the completion status of each pluggable node in real time. If the master device detects that any computing device has completed processing of an assigned pluggable node, it can reassign the assigned pluggable node of another computing device that has not yet started processing to any other computing device for processing. This achieves fine-grained task transfer and balancing between computing devices, making the task execution time of each computing device similar, avoiding idle or wasted computing resources, and ultimately improving the overall task processing efficiency.

[0052] Preferably, if the main device detects that any computing device has finished processing an assigned pluggable node, it can determine the computing device with the largest number of assigned pluggable nodes that have not yet been processed, and reassign at least one of the assigned pluggable nodes of that computing device to any computing device for processing, thereby achieving a more reasonable task balance.

[0053] Step S209: Remove the completed pluggable node from the data structure. In this step, if the main device detects that any pluggable node has been processed, it can perform a removal operation on that pluggable node, thereby avoiding the scheduling device from repeatedly polling and scheduling completed tasks ineffectively, and improving task execution efficiency.

[0054] Figure 3 This is a schematic diagram of the first system architecture of the task processing method in an embodiment of the present invention. Figure 4 This is a schematic diagram of the second system architecture of the task processing method in an embodiment of the present invention. See [link / reference] Figure 3 and Figure 4 .

[0055] The data balancing unit maps source data from multiple database tables with unequal data volumes to multiple storage spaces with equal data volumes, thereby avoiding the impact of data skew on the overall efficiency of task execution. The data structure control unit controls data structures (such as circular linked lists, circular buffers, circular queues, etc.) to store the task data to be executed in the storage space in finer-grained pluggable nodes, enabling fine-grained task transfer and balancing between computing devices. The monitoring center can monitor the status of each computing device and the integrity of each pluggable node in real time to perform task reallocation between computing devices, and can serve as the main device in the task processing method of this invention.

[0056] The logic processing unit provides the data structure control unit with specific task computation methods and interacts with the monitoring center to determine the current state of the data structure. Specifically, after startup, the data structure is in the initialization state (init). Once the computing device resources and storage space are ready, it transitions to the runnable state. When the computing device begins executing a specific computation task, it transitions to the running state (running). After the task is completed, it transitions to the finished state (finished). If the task fails, it transitions from the running state to the dead state (dead). After recovery, it transitions from the dead state to the ready state.

[0057] The logic processing unit can also directly manipulate the task data to be executed in the storage space through the business logic processor to achieve specific business objectives. The monitoring center can also monitor the relevant data of the business logic processor to perform corresponding operations. The system may also include a data inspection unit, pluggable components, and a data recovery center. The data inspection unit is used to perform data correctness checks, the pluggable components provide other pluggable business components, and the data recovery center is used to recover data when errors occur.

[0058] In the technical solution of this invention, the problem of uneven data processing volume between different servers caused by data skew is solved. It also solves the problem that although different servers process the same amount of data, the execution time of different server tasks is different due to differences in data quality, server network and configuration. The long tail effect is avoided. The system automatically senses and executes task transfer and balancing on a unit of pluggable nodes. It also promptly removes completed pluggable nodes to avoid invalid polling scheduling of servers, thereby improving the overall task execution efficiency.

[0059] It should be noted that the technical solutions of this invention, including the collection, updating, analysis, processing, use, transmission, and storage of user personal information, all comply with relevant laws and regulations, are used for legitimate purposes, and do not violate public order and good morals. Necessary measures are taken to prevent unauthorized access to user personal information data and to safeguard user personal information security, network security, and national security.

[0060] For the foregoing method embodiments, they are described as a series of actions for ease of description. However, those skilled in the art should understand that the present invention is not limited to the described order of actions, and some steps may actually be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential for implementing the present invention.

[0061] To facilitate better implementation of the above-described solutions of the embodiments of the present invention, related apparatus for implementing the above-described solutions is also provided below.

[0062] Please see Figure 5 As shown, the task processing device 500 provided in this embodiment of the invention may include: a data structure establishment unit 501, a node determination unit 502, and a node allocation unit 503.

[0063] The data structure establishment unit 501 can be used to establish a data structure containing multiple pluggable nodes; the pluggable nodes are the constituent elements of the data structure, entering a working state based on an insertion operation and a stopping state based on a removal operation; the node determination unit 502 can be used to determine multiple pluggable nodes for each of the preset multiple storage spaces, store the task data to be executed in each storage space into the corresponding pluggable node, and perform an insertion operation on the determined pluggable nodes; the node allocation unit 503 can be used to allocate the pluggable nodes in the working state to multiple preset computing devices for processing; in response to any computing device completing the processing of the allocated pluggable nodes, the allocated pluggable nodes that have not been processed by another computing device are reassigned to any computing device for processing.

[0064] In this embodiment of the invention, the node determination unit 502 is further configured to: determine an equal number of multiple pluggable nodes for each storage space.

[0065] Preferably, each storage space stores an equal amount of task data to be executed; and the node determination unit 502 is further configured to: divide the task data to be executed in each storage space into equal amounts and store them in the corresponding pluggable nodes, so that each pluggable node of the data structure stores an equal amount of task data to be executed.

[0066] As a preferred embodiment, the node allocation unit 503 is also used to allocate an equal number of pluggable nodes in operation to each computing device.

[0067] In one embodiment, the node allocation unit 503 is further configured to: in response to any computing device completing the processing of an allocated pluggable node, determine the computing device with the largest number of allocated pluggable nodes that have not been processed, and reassign at least one of the allocated pluggable nodes of that computing device that has not been processed to any computing device for processing.

[0068] In an alternative technical solution, the node allocation unit 503 is further configured to: perform a pluggable node unplugging operation in response to the completion of processing of any pluggable node.

[0069] In some embodiments, the task processing apparatus 500 further includes a data balancing unit, configured to: perform hash processing on the identifier of the source data in multiple database tables with unequal data volumes, perform a modulo operation on the obtained hash value; store the identifier of the source data in a storage space according to the result of the modulo operation; and include the identifier of the source data in the task data to be executed in the storage space.

[0070] Furthermore, in this embodiment of the invention, the data structure is a circular linked list, a circular buffer, or a circular queue; the computing device performs task processing based on source data determined from a database table using source data identifiers stored in allocated pluggable nodes.

[0071] According to the technical solution of the present invention, the tasks of each storage space can be divided into refined tasks based on pluggable nodes and executed by computing devices. The computing devices perform task transfer and balancing based on pluggable nodes, thereby improving the overall processing efficiency of tasks and avoiding the occurrence of long-tail effects.

[0072] Figure 6 An exemplary system architecture 600 is shown that can be applied to the task processing method or task processing apparatus of the present invention.

[0073] like Figure 6 As shown, system architecture 600 may include terminal devices 601, 602, and 603, network 604, and server 605 (this architecture is merely an example; the components included in a specific architecture may be adjusted according to the specific circumstances of the invention). Network 604 serves as the medium for providing a communication link between terminal devices 601, 602, and 603 and server 605. Network 604 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0074] Users can use terminal devices 601, 602, and 603 to interact with server 605 via network 604 to receive or send messages, etc. Various client applications can be installed on terminal devices 601, 602, and 603, such as task scheduling applications (for example only).

[0075] Terminal devices 601, 602, and 603 can be various electronic devices with displays and web browsing capabilities, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0076] Server 605 can be a server that provides various services, such as a monitoring server (for example only) that supports task scheduling applications operated by users using terminal devices 601, 602, and 603. The monitoring server can process received task execution requests and feed back the processing results (such as whether the execution was successful - for example only) to terminal devices 601, 602, and 603.

[0077] It should be noted that the task processing method provided in the embodiments of the present invention is generally executed by the server 605, and correspondingly, the task processing device is generally set in the server 605.

[0078] It should be understood that Figure 6 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0079] The present invention also provides an electronic device. The electronic device according to an embodiment of the present invention includes: one or more processors; and a storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the task processing method provided by the present invention.

[0080] The following is for reference. Figure 7 It shows a schematic diagram of the structure of a computer system 700 suitable for implementing an electronic device according to embodiments of the present invention. Figure 7 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0081] like Figure 7 As shown, the computer system 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 702 or programs loaded from storage section 708 into random access memory (RAM) 703. The RAM 703 also stores various programs and data required for the operation of the computer system 700. The CPU 701, ROM 702, and RAM 703 are interconnected via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0082] The following components are connected to the I / O interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the I / O interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0083] In particular, according to the embodiments disclosed in this invention, the processes described in the above main step diagrams can be implemented as computer software programs. For example, embodiments of this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the main step diagrams. In the above embodiments, the computer program can be downloaded and installed from a network via communication section 709, and / or installed from removable medium 711. When the computer program is executed by central processing unit 701, it performs the functions defined in the system of this invention.

[0084] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0086] The units described in the embodiments of the present invention can be implemented in software or hardware. The described units can also be located in a processor; for example, a processor can be described as including: a data structure establishment unit, a node determination unit, and a node allocation unit. The names of these units do not necessarily limit the specific unit; for example, the data structure establishment unit can also be described as "a unit that provides pluggable nodes to the node determination unit."

[0087] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs, and when the one or more programs are executed by the device, the steps performed by the device include: establishing a data structure containing multiple pluggable nodes; the pluggable nodes are components of the data structure, entering a working state based on an insertion operation and a stopped state based on a removal operation; determining multiple pluggable nodes for each of a plurality of preset storage spaces, storing the task data to be executed in each storage space into the corresponding pluggable node, and performing an insertion operation on the determined pluggable nodes; allocating the pluggable nodes in the working state to multiple preset computing devices for processing; and, in response to any computing device completing the processing of an allocated pluggable node, reassigning an allocated pluggable node that has not been processed by another computing device to any computing device for processing.

[0088] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the task processing method provided by the present invention.

[0089] In the technical solution of this invention embodiment, the tasks of each storage space can be divided into refined tasks based on pluggable nodes and executed by computing devices. The computing devices perform task transfer and balancing based on pluggable nodes, thereby improving the overall processing efficiency of tasks and avoiding the occurrence of long tail effects.

[0090] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A task processing method, characterized in that, include: Establish a data structure containing multiple pluggable nodes; The pluggable node is a component of the data structure, and enters a working state based on an insertion operation and a stopped state based on a removal operation. For each of the pre-defined multiple storage spaces, determine multiple pluggable nodes, store the task data to be executed in each storage space into the corresponding pluggable node, and perform an insertion operation on the determined pluggable node. The pluggable nodes that are in operation are distributed to multiple pre-configured computing devices for processing; In response to any computing device completing the processing of an assigned pluggable node, an assigned pluggable node that has not yet been processed by another computing device is reassigned to the first computing device for processing.

2. The method according to claim 1, characterized in that, The step of defining multiple pluggable nodes for each of the pre-defined multiple storage spaces includes: For each storage space, determine an equal number of multiple pluggable nodes.

3. The method according to claim 2, characterized in that, Each storage space contains an equal amount of data for tasks to be executed; And, the step of storing the task data to be executed in each storage space into the corresponding pluggable node includes: The data of the tasks to be executed in each storage space is divided equally and stored in the corresponding pluggable nodes, so that each pluggable node of the data structure stores an equal amount of data of the tasks to be executed.

4. The method according to claim 3, characterized in that, The process of allocating the pluggable nodes in working condition to multiple pre-set computing devices for processing includes: Allocate an equal number of pluggable nodes that are in operation to each computing device.

5. The method according to claim 1, characterized in that, The step of reassigning an allocated pluggable node that has not yet been processed by any computing device to the other computing device for processing includes: In response to any computing device completing the processing of an assigned pluggable node, the computing device with the largest number of assigned pluggable nodes that have not yet been processed is identified, and at least one of the assigned pluggable nodes of that computing device that has not yet been processed is reassigned to the aforementioned computing device for processing.

6. The method according to claim 1, characterized in that, The method further includes: In response to the completion of processing of any pluggable node, a pluggable node is unplugged.

7. The method according to claim 3, characterized in that, The method further includes: For source data in multiple database tables with varying amounts of data, perform hash processing on the identifier of the source data, and perform a modulo operation on the resulting hash value; The identifier of the source data is stored in the storage space based on the result of the modulo operation; the task data to be executed in the storage space includes the identifier of the source data.

8. The method according to claim 7, characterized in that, The data structure is a circular linked list, a circular buffer, or a circular queue; The computing device processes tasks based on source data determined from the database table using source data identifiers stored in the allocated pluggable nodes.

9. A task processing device, characterized in that, include: The data structure creation unit is used to create a data structure containing multiple pluggable nodes. The pluggable node is a component of the data structure, and enters a working state based on an insertion operation and a stopped state based on a removal operation. The node determination unit is used to determine multiple pluggable nodes for each of the multiple preset storage spaces, store the task data to be executed in each storage space into the corresponding pluggable node, and perform an insertion operation on the determined pluggable node. The node allocation unit is used to allocate pluggable nodes that are in operation to multiple pre-set computing devices for processing. In response to any computing device completing the processing of an assigned pluggable node, an assigned pluggable node that has not yet been processed by another computing device is reassigned to the first computing device for processing.

10. An electronic device, characterized in that, include: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the method as described in any one of claims 1-8.

12. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images