Distributed task processing method, system and storage medium

Abstract

The application discloses a distributed task processing method, a system and a storage medium, and the distributed task processing method comprises the following steps: a first data packet containing task parameters and identification information for indicating a node for performing an aggregation operation is sent; the identification information is used for indicating that the aggregation operation is performed on the task parameters by a forwarding node or a parameter node; the task parameters are obtained by performing a distributed computing task; a second data packet containing aggregation parameters is received; the aggregation parameters are used for data processing of each working node corresponding to the same distributed computing task; and the aggregation parameters are obtained by performing the aggregation operation on the task parameters in the first data packet by the forwarding node or the parameter node. The application solves the communication bottleneck problem existing in the distributed system and improves the processing efficiency of the distributed task.

Description

Technical Field

[0001] This application relates to the field of computer data processing, specifically to a distributed task processing method, system, and storage medium. Background Technology

[0002] Currently, a common network architecture for distributed computing systems includes a parameter server and multiple nodes. The nodes are responsible for calculating and obtaining model parameters and synchronizing them to the parameter server, while the parameter server is responsible for calculating and updating global parameters based on the model parameters transmitted by the nodes. However, as the amount of data transmitted between the nodes and the parameter server in this architecture increases, communication bottlenecks inevitably arise. Summary of the Invention

[0003] In view of the shortcomings of the above-mentioned related technologies, the purpose of this application is to provide a distributed task processing method, system and storage medium to overcome the technical problem of communication bottleneck in the distributed computing system in the above-mentioned related technologies.

[0004] To achieve the above and other related objectives, the first aspect of this application discloses a distributed task processing method, comprising: sending a first data packet containing task parameters and identification information for indicating a node to perform an aggregation operation; wherein the identification information is used to indicate that an aggregation operation is performed on the task parameters by a forwarding node or a parameter node; wherein the task parameters are obtained by executing a distributed computing task; receiving a second data packet containing aggregation parameters; wherein the aggregation parameters are used for data processing by each working node corresponding to the same distributed computing task; wherein the aggregation parameters are obtained by the forwarding node or parameter node performing an aggregation operation on the task parameters in the first data packet.

[0005] The second aspect of this application discloses a distributed task processing system, comprising: a sending module, configured to send a first data packet containing task parameters and identification information for indicating nodes performing an aggregation operation; wherein the identification information is used to indicate that a forwarding node or a parameter node performs an aggregation operation on the task parameters; the task parameters are obtained by executing a distributed computing task; and a receiving module, configured to receive a second data packet containing aggregation parameters; wherein the aggregation parameters are used for data processing by each working node corresponding to the same distributed computing task; wherein the aggregation parameters are obtained by the forwarding node or the parameter node performing an aggregation operation based on the task parameters in the first data packet.

[0006] The third aspect disclosed in this application provides a working node, comprising: at least one memory for storing at least one program; and at least one processor connected to the at least one memory for executing and implementing the distributed task processing method as described in the first aspect when running the at least one program.

[0007] The fourth aspect disclosed in this application provides a distributed task processing method, comprising: receiving a plurality of first data packets containing task parameters and identification information for indicating nodes to perform aggregation operations, wherein the identification information is used to indicate that a forwarding node or a parameter node performs an aggregation operation on the task parameters; wherein the plurality of task parameters are obtained by a plurality of worker nodes by executing a distributed computing task; performing an aggregation operation on each task parameter according to the identification information in the first data packet to obtain aggregate parameters, and sending the aggregate parameters to the parameter nodes for verification; and feeding back a second data packet containing the aggregate parameters sent by the parameter nodes to the corresponding worker nodes.

[0008] The fifth aspect of this application discloses a distributed task processing system, comprising: a receiving module, configured to receive a plurality of first data packets containing task parameters and identification information for indicating nodes to perform aggregation operations, wherein the identification information is used to indicate that a forwarding node or a parameter node performs an aggregation operation on the task parameters; wherein the task parameters are obtained by a plurality of worker nodes through the execution of a distributed computing task; a processing module, configured to perform aggregation operations on each task parameter according to the identification information in the first data packet to obtain aggregate parameters, and to send the aggregate parameters to the parameter nodes for verification; and a feedback module, configured to feed back a second data packet containing the aggregate parameters sent by the parameter nodes to the corresponding worker nodes.

[0009] The sixth aspect of this application discloses a forwarding node, comprising: at least one memory for storing at least one program; and at least one processor connected to the at least one memory for executing and implementing the distributed task processing method as described in the fourth aspect when running the at least one program.

[0010] The seventh aspect of this application discloses a distributed task processing method, comprising: receiving a second data packet containing aggregation parameters; wherein the aggregation parameters are obtained by a forwarding node performing an aggregation operation on task parameters in a plurality of first data packets, the first data packets containing identification information indicating that the aggregation operation is performed by the forwarding node; wherein the plurality of task parameters are obtained by a plurality of worker nodes performing a distributed computing task; performing a verification operation on the received second data packet, and feeding back the verified second data packet to the forwarding node, so that the forwarding node feeds back the second data packet containing the aggregation parameters to the corresponding worker node.

[0011] The eighth aspect of this application discloses a distributed task processing system, comprising: a receiving module for receiving a second data packet containing aggregation parameters; wherein the aggregation parameters are obtained by a forwarding node performing an aggregation operation on task parameters in a plurality of first data packets, the first data packets containing identification information indicating that the aggregation operation is performed by the forwarding node; wherein the task parameters are obtained by a plurality of worker nodes performing a distributed computing task; and a processing module for performing a verification operation on the aggregation parameters in the received second data packet and feeding back the verified second data packet to the forwarding node, so that the forwarding node feeds back the second data packet containing the aggregation parameters to the corresponding worker node.

[0012] The ninth aspect of this application discloses a parameter node, comprising: at least one memory for storing at least one program; and at least one processor connected to the at least one memory for executing and implementing the distributed task processing method as described in the seventh aspect when running the at least one program.

[0013] The tenth aspect of this application discloses a distributed task processing system, comprising: a plurality of worker nodes as described in the third aspect; at least one forwarding node as described in the sixth aspect, communicatively connected to the worker nodes, for performing an aggregation operation based on task parameters in a first data packet sent by the worker nodes to obtain an aggregation result or aggregation parameters; and a parameter node as described in the ninth aspect, communicatively connected to the forwarding node, for performing an aggregation operation on the aggregation result to obtain aggregation parameters and / or performing a verification operation on the aggregation parameters.

[0014] The eleventh aspect of this application discloses a computer-readable storage medium storing at least one program, which, when executed by a processor, executes and implements the distributed task processing method as described in the first aspect, or executes and implements the distributed task processing method as described in the fourth aspect, or executes and implements the distributed task processing method as described in the seventh aspect.

[0015] In summary, the distributed task processing method, system, and storage medium provided in this application, within the distributed task processing architecture, have the forwarding node perform an aggregation operation on the task parameters received from the worker node, and send the aggregated parameters obtained from the aggregation operation to the parameter node for verification. This solves the communication bottleneck problem between the forwarding node and the parameter node and improves the processing efficiency of distributed tasks.

[0016] Other aspects and advantages of this application will readily be apparent to those skilled in the art from the detailed description below. Only exemplary embodiments of this application are shown and described in the following detailed description. As will be appreciated by those skilled in the art, the content of this application enables them to make modifications to the disclosed specific embodiments without departing from the spirit and scope of the invention to which this application pertains. Accordingly, the descriptions in the accompanying drawings and specification of this application are merely exemplary and not restrictive. Attached Figure Description

[0017] The specific features of the invention involved in this application are shown in the appended claims. The features and advantages of the invention can be better understood by referring to the exemplary embodiments and drawings described in detail below. A brief description of the drawings is as follows:

[0018] Figure 1 The diagram shown is a schematic representation of the distributed task processing architecture of this application in one embodiment.

[0019] Figure 2A and Figure 2B The diagram shown is a schematic representation of the distributed task processing architecture of this application in one embodiment.

[0020] Figure 3 The diagram shown is a schematic representation of the distributed task processing architecture of this application in one embodiment.

[0021] Figure 4 The diagram shown is a flowchart of one embodiment of the distributed task processing method of this application.

[0022] Figure 5 The diagram shown is a flowchart of another embodiment of the distributed task processing method of this application.

[0023] Figure 6 The diagram shown is a flowchart of another embodiment of the distributed task processing method of this application.

[0024] Figure 7 The diagram shown is a flowchart of another embodiment of the distributed task processing method of this application.

[0025] Figure 8 The diagram shown is a flowchart of another embodiment of the distributed task processing method of this application.

[0026] Figure 9 The diagram shown is a flowchart of another embodiment of the distributed task processing method of this application.

[0027] Figure 10 The diagram shown is a flowchart of an embodiment of the exception handling method for distributed tasks in this application.

[0028] Figure 11 The diagram shown is a flowchart of another embodiment of the exception handling method for distributed tasks in this application.

[0029] Figure 12 The diagram shown is a block diagram of the module composition of the distributed task processing system of this application in one embodiment.

[0030] Figure 13 The diagram shown is a block diagram of the module composition of the distributed task processing system of this application in another embodiment.

[0031] Figure 14 The diagram shown is a block diagram of the module composition of the distributed task processing system of this application in another embodiment.

[0032] Figure 15 The diagram shown is a block diagram of the module composition of the exception handling system for distributed tasks in this application in one embodiment.

[0033] Figure 16 The diagram shown is a block diagram of the module composition of the exception handling system for distributed tasks in this application in another embodiment.

[0034] Figure 17 The diagram shown is a block diagram of the module composition of a working node in one embodiment of this application.

[0035] Figure 18 The diagram shown is a block diagram of the module composition of a forwarding node in one embodiment of this application.

[0036] Figure 19 The diagram shown is a block diagram of the module composition of a parameter node in one embodiment of this application. Detailed Implementation

[0037] The following specific embodiments illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification.

[0038] In the following description, reference is made to the accompanying drawings, which illustrate several embodiments of the present application. It should be understood that other embodiments may also be used, and changes in module or unit composition, electrical and operational aspects may be made without departing from the spirit and scope of this disclosure. The following detailed description should not be considered limiting, and the scope of the embodiments of the present application is defined solely by the claims of the published patents. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the present application.

[0039] While the terms first, second, etc., are used in some instances herein to describe various elements, information, or parameters, these elements or parameters should not be limited by these terms. These terms are used only to distinguish one element or parameter from another. For example, a first data format may be referred to as a second data format, and similarly, a second data format may be referred to as a first data format, without departing from the scope of the various described embodiments. Both the first and second data formats describe a data format, but they are not the same data format unless the context otherwise explicitly indicates otherwise. Depending on the context, the word "if," as used herein, may be interpreted as "when" or "when...".

[0040] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are to be interpreted inclusively, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition occur only when combinations of elements, functions, steps, or operations are inherently mutually exclusive in some way.

[0041] With the explosive growth of data and the improvement of computer performance, distributed computing systems have been increasingly widely used. A common application of distributed computing systems is distributed machine learning model training. Taking gradient training of a distributed machine learning model as an example, its distributed architecture typically includes multiple worker nodes and parameter nodes. In some examples, this distributed architecture executes the machine learning model training process as follows: multiple worker nodes are responsible for training their respective datasets to obtain gradient parameters, and then transmit these gradient parameters to the parameter nodes, which update the global gradient parameters accordingly. In other examples, this distributed architecture executes the machine learning model training process as follows: multiple worker nodes share the training operations of different parts of a model, using the dataset to train each part to obtain the corresponding gradient parameters, and then transmitting these gradient parameters to the parameter nodes, which update the global gradient parameters accordingly. However, as the amount of data transmitted increases, setting only one parameter node cannot avoid the bottleneck problem of transmission bandwidth; while setting multiple parameter nodes can lead to network saturation and excessive load.

[0042] In view of this, this application provides a distributed task processing architecture that can be used to implement distributed computing tasks in single-tenant and multi-tenant modes.

[0043] Please see Figure 1 , Figures 2A-2B The diagram shown is a schematic representation of the distributed task processing architecture of this application in one embodiment. Figure 1 As shown, the distributed task processing architecture includes multiple worker nodes, at least one forwarding node, and parameter nodes.

[0044] In this system, multiple worker nodes are used to execute one or more distributed computing tasks. Each worker node is connected to a forwarding node to transmit the task parameters obtained from executing the distributed computing task to the forwarding node. The forwarding node receives the task parameters sent by the worker nodes, performs an aggregation operation to obtain aggregate parameters, and transmits the aggregate parameters to the parameter node for verification. The parameter node receives the aggregate parameters sent by the forwarding node and performs a verification operation.

[0045] Once the parameter node verifies and confirms that the aggregated parameters are correct, it returns the aggregated parameters to the forwarding node, which then distributes them to each worker node. Each worker node then executes a new round of distributed computing tasks based on the received aggregated parameters.

[0046] Therefore, the distributed task processing architecture solves the bottleneck problem of transmission bandwidth by having the forwarding node perform the aggregation operation and the parameter node perform the verification operation, thereby improving the computational efficiency of distributed computing tasks and effectively reducing the communication overhead in the network.

[0047] In situations requiring the processing of large-scale distributed computing tasks, the number of worker nodes increases. To avoid the transmission bandwidth between forwarding nodes and worker nodes becoming a bottleneck, the forwarding nodes can be further configured as primary forwarding nodes and secondary forwarding nodes.

[0048] In some embodiments, the distributed task processing architecture includes multiple worker nodes, multi-level forwarding nodes, and parameter nodes. For example, as shown... Figure 2A As shown, the multi-level forwarding node includes multiple first-level forwarding nodes and at least one second-level forwarding node. Each first-level forwarding node is communicatively connected to multiple worker nodes, and each second-level forwarding node is communicatively connected to each first-level forwarding node. Figure 1 The illustrated embodiment is similar. The first-level forwarding node receives the task parameters obtained by the worker node from the distributed computing task and performs an aggregation operation to obtain aggregate parameters. These aggregate parameters are then sent to the parameter node via the second-level forwarding node, where the parameter node performs a verification operation.

[0049] In some embodiments, the first-level forwarding node performs an aggregation operation to obtain a partially completed intermediate result (for distinction, the final result of the completed aggregation is referred to as the "aggregation parameter" and the partially completed intermediate result is referred to as the "aggregation result"); then the second-level forwarding node receives the aggregation result sent by the first-level forwarding node and performs the aggregation operation to obtain the final aggregation parameter.

[0050] To improve the efficiency of distributed computing task processing and make full use of resources, secondary forwarding nodes can also connect to at least one worker node. Therefore, in addition to receiving the aggregation results sent by the primary forwarding node, the secondary forwarding node also receives task parameters sent by the worker node it is connected to, and performs an aggregation operation on the aggregation results and task parameters to obtain the final aggregation parameters.

[0051] As the amount of data to be processed increases, the distributed task processing architecture may, for example, be: Figure 2B The architecture is shown in the figure. As shown, the multi-level forwarding nodes may also include a data center, which can be used to connect the first-level forwarding nodes and the second-level forwarding nodes and realize communication transmission, data interaction and other functions between them.

[0052] It should be understood that, such as Figure 2A and Figure 2B The distributed task processing architecture shown can be adjusted according to actual needs in specific application scenarios, such as adding or removing one or more worker nodes. However, those skilled in the art will understand that, without departing from the existing architecture, the distributed task processing architecture can be further modified. Figure 1 Any modifications or alterations made to the distributed task processing architecture shown herein are within the scope of protection of this application.

[0053] For ease of understanding, the terms mentioned in this application will be explained below.

[0054] The worker node is responsible for calculating parameters for distributed computing tasks and can be a single computer device, or a physical or virtual device used in a cloud-based service system. The single computer device can be a self-configured computer capable of executing distributed computing tasks based on computational instructions, and can be located in a private data center or a rented space in a public data center. The cloud-based service system includes public cloud servers and private cloud servers, including IaaS (Infrastructure-as-a-Service), PaaS (Platform-as-a-Service), and SaaS (Software-as-a-Service). Examples of private cloud servers include Alibaba Cloud Computing Service Platform, Amazon Web Services Platform, Baidu Cloud Computing Platform, and Tencent Cloud Computing Platform. The worker node can also be a virtual device, where the computational instructions configured on the worker node are software programs executable by the virtual device, and the physical or virtual worker node is configured within the distributed architecture / system. In some embodiments, the hardware devices for the actual computational instructions executed by the working node may be located on a single server or in multiple servers, and the computational instructions executed by the working node may be completed through data communication between the servers.

[0055] The forwarding node possesses aggregation computing and forwarding functions. For example, the forwarding node is a programmable switch or a dedicated network device with an embedded FPGA, providing pathways for its connected worker nodes and parameter nodes, and receiving data sent by the worker nodes to perform corresponding computational processing. Since servers are typically densely distributed, for example, the worker nodes and forwarding nodes are connected in a TOR configuration, meaning a switch is connected to the server rack, so that the network ports of each server are connected to the switch at the top of the rack, thereby simplifying cabling and facilitating maintenance and management.

[0056] The parameter node is responsible for managing the storage and updating of parameters. In some embodiments, the parameter node also has aggregation calculation and verification functions to ensure the correctness of the aggregation results. For example, the parameter node is a single server or a cluster of multiple servers.

[0057] The communication connection refers to a connection method between different nodes that establishes connections based on certain communication protocols or other matching rules to achieve data transmission and exchange. This includes connections established between nodes through matching identification information. The communication protocol is the rule that nodes must follow to achieve communication or services. The protocol defines the format of data units, the connection method of data units, and the timing of data transmission and reception to ensure data identification and transmission between nodes. The communication protocol includes protocols such as TCP / IP. The identification information is used to determine the attributes of different nodes to ensure that the connections between nodes are correctly matched, thereby enabling subsequent data transmission between nodes.

[0058] The single-tenant mode refers to the distributed processing architecture being used to process the same distributed computing task, meaning that all worker nodes execute the computation of a single distributed computing task. Conversely, the multi-tenant mode refers to the distributed processing architecture being able to process multiple distributed computing tasks simultaneously, meaning that worker nodes are divided into multiple clusters, and worker nodes within the same cluster execute the same distributed computing task, thereby improving computing efficiency and avoiding waste of computing resources.

[0059] For example, the distributed computing task includes a computational task of gradient training of a machine learning model using distributed computing. The machine learning model is a model obtained using a machine learning algorithm, such as a classification model. In this distributed computing task, each worker node is assigned a subset of the complete training dataset and completes gradient updates of a machine learning model through multiple rounds of iterative training.

[0060] To clearly illustrate the principles, this paper uses the distributed task processing architecture to process a distributed computing task (hereinafter referred to as "task") based on machine learning model training as an example; and unless otherwise specified, each worker node in this application is used to execute one task (i.e., single-tenant mode) as an example.

[0061] To clearly demonstrate the data processing and transmission flow between nodes in a distributed task processing architecture, the following sections will explain the process from the perspective of different execution entities.

[0062] Example 1

[0063] Please see Figure 3 and Figure 4 , Figure 3 The diagram shown is a schematic representation of the distributed task processing architecture of this application in one embodiment. Figure 4 The diagram shows a flowchart of the distributed task processing method of this application in one embodiment. Taking the single-tenant mode as an example, after the worker node executes the training task of the current round and obtains the task parameters (e.g., the gradient values ​​obtained from the training subset), as follows... Figure 3 and Figure 4 As shown, the working node executes step S101.

[0064] In step S101, a first data packet containing task parameters and identification information for indicating the node performing the aggregation operation is sent; wherein the identification information is used to indicate that the aggregation operation is performed on the task parameters by a forwarding node or a parameter node; wherein the task parameters are obtained by executing a distributed computing task.

[0065] Here, each worker node sends a packet containing task parameters to the forwarding node. For distinction, the packet containing task parameters sent by the worker node is called the first packet, and the packet containing aggregation parameters obtained after the aggregation operation is called the second packet.

[0066] It should be noted that the terms "first" and "second" are only used to distinguish whether the data packets have been aggregated. Although other information in the data packets (such as source port number, destination port number, etc.) may change during the transmission process in actual scenarios, they are not used as the basis for distinguishing the first data packet and the second data packet. Instead, the parameter data (task parameters or aggregation parameters) carried in the data packets are used to distinguish the first data packet and the second data packet.

[0067] The task parameters are parameters calculated by the worker nodes during the execution of distributed computing tasks, such as gradient values ​​obtained from training a deep neural network model. However, this is not a limitation. For example, when the distributed task processing architecture is used to execute computational tasks on privacy-preserving data based on a distributed architecture, such as training tasks for federated learning, the task parameters can also be parameters encrypted by each worker node.

[0068] The first data packet further includes identification information indicating the node performing the aggregation operation. When the identification information indicates that the aggregation operation is performed by a forwarding node, the forwarding node receives the first data packet and performs the aggregation operation; when the identification information indicates that the aggregation operation is performed by a parameter node, the forwarding node receives the first data packet and forwards it to the parameter node, which then performs the aggregation operation. Under normal circumstances (where "normal circumstances" refers to the absence of communication link interruptions, network delays, or packet loss during data transmission), the identification information in the first data packet sent by the working node indicates that the aggregation operation is performed by a forwarding node.

[0069] In some embodiments, the worker node adds the identification information to the header of the data packet when encapsulating it. For example, the identification information is composed of one or more flags in the header. Exemplarily, the identification information used to indicate the node performing the aggregation operation is represented by "0" or "1": when the identification information is "0", it indicates that the aggregation operation is performed by the forwarding node; when the identification information is "1", it indicates that the aggregation operation is performed by the parameter node, and the forwarding node only performs forwarding operations.

[0070] It should be understood that the above embodiments are merely examples of principles and not limitations. In specific application scenarios, the form or type of the identification information can be changed according to actual needs. For example, it can also be represented as a field represented by multiple bits.

[0071] In some embodiments, the first data packet further includes node identification information, allowing the node performing the aggregation operation to confirm which worker nodes are collaboratively executing the current task. Here, the worker nodes add node identification information to the header of the first data packet. When a forwarding node or parameter node receives and parses the first data packet, it can determine the total number of worker nodes executing the task corresponding to the first data packet based on the node identification information.

[0072] In some embodiments, the node identification information can also be used to confirm which worker node the currently received data packet originates from. For example, the node identification information is represented by a field of multiple bits. For instance, when the current task is executed by four worker nodes, the node identification information might be: 0001, 0010, 0100, 1000, where each "0" or "1" indicates whether the corresponding worker node sent the first data packet. The forwarding node or parameter node can then use the node identification information to determine that the current task is executed by four worker nodes, and further, to determine which worker nodes have already sent data packets, which worker nodes are still waiting to send data packets, and which worker node the currently received first data packet originated from.

[0073] After the worker node sends the first data packet, the forwarding node receives the first data packet and performs an aggregation operation. Please refer to [link to relevant documentation]. Figure 5 The diagram shown is a flowchart of another embodiment of the distributed task processing method of this application. Figure 3 and Figure 5 As shown, the forwarding node executes steps S201 and S202.

[0074] In step S201, a first data packet containing task parameters and identification information of the node that performs the aggregation operation is received.

[0075] Here, the forwarding node receives first data packets sent by multiple worker nodes and extracts the task parameters from them through parsing. Normally, the identification information in the first data packets sent by the worker nodes indicates that the forwarding node will perform the aggregation operation.

[0076] In step S202, an aggregation operation is performed on each task parameter based on the identification information in the first data packet to obtain aggregated parameters, and the aggregated parameters are sent to the parameter node for verification.

[0077] Here, the forwarding node performs an aggregation operation on the task parameters sent by each working node under the same task based on one or more of the node identification information, task identification information, and node identification information used to instruct the forwarding node to perform the aggregation operation, and obtains the aggregation parameters.

[0078] The aggregation operation refers to the operation of integrating and calculating the parameters of each task. For example, it may involve operations such as accumulating, superimposing, merging, averaging, or summing the parameters. For instance, the aggregation operation might calculate and sum the gradient values ​​(corresponding to the task parameters) provided by each node, thus obtaining the sum of gradient values ​​as the aggregation parameter. Another example is the aggregation operation summing the encrypted values ​​provided by each node based on multi-party secure computation. The aggregation parameter obtained through the aggregation operation is used by the worker nodes corresponding to the same distributed computing task for data processing. For example, each worker node updates the gradient of the next round of the distributed computing task based on the gradient values ​​(corresponding to the aggregation parameter) updated by the forwarding node or parameter node.

[0079] In some embodiments, the forwarding node can extract the task parameters from each first data packet to perform the aggregation operation after all first data packets under a task have arrived. In some embodiments, since the arrival order of the first data packets under the same task is different, the forwarding node can obtain a partial aggregation result based on the first arriving first data packets. When subsequent first data packets arrive, the task parameters in the subsequent first data packets are then aggregated with the obtained partial aggregation result to obtain the final aggregation parameters.

[0080] After obtaining the aggregation parameters, the forwarding node sends the aggregation parameters to the parameter node for verification. Upon receiving the first data packet from the forwarding node, the parameter node performs the verification operation. Please refer to [link to relevant documentation]. Figure 6 The diagram shown is a flowchart of another embodiment of the distributed task processing method of this application. Figure 3 and Figure 6 As shown, the parameter node executes steps S301 and S302.

[0081] In step S301, a second data packet containing aggregation parameters is received; wherein the aggregation parameters are obtained by the forwarding node by performing an aggregation operation on task parameters in multiple first data packets, and the first data packets contain identification information indicating that the aggregation operation is performed by the forwarding node.

[0082] Here, the parameter node receives the second data packet sent by the forwarding node. The second data packet contains the aggregated parameters obtained after the forwarding node performs the aggregation operation. For the specific process, please refer to the aforementioned embodiment, which will not be repeated here.

[0083] In step S302, a verification operation is performed on the received second data packet, and the verified second data packet is fed back to the forwarding node so that the forwarding node can feed back the second data packet containing the aggregation parameters to the corresponding working node.

[0084] Here, the parameter node performs a verification operation on the received second data packet. The verification operation includes at least one of the following: verifying whether the aggregation operation has omitted task parameters provided by the worker node, and verifying whether the aggregation parameters have overflowed.

[0085] The verification operation to check for omissions in the aggregation operation refers to verifying whether the aggregation parameters of a certain task actually correspond to the corresponding number of worker nodes. For example, when a task is completed collaboratively by four worker nodes, the parameter node verifies whether the aggregation parameters of the forwarding node are obtained based on the complete aggregation of task parameters issued by the four worker nodes.

[0086] The verification of whether the aggregation parameters overflow refers to verifying whether the aggregation parameters obtained by the forwarding nodes exceed the predetermined data representation range. For example, the data obtained by worker nodes when performing distributed computing is usually in floating-point number format, referred to as the first data format; while the data format that forwarding nodes can process is integer, referred to as the second data format. Compared to floating-point numbers, the data representation range of integers is limited (2...). -15 ~2 15 Therefore, sometimes the result obtained from aggregation exceeds the data representation range that the forwarding node can represent.

[0087] If the aggregation verification operation is missed, the parameter node will not return any data packets until the worker node detects packet loss due to not receiving the second data packet within the predetermined time and retransmits the first data packet. As mentioned above, when the worker node detects packet loss and retransmits the first data packet, the forwarding node only performs the forwarding operation on the retransmitted first data packet, while the aggregation operation is completed by the parameter node.

[0088] If the aggregation operation and aggregation parameters are verified to be correct, the parameter node will return the verified second data packet to the forwarding node. The forwarding node will then receive the verified second data packet and perform the distribution operation. For example... Figure 3 and Figure 5 As shown, the forwarding node then executes step S203.

[0089] In step S203, the second data packet containing the aggregation parameters sent by the parameter node is fed back to the corresponding working node.

[0090] Here, the forwarding node receives a second data packet containing aggregation parameters sent by the parameter node and forwards the second data packet to the corresponding worker node. The forwarding node may use, for example, multicast to distribute the second data packet to each worker node performing the same task.

[0091] After receiving the second data packet, the worker node extracts the aggregation parameters and executes the next round of tasks accordingly. For example... Figure 3 and Figure 4 As shown, the working node then executes step S102.

[0092] In step S102, a second data packet containing aggregation parameters is received; wherein the aggregation parameters are used for data processing by each working node corresponding to the same distributed computing task.

[0093] Here, the worker node receives a second data packet containing aggregation parameters and performs data processing based on these parameters. This data processing refers to the worker node updating or replacing parameters for the corresponding task based on the aggregation parameters. For example, the worker node updates the gradient based on the average gradient value (corresponding to the aggregation parameters) in the second data packet, thereby executing the next round of tasks with the new gradient value. Alternatively, the worker node decrypts the encrypted value (corresponding to the aggregation parameters) in the second data packet, thereby executing the next round of multi-party secure computation based on the decrypted value.

[0094] Thus, the working node, forwarding node, and parameter node system complete one round of distributed computing task.

[0095] Example 2

[0096] The above embodiments describe the workflow of each node in a single-tenant mode. In a multi-tenant mode (where worker nodes execute multiple tasks simultaneously), the node performing the aggregation operation needs to know which task the currently received data packet corresponds to in order to prevent confusion between parameters of different tasks. Therefore, based on any of the above embodiments or any combination thereof, in some embodiments, the first data packet sent by the worker node also includes task identification information, so that the node performing the aggregation operation can confirm the distributed computing task corresponding to the first data packet.

[0097] Here, the worker node adds task identification information to the header of the first data packet. When the forwarding node or parameter node receives and parses the first data packet, it can determine the task corresponding to the current first data packet based on the task identification information. For example, the task identification information may be a sequence number (Job ID) assigned to each task by the worker node. It should be understood that the task identification information for each task is unique.

[0098] Example 3

[0099] The above embodiments illustrate the process under normal circumstances (where "normal" refers to a situation where packet loss does not occur due to communication link interruption, network latency, or other factors). However, in real-world scenarios, packet loss may occur due to communication link interruption, network latency, or other factors.

[0100] Therefore, based on any of the above embodiments or any combination thereof, some embodiments further include performing packet loss detection during the transmission of the first data packet and retransmitting the first data packet when packet loss is detected.

[0101] Here, with Figure 3 As shown in the example, the worker node starts timing when it sends the first data packet. If no data packet is received after the timing reaches a preset threshold, the worker node determines that there is packet loss and resends the first data packet containing the task parameters. When the worker node receives the returned second data packet, it resets the timing to zero and restarts timing the next time a data packet is sent.

[0102] In other specific examples, the working node groups N consecutive first data packets and marks the sending order. During the sequential sending of the first data packets in the group, it listens for the feedback information of each first data packet. Based on the number and / or order of the received feedback information in the group, it determines the packet loss situation of the group and retransmits the first data packets that are determined to be lost.

[0103] To enable forwarding nodes or parameter nodes to identify whether the first data packet they receive is a retransmitted data packet due to packet loss, in some embodiments, the working node adds retransmission identification information to the first data packet when retransmitting it. This retransmission identification information allows forwarding nodes and parameter nodes to know that the corresponding first data packet is a retransmitted data packet, thus distinguishing it from the first data packet sent by the working node under normal circumstances.

[0104] In some embodiments, the identification information in the first data packet retransmitted by the worker node, which indicates that the node performing the aggregation operation is performed by the parameter node, allows the forwarding node to perform a forwarding operation on the first data packet based on the identification information, and the parameter node to perform an aggregation operation on the task parameters.

[0105] Here, when the identification information in the first data packet received by the forwarding node indicates that the aggregation operation is to be performed by the parameter node, the forwarding node performs a forwarding operation on the first data packet, and the parameter node performs the aggregation operation. Therefore, as follows... Figure 7 As shown, the parameter node also executes steps S401 and S402.

[0106] In step S401, a first data packet containing task parameters and identification information indicating that the parameter node performs the aggregation operation is received.

[0107] Here, each working node sends a first data packet containing task parameters. The identification information in the first data packet, indicating that the aggregation operation is performed by a parameter node, is used to indicate that the aggregation operation is performed by the parameter node. A forwarding node receives the first data packet and performs a forwarding operation based on one or more of the identification information indicating that the aggregation operation is performed by the parameter node and retransmission identification information; the parameter node receives the first data packet forwarded by the forwarding node.

[0108] In step S402, an aggregation operation is performed on each task parameter according to the identification information in the first data packet to obtain the aggregation parameter, and a second data packet containing the aggregation parameter is sent to the forwarding node so that the forwarding node can feed back the second data packet containing the aggregation parameter to the corresponding working node.

[0109] Here, the parameter node performs an aggregation operation on the task parameters in each first data packet based on one or more of the following identifiers: task identifier information, node identifier information, identifier information indicating that the parameter node will perform the aggregation operation, and retransmission identifier information, to obtain aggregation parameters. After obtaining the aggregation parameters, the parameter node sends a second data packet containing the aggregation parameters to the forwarding node, which then distributes it to each working node. For the specific process, please refer to the aforementioned embodiment; it will not be repeated here.

[0110] Example 4

[0111] Based on any of the above embodiments or any combination thereof, in order to improve the efficiency of aggregation operations, the forwarding node can further partition its internal space, where each partition is a computing resource, and each computing resource is used to perform aggregation operations on task parameters. The computing resource refers to a unit with computing processing capabilities, and can also be called an aggregator.

[0112] Therefore, upon receiving the first data packet, the forwarding node determines a computing resource among the various computing resources to perform the aggregation operation. For example, the forwarding node can poll according to the index of the computing resources to determine the available computing resources. Alternatively, the forwarding node can randomly access and randomly determine a computing resource to perform the aggregation operation.

[0113] In order to provide forwarding nodes with internal computing resources to perform corresponding aggregation operations, such as Figure 8 As shown, the working node also executes steps S501 and S502.

[0114] In step S501, resource configuration identification information corresponding to the same distributed computing task is generated. The resource configuration identification information is used to instruct the forwarding node to allocate computing resources for performing the aggregation operation.

[0115] Here, the worker node generates resource configuration identification information for the distributed computing tasks it executes. For example, the resource configuration identification information is obtained by hashing the task identification information and the sequence number of the computing resource. It should be noted that the resource configuration identification information added by the worker node for the same task is identical, to ensure that the first data packet sent by the worker node executing the same task is aggregated by the same computing resource.

[0116] In step S502, a first data packet containing task parameters and the resource configuration identification information is sent.

[0117] Here, the working node adds the resource configuration identification information to the first data packet and sends it to the forwarding node, so that the forwarding node can allocate corresponding computing resources to perform the aggregation operation according to the resource configuration identification information.

[0118] After the forwarding node performs the aggregation operation to obtain the aggregation parameters, it sends a second data packet to the parameter node for verification. Once the parameter node returns the verified second data packet to the forwarding node, the forwarding node distributes it to each working node. Therefore, as follows... Figure 8 As shown, the worker node also executes step S503: receiving a second data packet containing aggregation parameters. Subsequent procedures are described in the aforementioned embodiment and will not be repeated here.

[0119] Example 5

[0120] In order to make full use of computing resources and avoid the resource waste and uneven distribution problems of the static allocation method of dividing regions as described above, based on any of the above embodiments or any combination thereof, in some embodiments, the forwarding node can also use a dynamic allocation method to allocate computing resources to the first data packet.

[0121] Therefore, after the forwarding node receives the first data packet containing the task parameters and the resource configuration identification information, as follows: Figure 9 As shown, steps S601 and S602 are executed.

[0122] In step S601, a first data packet containing task parameters and resource configuration identification information for indicating the allocation of computing resources to perform the aggregation operation is received.

[0123] Here, the forwarding node receives a first data packet containing task parameters sent by multiple worker nodes. The first data packet also contains resource configuration identification information indicating the allocation of computing resources to perform the aggregation operation. For the specific process, please refer to the foregoing embodiments; it will not be repeated here.

[0124] In step S602, computing resources corresponding to the same distributed computing task are dynamically allocated based on the resource configuration identifier information in each of the first data packets, and the computing resources are used to perform aggregation operations on the corresponding multiple task parameters to obtain the aggregated parameters corresponding to the distributed computing task.

[0125] Here, the forwarding node dynamically allocates a computing resource to perform an aggregation operation for the first data packet based on the resource configuration identifier information in the received first data packet. The forwarding node can also determine the task corresponding to the computing resource based on the task identifier information in the first data packet, and determine how many nodes are under the current task, so as to determine how many first data packets need to be aggregated, thereby keeping the computing resource occupied during the aggregation operation.

[0126] For example, the method by which the forwarding node dynamically allocates computing resources includes steps S611 and S612 (not shown).

[0127] In step S611, the computing resources of each first data packet corresponding to the same distributed computing task are determined based on the mapping relationship between the resource configuration identification information in each first data packet and each computing resource.

[0128] The mapping relationship represents a one-to-one correspondence between resource configuration identification information and computing resources. For example, the resource configuration identification information is the sequence number of the computing resource among all computing resources, and the forwarding node can directly determine the corresponding computing resource based on this. Exemplarily, the mapping relationship is determined by hashing the resource configuration identification information. For example, the resource configuration identification information is obtained by the worker node performing a hash operation on the task identification information and the sequence number of the computing resource.

[0129] In step S612, the allocated computing resources are used to perform an aggregation operation corresponding to the same distributed computing task to obtain aggregation parameters.

[0130] Here, after the forwarding node allocates the first data packet to a computing resource, it allocates all subsequent first data packets received from other nodes corresponding to the same task to the same computing resource, so that the computing resource can extract the task parameters from the first data packet, perform an aggregation operation on multiple task parameters, and obtain aggregated parameters.

[0131] Once the aggregation parameters are obtained, the forwarding node will send the aggregation parameters to the parameter node for verification. For the specific process, please refer to the aforementioned embodiment, which will not be repeated here.

[0132] Example 6

[0133] It should be understood that the dynamic allocation method described in the above embodiments can also be applied in multi-tenant mode. However, in multi-tenant mode, when forwarding nodes dynamically allocate computing resources, multiple tasks may be assigned to the same computing resource, which can lead to computing resource conflicts (such as hash collisions).

[0134] Therefore, based on any of the above embodiments or any combination thereof, in some embodiments, when the computing resources dynamically allocated for the distributed computing task corresponding to a first data packet are occupied, the forwarding node marks the conflict identification information in the first data packet and performs a forwarding operation, so that the parameter node can perform an aggregation operation on the task parameters in the forwarded first data packet according to the conflict identification information to obtain aggregate parameters.

[0135] Here, the forwarding node forwards the first data packet to the parameter node; correspondingly, the parameter node is also used to receive the first data packet containing conflict identification information sent by the forwarding node, and perform an aggregation operation on the task parameters in the forwarded first data packet according to the conflict identification information to obtain aggregation parameters. Here, the parameter node knows that the currently received first data packet is forwarded by the forwarding node due to a conflict based on the conflict identification information, and performs an aggregation operation based on one or more identification information such as task identification information and node identification information in the first data packet. For the specific process, please refer to the foregoing embodiment, which will not be repeated here.

[0136] After obtaining the aggregation parameters, the parameter node sends a second data packet containing the aggregation parameters to the forwarding node. In some embodiments, in order for the forwarding node to identify whether the currently received data packet is the first data packet sent by the worker node or the second data packet sent by the parameter node, the parameter node adds verification identification information to the second data packet when returning the verified second data packet; when the forwarding node parses the verification identification information in the data packet, it can determine that the currently received data packet is the verified second data packet sent by the parameter node.

[0137] In some embodiments, to prevent conflicts from occurring in the next allocation of computing resources, the parameter node is further configured to update the resource configuration identifier information based on the conflict identifier information, and send a second data packet containing the aggregation parameters and the updated resource configuration identifier information, so that the forwarding node can feed the second data packet back to the corresponding working node.

[0138] Therefore, the forwarding node also performs the following steps: receiving the second data packet containing the aggregation parameters sent by the parameter node, and feeding the second data packet back to the corresponding working node. For the specific process, please refer to the aforementioned embodiment; it will not be repeated here.

[0139] Example 7

[0140] Based on any of the above embodiments or any combination thereof, in some embodiments, when it is determined that the computing resources do not need to be occupied, the forwarding node releases the corresponding computing resources.

[0141] For example, when the forwarding node completes the aggregation operation and obtains the aggregation parameters, and is ready to send the second data packet containing the aggregation parameters to the parameter node, it releases the corresponding computing resources.

[0142] For example, when the forwarding node receives the second data packet containing the aggregation parameters returned by the parameter node and is ready to distribute it to the worker node, it releases the corresponding computing resources.

[0143] For example, when the forwarding node receives a first data packet sent by a portion of the worker nodes corresponding to a task (at which point it has already occupied one computing resource), and then receives a first data packet resent by another worker node corresponding to the same task due to packet loss, the forwarding node forwards the resent first data packet to the parameter node and releases the corresponding computing resources. At this time, the forwarding node also forwards the task parameters from the first data packet previously received from the portion of the worker nodes, or the aggregation result obtained based on the received first data packet from the portion of the worker nodes, to the worker node, whereby the parameter node performs the aggregation operation to obtain the aggregation parameters.

[0144] Example 8

[0145] In the above embodiments, the task parameters in the first data packet sent by the working node are all in the second data format. As mentioned above, when the forwarding node performs an aggregation operation on the task parameters in the second data format, sometimes the aggregation result exceeds the data representation range that the forwarding node can represent. In this case, the parameter node will check for aggregation parameter overflow.

[0146] To provide a solution for aggregation parameter overflow, based on any of the above embodiments or any combination thereof, when verifying aggregation parameter overflow, such as... Figure 10 As shown, the parameter node executes steps S701, S702, and S703.

[0147] In step S701, a second data packet containing aggregation parameters is received; wherein the aggregation parameters are obtained by the forwarding node through performing an aggregation operation on a first data packet containing task parameters in a second data format sent by multiple working nodes.

[0148] Here, following the process in the aforementioned embodiment, the parameter node receives a second data packet containing aggregation parameters sent by the forwarding node. The task parameters in the first data packet sent by the worker node are in a second data format (i.e., integers), and the aggregation parameters obtained by the forwarding node performing the aggregation operation are also in the second data format.

[0149] In step S702, when the aggregation result overflow is detected, an abnormal retransmission instruction is issued; wherein, the abnormal retransmission instruction is used to instruct the working node to perform data format conversion.

[0150] Here, the parameter node detects whether the received aggregated parameters overflow. For example, when the aggregated parameters obtained by the forwarding node overflow, the value of the aggregated parameters represents the maximum range of data formats that the forwarding node can process (e.g., 2). 31 Therefore, when the parameter node detects that the aggregation parameter has overflowed, it issues an abnormal retransmission instruction to the worker node under the corresponding task, instructing the worker node to retransmit the first data packet and to convert the task parameter into a data format during retransmission, so as to avoid overflow when the forwarding node performs the aggregation operation.

[0151] In some embodiments, the abnormal retransmission instruction is, for example, a data packet containing identification information for instructing the working node to perform a retransmission operation; the parameter node sends the data packet and it is forwarded to the working node by the forwarding node; after receiving the data packet, the working node obtains the abnormal retransmission instruction by parsing and identifying the identification information therein.

[0152] After receiving the abnormal retransmission instruction from the parameter node, the worker node retransmits the first data packet. Therefore, as follows: Figure 11 As shown, the working node also performs step S801.

[0153] In step S801, when an abnormal retransmission instruction is received, a first data packet containing task parameters in a first data format and identification information indicating that the parameter node shall perform an aggregation operation is sent; wherein, the abnormal retransmission instruction is used to instruct the worker node to perform data format conversion.

[0154] Here, after receiving the abnormal retransmission instruction, the worker node retransmits the first data packet containing the task parameters. At this time, because the task parameters in the second data format can cause the aggregated parameters to overflow, the worker node converts the task parameters in the second data format (integer) to the first data format (floating-point number) when retransmitting the first data packet. Since the forwarding node cannot process the task parameters in the first data format, the identification information in the first data packet retransmitted by the worker node indicates that the aggregation operation was performed by the parameter node.

[0155] When the parameter node receives the first data packet resent by the working node, the parameter node executes step S703.

[0156] In step S703, the first data packet retransmitted by the working node based on the abnormal retransmission instruction is received; according to the identification information in the first data packet, the task parameters of each first data format are aggregated to obtain a second data packet containing the aggregated parameters, and the data packet is fed back to each working node.

[0157] Here, the parameter node receives the first data packet retransmitted by the worker node, and performs an aggregation operation on the task parameters of the first data format based on one or more of the following identifiers in the first data packet: identifier information of the node indicating the node to perform the aggregation operation, task identifier information, node identifier information, and retransmission identifier information. The process of the parameter node performing the aggregation operation is described in the foregoing embodiments and will not be repeated here.

[0158] After the parameter node performs the aggregation operation to obtain the aggregation parameters in the first data format, it sends the second data packet containing the aggregation parameters to the forwarding node, which then distributes it to each working node.

[0159] Therefore, the working node executes step S802: receiving a second data packet containing aggregation parameters. For the specific process, please refer to the aforementioned embodiment; it will not be repeated here.

[0160] Example 9

[0161] This application also provides a distributed task processing system for executing the distributed task processing methods corresponding to steps S101 to S102 in the foregoing embodiments, which has corresponding functional modules and can achieve the same technical effect.

[0162] Please see Figure 12 The diagram shows the module composition of the distributed task processing system of this application in one embodiment. Figure 12 As shown, the distributed task processing system includes a sending module 101 and a receiving module 102.

[0163] The sending module is configured to send a first data packet containing task parameters and identification information indicating the node performing the aggregation operation; wherein the identification information is used to indicate that the aggregation operation is performed on the task parameters by a forwarding node or a parameter node; the task parameters are obtained by executing a distributed computing task.

[0164] The receiving module is used to receive a second data packet containing aggregation parameters; wherein, the aggregation parameters are used for data processing by each working node corresponding to the same distributed computing task; wherein, the aggregation parameters are obtained by the forwarding node or parameter node after performing an aggregation operation based on the task parameters in the first data packet.

[0165] In some embodiments, the distributed task processing system further includes a generation module for generating resource configuration identification information corresponding to the same distributed computing task, wherein the resource configuration identification information is used to instruct forwarding nodes to allocate computing resources for performing aggregation operations.

[0166] Accordingly, the sending module is also configured to send a first data packet containing task parameters and the resource configuration identification information; wherein the task parameters are obtained by multiple worker nodes through executing the distributed computing task.

[0167] Accordingly, the receiving module is further configured to receive a second data packet containing aggregation parameters; wherein the aggregation parameters are used to update the task parameters of the distributed computing task; wherein the aggregation parameters are obtained by a computing resource in the forwarding node performing an aggregation operation based on the task parameters in the first data packet.

[0168] In this embodiment, for the sake of simplicity, the sending module and receiving module in the distributed task processing system can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions. Figure 4 The distributed task processing method in the illustrated embodiment will not be described in detail here.

[0169] Example 10

[0170] This application also provides a distributed task processing system for executing the distributed task processing methods corresponding to steps S201 to S203 of the aforementioned embodiments, which has corresponding functional modules and can achieve the same technical effects.

[0171] Please see Figure 13 The diagram shows the module composition of the distributed task processing system of this application in one embodiment. Figure 13 As shown, the distributed task processing system includes a receiving module 201, a processing module 202, and a feedback module 203.

[0172] The receiving module is configured to receive a first data packet containing multiple task parameters and identification information of nodes that indicate the execution of aggregation operations. The identification information is used to indicate that the aggregation operation is performed on the task parameters by a forwarding node or a parameter node. The multiple task parameters are obtained by multiple worker nodes by executing a distributed computing task.

[0173] The processing module is used to perform aggregation operations on each task parameter based on the identification information in the first data packet to obtain aggregated parameters, and to send the aggregated parameters to the parameter node for verification.

[0174] The feedback module is used to send the second data packet containing the aggregation parameters sent by the parameter node back to the corresponding working node.

[0175] In some embodiments, the receiving module is further configured to receive a first data packet containing multiple task parameters and resource configuration identification information for indicating the allocation of computing resources to perform the aggregation operation; wherein the task parameters are obtained by multiple worker nodes by executing a distributed computing task.

[0176] Accordingly, the processing module is also configured to dynamically allocate a computing resource to the first data packet based on the resource configuration identification information in the first data packet, perform an aggregation operation on the computing resource to obtain aggregation parameters, and send the aggregation parameters to the parameter node for verification.

[0177] Accordingly, the feedback module is also used to feed back the second data packet containing the aggregation parameters sent by the parameter node to the corresponding working node.

[0178] In this embodiment, for the sake of simplicity, the sending module and receiving module in the distributed task processing system can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions. Figure 5 The distributed task processing method in the illustrated embodiment will not be described in detail here.

[0179] Example 11

[0180] This application also provides a distributed task processing system for executing the distributed task processing methods corresponding to the aforementioned steps S301 to S302, which has corresponding functional modules and can achieve the same technical effect.

[0181] Please see Figure 14 The diagram shows the module composition of the distributed task processing system of this application in one embodiment. Figure 14 As shown, the distributed task processing system includes a receiving module 301 and a processing module 302.

[0182] The receiving module is used to receive a second data packet containing aggregation parameters; wherein the aggregation parameters are obtained by the forwarding node by performing an aggregation operation on task parameters in multiple first data packets, and the first data packets contain identification information indicating that the aggregation operation is performed by the forwarding node; wherein the multiple task parameters are obtained by multiple worker nodes by executing a distributed computing task.

[0183] The processing module is used to perform a verification operation on the received second data packet and feed back the verified second data packet to the forwarding node, so that the forwarding node can feed back the second data packet containing the aggregation parameters to the corresponding working node.

[0184] In this embodiment, for the sake of simplicity, the sending module and receiving module in the distributed task processing system can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions. Figure 6 The distributed task processing method in the illustrated embodiment will not be described in detail here.

[0185] Example 12

[0186] This application also provides an exception handling system for distributed tasks, used to execute the exception handling method for distributed tasks corresponding to the aforementioned steps S701 to S703, having corresponding functional modules and achieving the same technical effect.

[0187] Please refer to Figure 15 The diagram shows the module composition of the distributed task exception handling system of this application in one embodiment. Figure 15 As shown, the anomaly handling system includes a receiving module 401, a detection module 402, and a processing module 403.

[0188] The receiving module is used to receive a second data packet containing the aggregation result; wherein the aggregation result is obtained by the forwarding node by performing an aggregation operation on a first data packet containing task parameters in a second data format sent by multiple worker nodes; wherein the task parameters are obtained by the worker nodes by executing a distributed computing task.

[0189] The detection module is used to issue an abnormal retransmission command when the aggregation result overflow is detected; wherein, the abnormal identification information is used for the worker node to adjust the task parameters and perform the retransmission operation.

[0190] The processing module is used to receive a first data packet resent by a worker node based on an abnormal retransmission instruction. The resent first data packet contains task parameters of a first data format and identification information indicating that an aggregation operation is performed by a parameter node. Based on the identification information in the first data packet, an aggregation operation is performed on the task parameters of each first data format to obtain a second data packet containing aggregation parameters, and then fed back to each worker node.

[0191] In this embodiment, for the sake of simplicity, the sending module and receiving module in the distributed task processing system can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions. Figure 10 The exception handling method for distributed tasks in the illustrated embodiment will not be described in detail here.

[0192] Example 13

[0193] This application also provides an exception handling system for distributed tasks, used to execute the exception handling method for distributed tasks corresponding to the aforementioned steps S801 to S802, which has corresponding functional modules and can achieve the same technical effect.

[0194] Please refer to Figure 16 The diagram shows the module composition of the distributed task exception handling system of this application in one embodiment. Figure 16 As shown, the exception handling system includes a sending module 501 and a receiving module 502.

[0195] The sending module is configured to send a first data packet containing task parameters in a first data format and identification information indicating that the parameter node will perform an aggregation operation when an abnormal retransmission instruction is received; wherein the abnormal retransmission instruction is used to instruct the worker node to perform data format conversion; the task parameters are obtained by executing a distributed computing task;

[0196] The receiving module is used to receive a second data packet containing aggregation parameters; wherein, the aggregation parameters are used for data processing by each working node corresponding to the same distributed computing task; wherein, the aggregation parameters are obtained by the parameter node performing an aggregation operation based on the first data packet.

[0197] In this embodiment, for the sake of simplicity, the sending module and receiving module in the distributed task processing system can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or by a combination of dedicated hardware and computer instructions. Figure 11 The distributed task processing method in the illustrated embodiment will not be described in detail here.

[0198] Example 14

[0199] This application also provides a working node. Please refer to... Figure 17 This is shown as a block diagram of the module composition of a working node in one embodiment of this application. Figure 17 As shown, the working node includes at least one memory 601 and at least one processor 602.

[0200] The at least one memory is used to store at least one program. In embodiments, the memory may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some embodiments, the memory may also include memory located remotely from one or more processors, such as network-attached memory accessed via RF circuitry or external ports and communication networks, wherein the communication network may be the Internet, one or more intranets, local area networks, wide area networks, storage area networks, etc., or suitable combinations thereof. The memory controller can control access to the memory by other components of the device, such as the CPU and peripheral interfaces.

[0201] In some embodiments, the at least one processor is connected to the at least one memory and, when running the at least one program, executes and implements at least one embodiment as described in the distributed task processing method above, such as... Figure 4 The embodiments described herein correspond to the processor. In these embodiments, the processor is operatively coupled to memory and / or non-volatile storage devices. More specifically, the processor can execute instructions stored in the memory and / or non-volatile storage devices to perform operations in a computing device, such as generating image data and / or transmitting image data to an electronic display. Thus, the processor may include one or more general-purpose microprocessors, one or more special-purpose processors, one or more field-programmable logic arrays, or any combination thereof.

[0202] Example 15

[0203] This application also provides a forwarding node. Please refer to... Figure 18 This is shown as a block diagram of the module composition of a forwarding node in one embodiment of this application. Figure 18 As shown, the forwarding node includes at least one memory 701 and at least one processor 702.

[0204] The at least one memory is used to store at least one program. In embodiments, the memory may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some embodiments, the memory may also include memory located remotely from one or more processors, such as network-attached memory accessed via RF circuitry or external ports and communication networks, wherein the communication network may be the Internet, one or more intranets, local area networks, wide area networks, storage area networks, etc., or suitable combinations thereof. The memory controller can control access to the memory by other components of the device, such as the CPU and peripheral interfaces.

[0205] In some embodiments, the at least one processor is connected to the at least one memory and, when running the at least one program, executes and implements at least one embodiment as described in the distributed task processing method above, such as... Figure 5 The embodiments described herein correspond to the processor. In these embodiments, the processor is operatively coupled to memory and / or non-volatile storage devices. More specifically, the processor can execute instructions stored in the memory and / or non-volatile storage devices to perform operations in a computing device, such as generating image data and / or transmitting image data to an electronic display. Thus, the processor may include one or more general-purpose microprocessors, one or more special-purpose processors, one or more field-programmable logic arrays, or any combination thereof.

[0206] Example 16

[0207] This application also provides a parameter node. Please refer to... Figure 19 This is shown as a block diagram of the module composition of the parameter nodes in one embodiment of this application. Figure 19 As shown, the parameter node includes at least one memory 801 and at least one processor 802.

[0208] The at least one memory is used to store at least one program. In embodiments, the memory may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage devices, flash memory devices, or other non-volatile solid-state storage devices. In some embodiments, the memory may also include memory located remotely from one or more processors, such as network-attached memory accessed via RF circuitry or external ports and communication networks, wherein the communication network may be the Internet, one or more intranets, local area networks, wide area networks, storage area networks, etc., or suitable combinations thereof. The memory controller can control access to the memory by other components of the device, such as the CPU and peripheral interfaces.

[0209] In some embodiments, the at least one processor is connected to the at least one memory and, when running the at least one program, executes and implements at least one embodiment as described in the distributed task processing method above, such as... Figure 6 The embodiments described herein correspond to the processor. In these embodiments, the processor is operatively coupled to memory and / or non-volatile storage devices. More specifically, the processor can execute instructions stored in the memory and / or non-volatile storage devices to perform operations in a computing device, such as generating image data and / or transmitting image data to an electronic display. Thus, the processor may include one or more general-purpose microprocessors, one or more special-purpose processors, one or more field-programmable logic arrays, or any combination thereof.

[0210] This application also provides a computer-readable and writable storage medium storing a computer program, which, when executed, implements at least one embodiment described above for the distributed task processing method, such as... Figure 4 The described embodiments.

[0211] This application also provides a computer-readable and writable storage medium storing a computer program, which, when executed, implements at least one embodiment described above for the distributed task processing method, such as... Figure 5 The described embodiments.

[0212] This application also provides a computer-readable and writable storage medium storing a computer program, which, when executed, implements at least one embodiment described above for the distributed task processing method, such as... Figure 6 The described embodiments.

[0213] This application also provides a computer-readable and writable storage medium storing a computer program, which, when executed, implements at least one embodiment described above for the distributed task processing method, such as... Figure 8 The described embodiments.

[0214] This application also provides a computer-readable and writable storage medium storing a computer program, which, when executed, implements at least one embodiment described above for the distributed task processing method, such as... Figure 9 The described embodiments.

[0215] This application also provides a computer-readable and writable storage medium storing a computer program, which, when executed, implements at least one embodiment described above for the exception handling method for distributed tasks, such as... Figure 10 The described embodiments.

[0216] This application also provides a computer-readable and writable storage medium storing a computer program, which, when executed, implements at least one embodiment described above for the exception handling method for distributed tasks, such as... Figure 11 The described embodiments.

[0217] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0218] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0219] In one or more exemplary aspects, the functionality described by the computer program of the methods of this application can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored or transmitted as one or more instructions or code onto a computer-readable medium. The steps of the methods or algorithms disclosed in this application can be embodied in processor-executable software modules, wherein the processor-executable software modules can reside on a tangible, non-transitory computer-readable and writable storage medium. The tangible, non-transitory computer-readable and writable storage medium can be any available medium accessible to a computer.

[0220] The flowcharts and block diagrams in the foregoing figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Accordingly, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the figures. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system that performs the specified function or operation, or by a combination of dedicated hardware and computer instructions.

[0221] The above embodiments are merely illustrative of the principles and effects of this application and are not intended to limit this application. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of this application. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in this application should still be covered by the claims of this application.

Claims

1. A distributed task processing method, characterized in that, include: Send a first data packet containing task parameters and identification information of the node to perform the aggregation operation; wherein the task parameters are obtained by executing a distributed computing task; Receive a second data packet containing aggregation parameters; wherein the aggregation parameters are used for data processing by each worker node corresponding to the same distributed computing task; Wherein, the identification information is used to indicate that the forwarding node performs an aggregation operation on the task parameters, and the aggregated parameters are obtained by the forwarding node performing an aggregation operation on the task parameters in the first data packet; or, the identification information is used to indicate that the parameter node performs an aggregation operation on the task parameters, and the aggregated parameters are obtained by the parameter node performing an aggregation operation on the task parameters in the first data packet; Wherein, the identification information used to indicate the node performing the aggregation operation is indicated by the parameter node performing the aggregation operation when it is determined to retransmit, so that the forwarding node performs the forwarding operation on the first data packet according to the identification information, and the parameter node performs the aggregation operation on the task parameters.

2. The distributed task processing method according to claim 1, characterized in that, The first data packet also contains task identification information, so that the forwarding node or parameter node performing the aggregation operation can identify the distributed computing task corresponding to the first data packet.

3. The distributed task processing method according to claim 1, characterized in that, The first data packet also contains node identification information, so that the forwarding node or parameter node performing the aggregation operation can identify the working node corresponding to the same distributed computing task.

4. The distributed task processing method according to claim 1, characterized in that, Also includes: Packet loss detection is performed during the transmission of the first data packet, and the first data packet is retransmitted when packet loss is detected.

5. The distributed task processing method according to claim 1, characterized in that, The first data packet also includes resource configuration identification information, so that the forwarding node can allocate computing resources to the first data packet based on the resource configuration identification information.

6. The distributed task processing method according to claim 1, characterized in that, Also includes: A new round of distributed computing tasks is executed based on the aggregation parameters in the second data packet.

7. The distributed task processing method according to claim 1, characterized in that, The distributed computing task includes a computational task of performing gradient training on a machine learning model using distributed computing.

8. A distributed task processing system, characterized in that, include: A sending module is configured to send a first data packet containing task parameters and identification information indicating the node performing the aggregation operation; wherein the task parameters are obtained by executing a distributed computing task; A receiving module is used to receive a second data packet containing aggregation parameters; wherein the aggregation parameters are used for data processing by each working node corresponding to the same distributed computing task. Wherein, the identification information is used to indicate that the forwarding node performs an aggregation operation on the task parameters, and the aggregated parameters are obtained by the forwarding node performing an aggregation operation on the task parameters in the first data packet; or, the identification information is used to indicate that the parameter node performs an aggregation operation on the task parameters, and the aggregated parameters are obtained by the parameter node performing an aggregation operation based on the task parameters in the first data packet; Wherein, the identification information used to indicate the node performing the aggregation operation is indicated by the parameter node performing the aggregation operation when it is determined to retransmit, so that the forwarding node performs the forwarding operation on the first data packet according to the identification information, and the parameter node performs the aggregation operation on the task parameters.

9. A working node, characterized in that, include: At least one memory for storing at least one program; At least one processor, connected to the at least one memory, is configured to execute and implement the distributed task processing method as described in any one of claims 1 to 7 when running the at least one program.

10. A distributed task processing method, characterized in that, include: Receive a first data packet containing multiple task parameters and identification information of nodes indicating the execution of an aggregation operation; wherein the multiple task parameters are obtained by multiple worker nodes by executing a distributed computing task; The identification information is used to instruct the forwarding node to perform an aggregation operation on the task parameters, thereby obtaining aggregated parameters by performing aggregation operations on each task parameter based on the identification information in the first data packet, and sending the aggregated parameters to the parameter node for verification; or, the identification information is used to instruct the parameter node to perform an aggregation operation on the task parameters, thereby obtaining aggregated parameters by performing aggregation operations on each task parameter through the parameter node based on the identification information in the first data packet. The second data packet containing the aggregation parameters sent by the parameter node is fed back to the corresponding working node.

11. The distributed task processing method according to claim 10, characterized in that, The first data packet received also contains task identification information for confirming the distributed computing task corresponding to the first data packet.

12. The distributed task processing method according to claim 10, characterized in that, The first data packet received also contains node identification information used to identify the worker node corresponding to the same distributed computing task.

13. The distributed task processing method according to claim 10, characterized in that, The received first data packet also includes resource configuration identification information for indicating the allocation of computing resources to perform the aggregation operation; correspondingly, it also includes: allocating a computing resource to the first data packet to perform the aggregation operation based on the resource configuration identification information.

14. The distributed task processing method according to claim 10, characterized in that, When the identification information in the received first data packet indicates that an aggregation operation is performed by a parameter node, a forwarding operation is performed on the first data packet based on the identification information.

15. The distributed task processing method according to claim 10, characterized in that, The distributed computing task includes a computational task of performing gradient training on a machine learning model using distributed computing.

16. A distributed task processing system, characterized in that, include: A receiving module is configured to receive multiple first data packets containing task parameters and identification information for indicating nodes to perform aggregation operations, wherein the identification information is used to indicate that a forwarding node performs an aggregation operation on the task parameters; wherein the task parameters are obtained by multiple worker nodes by executing a distributed computing task; and a processing module is configured to perform aggregation operations on each task parameter according to the identification information in the first data packet to obtain aggregate parameters, and send the aggregate parameters to a parameter node for verification. Alternatively, the receiving module is configured to receive a first data packet containing multiple task parameters and identification information indicating a node to perform an aggregation operation, wherein the identification information indicates that a parameter node performs an aggregation operation on the task parameters; wherein the task parameters are obtained by multiple worker nodes executing a distributed computing task; and the processing module is configured to obtain aggregation parameters by performing aggregation operations on each task parameter through the parameter node based on the identification information in the first data packet; and The feedback module is used to send the second data packet containing the aggregation parameters sent by the parameter node back to the corresponding working node.

17. A forwarding node, characterized in that, include: At least one memory for storing at least one program; At least one processor, connected to the at least one memory, is configured to execute and implement the distributed task processing method as described in any one of claims 10 to 15 when running the at least one program.

18. A distributed task processing method, characterized in that, include: Receive a second data packet containing aggregation parameters; wherein the aggregation parameters are obtained by the forwarding node by performing an aggregation operation on task parameters in a plurality of first data packets, and the first data packets contain identification information indicating that the aggregation operation is performed by the forwarding node; wherein the plurality of task parameters are obtained by a plurality of worker nodes by executing a distributed computing task; The received second data packet is verified, and the verified second data packet is fed back to the forwarding node, so that the forwarding node can feed back the second data packet containing the aggregation parameters to the corresponding working node.

19. The distributed task processing method according to claim 18, characterized in that, Also includes: Receive multiple first data packets containing task parameters and identification information indicating that the parameter node will perform the aggregation operation; Based on the identification information in the first data packet, an aggregation operation is performed on each task parameter to obtain an aggregation parameter, and a second data packet containing the aggregation parameter is sent to the forwarding node so that the forwarding node can feed back the second data packet containing the aggregation parameter to the corresponding working node.

20. The distributed task processing method according to claim 18, characterized in that, The first data packet also includes task identification information for confirming the distributed computing task corresponding to the first data packet.

21. The distributed task processing method according to claim 18, characterized in that, The first data packet also includes node identification information used to identify the worker node corresponding to the same distributed computing task.

22. The distributed task processing method according to claim 18, characterized in that, The distributed computing task includes a computational task of performing gradient training on a machine learning model using distributed computing.

23. A distributed task processing system, characterized in that, include: A receiving module is configured to receive a second data packet containing aggregation parameters; wherein the aggregation parameters are obtained by a forwarding node performing an aggregation operation on task parameters in multiple first data packets, and the first data packets contain identification information indicating that the aggregation operation is performed by the forwarding node; wherein the task parameters are obtained by multiple worker nodes performing a distributed computing task. The processing module is used to perform a verification operation on the aggregation parameters in the received second data packet, and to feed back the verified second data packet to the forwarding node, so that the forwarding node can feed back the second data packet containing the aggregation parameters to the corresponding working node.

24. A parameter node, characterized in that, include: At least one memory for storing at least one program; At least one processor, connected to the at least one memory, is configured to execute and implement the distributed task processing method as described in any one of claims 18 to 22 when running the at least one program.

25. A distributed task processing system, characterized in that, include: Multiple working nodes as described in claim 9; At least one forwarding node as described in claim 17 is communicatively connected to the working node and is used to perform an aggregation operation based on the task parameters in the first data packet sent by the working node to obtain an aggregation result; the parameter node as described in claim 24 is communicatively connected to the forwarding node and is used to perform an aggregation operation on the aggregation result to obtain an aggregation parameter and to perform a verification operation on the aggregation parameter; Alternatively, at least one forwarding node as described in claim 17 is communicatively connected to the working node and is used to perform an aggregation operation to obtain aggregation parameters based on the task parameters in the first data packet sent by the working node; and the parameter node as described in claim 24 is communicatively connected to the forwarding node and is used to perform a verification operation on the aggregation parameters.

26. The distributed task processing system according to claim 25, characterized in that, The forwarding node and the worker node are connected in TOR mode.

27. The distributed task processing system according to claim 25, characterized in that, The forwarding nodes include primary forwarding nodes and secondary forwarding nodes; wherein, the primary forwarding nodes are communicatively connected to multiple working nodes, and the secondary forwarding nodes are communicatively connected to at least one primary forwarding node.

28. The distributed task processing system according to claim 27, characterized in that, The parameter node is communicatively connected to the secondary forwarding node and at least one working node.

29. A computer-readable storage medium, characterized in that, The system stores at least one program that, when executed by a processor, executes and implements the distributed task processing method as described in any one of claims 1 to 7, or executes and implements the distributed task processing method as described in any one of claims 10 to 15, or executes and implements the distributed task processing method as described in any one of claims 18 to 22.

Images