Method and device for processing asynchronous transmission of messages in a grid computing system

Abstract

The application discloses a processing method and device for asynchronous transmission of messages in a net computing system. The method comprises: obtaining current message sequence numbers of data messages transmitted by each computing node in a communication group to a network device in a same polling cycle; determining whether there is message transmission asynchronism between the computing nodes in the communication group according to an asynchronous proportion coefficient and the current message sequence numbers corresponding to each computing node; the asynchronous proportion coefficient represents an adjustment degree of an upper limit of a progress gap of the computing nodes in the communication group allowed to transmit the data messages; if there is message transmission asynchronism between the computing nodes in the communication group, determining leading nodes and lagging nodes in the communication group according to a leading proportion coefficient, a lagging proportion coefficient and the current message sequence numbers corresponding to each computing node; the leading proportion coefficient is used to determine a leading threshold, and the lagging proportion coefficient is used to determine a lagging threshold; and performing corresponding message transmission congestion control operations on the leading nodes and the lagging nodes respectively.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a processing method and apparatus for asynchronous message transmission in an on-grid computing system. Background Technology

[0002] With the rapid growth of large model parameter scale, single-processor accelerator cards can no longer handle the complete training task. Therefore, it is necessary to use a distributed cluster composed of multiple computing nodes and complete data aggregation calculation in the switch.

[0003] However, when the network link load is uneven, the packet transmission progress of each computing node will differ. Some nodes will be ahead in transmission progress, while others will be behind. Excessive differences in transmission progress will make it difficult for the switch to align data, directly reducing the efficiency of aggregation computing. Summary of the Invention

[0004] In view of this, this application provides a method and apparatus for processing asynchronous message transmission in a network computing system, as follows:

[0005] A method for processing asynchronous message transmission in a network computing system, the method comprising:

[0006] Within the same polling cycle, obtain the current message sequence number of the data packets transmitted from each computing node in the communication group to the network device;

[0007] Based on the asynchronous ratio coefficient and the current message sequence number corresponding to each computing node, it is determined whether there is asynchronous message transmission between computing nodes in the communication group; the asynchronous ratio coefficient represents the degree of adjustment of the upper limit of the progress gap of data message transmission between computing nodes in the communication group.

[0008] If there is asynchronous message transmission among the computing nodes in the communication group, the leading node and the lagging node in the communication group are determined according to the leading ratio coefficient, the lagging ratio coefficient, and the current message sequence number corresponding to each computing node;

[0009] Wherein, the leading ratio coefficient is used to determine the leading threshold, and the lagging ratio coefficient is used to determine the lagging threshold;

[0010] Perform corresponding message transmission congestion control operations on the leading node and the lagging node respectively.

[0011] Optionally, the above method involves determining whether asynchronous message transmission exists between computing nodes within the communication group based on the asynchronous ratio coefficient and the current message sequence number corresponding to each computing node, including:

[0012] Select the maximum and minimum message sequence numbers from the current message sequence numbers corresponding to all the computing nodes;

[0013] Based on the asynchronous ratio coefficient, the maximum message sequence number, and the minimum message sequence number, it is determined whether there is asynchronous message transmission between computing nodes within the communication group.

[0014] Optionally, the above method involves determining whether asynchronous message transmission exists between computing nodes within the communication group based on the asynchronous ratio coefficient, the maximum message sequence number, and the minimum message sequence number, including:

[0015] Obtain the sequence number difference between the maximum message sequence number and the minimum message sequence number;

[0016] If the sequence number difference is greater than a first threshold, it is determined that there is asynchronous message transmission between computing nodes within the communication group;

[0017] If the sequence number difference is less than or equal to the first threshold, it is determined that there is no asynchronous message transmission between the computing nodes in the communication group;

[0018] The first threshold is determined based on the asynchronous scaling factor and the maximum value of the sending window set for the communication group.

[0019] Optionally, the method described above determines the leading and lagging nodes within the communication group based on the leading ratio coefficient, the lagging ratio coefficient, and the current message sequence number corresponding to each computing node, including:

[0020] The current message sequence number corresponding to the computing node is compared with the leading threshold and the lagging threshold respectively;

[0021] If the current message sequence number of a first node in the communication group is greater than or equal to the leading threshold, the first node is determined to be the leading node.

[0022] If the current message sequence number of a second node in the communication group is less than or equal to the lagging threshold, the second node is determined to be a lagging node.

[0023] Wherein, the leading threshold is greater than the lagging threshold.

[0024] Optionally, in the above method, the leading threshold is determined based on the leading ratio coefficient, the maximum value of the sending window, and the minimum message sequence number; the lagging threshold is determined based on the lagging ratio coefficient, the maximum value of the sending window, and the minimum message sequence number.

[0025] Wherein, the maximum value of the sending window is the maximum value of the sending window set for the communication group; the minimum message sequence number is the minimum value among the current message sequence numbers corresponding to the computing node within the communication group; and the leading ratio coefficient is greater than the lagging ratio coefficient.

[0026] Optionally, the above method may involve performing corresponding message transmission congestion control operations on the leading node, including:

[0027] According to the transmission cycle, a first message is sent to the leading node; the first message is used to instruct the leading node to reduce its transmission window;

[0028] The sending period is longer than the polling period.

[0029] Optionally, in the above method, the first message includes a first control field; the first control field records the value of instructing the leading node to reduce the sending window.

[0030] Optionally, the above method may involve performing corresponding message transmission congestion control operations on the lagging node, including:

[0031] A second message is sent to the lagging node, the second message being used to instruct the lagging node to reduce the time interval of the sliding transmission window.

[0032] Optionally, in the above method, the second message is a downlink result message obtained by the network device through aggregation processing of the data packets; the downlink result message includes a valid acknowledgment identifier; the valid acknowledgment identifier is used to instruct the lagging node to reduce the number of accumulated messages;

[0033] The cumulative number of packets is used to trigger the computing node to send a corresponding uplink reply packet to the network device every time it receives a downlink result packet of the cumulative number of packets. In response to the uplink reply packet, the network device sends a downlink reply packet to the computing node. The downlink reply packet is used to trigger the computing node to continue sending the data packets to the network device.

[0034] A processing apparatus for asynchronous message transmission in a network computing system, deployed on a network device or an electronic device connected to the network device, the apparatus comprising:

[0035] The sequence number acquisition unit is used to obtain the current sequence number of the data packets transmitted from each computing node in the communication group to the network device within the same polling cycle;

[0036] An asynchronous determination unit is used to determine whether there is asynchronous message transmission between computing nodes in the communication group based on an asynchronous ratio coefficient and the current message sequence number corresponding to each computing node; the asynchronous ratio coefficient represents the degree of adjustment of the upper limit of the progress gap allowed for data message transmission between computing nodes in the communication group;

[0037] The device determination unit is used to determine the leading node and the lagging node in the communication group based on the leading ratio coefficient, the lagging ratio coefficient and the current message sequence number corresponding to each computing node if there is asynchronous message transmission between computing nodes in the communication group.

[0038] Wherein, the leading ratio coefficient is used to determine the leading threshold, and the lagging ratio coefficient is used to determine the lagging threshold;

[0039] The congestion control unit is used to perform corresponding message transmission congestion control operations on the leading node and the lagging node, respectively.

[0040] As can be seen from the above technical solution, in the asynchronous message transmission processing method and apparatus disclosed in this application for a network computing system, by monitoring whether there are computing nodes in the communication group with asynchronous message transmission, it is possible to filter out leading nodes and lagging nodes when asynchronous transmission is detected, and then perform corresponding congestion control operations on the leading and lagging nodes respectively. It is evident that the congestion control operation in this application can prevent excessive differences in the progress of data message transmission between computing nodes in the communication group, thereby ensuring a high degree of alignment between data messages received on the network device. This reduces the time spent waiting for data messages transmitted by lagging nodes on the network device, allowing for faster aggregation and computation of data messages, thus improving the efficiency of data message aggregation and computation on the network device. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating a method for asynchronous message transmission in an on-grid computing system, provided as an embodiment of this application;

[0043] Figure 2 This is a schematic diagram of the architecture of the online computing system in the embodiments of this application;

[0044] Figure 3A partial flowchart illustrating a method for asynchronous message transmission in an on-grid computing system, provided as an embodiment of this application;

[0045] Figure 4 Another part of the flowchart of a method for processing asynchronous message transmission in an on-grid computing system provided in the embodiments of this application;

[0046] Figure 5 A schematic diagram of the structure of a processing device for asynchronous message transmission in an on-grid computing system provided in an embodiment of this application;

[0047] Figure 6 This is an example diagram illustrating how the on-net computing system in this application accelerates all-reduce set communication to complete gradient aggregation based on an on-net computing accelerator in an embodiment of the application.

[0048] Figure 7 This is a schematic diagram of the interaction of the end-to-end protocol stack in the on-grid computing system in this application embodiment;

[0049] Figure 8 This is a schematic diagram of the congestion control process in the online computing system according to an embodiment of this application. Detailed Implementation

[0050] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0051] refer to Figure 1 This is a flowchart illustrating the implementation of an asynchronous message transmission processing method in an on-grid computing system, as provided in this application embodiment. This method can be applied to network devices or electronic devices connected to network devices in an on-grid computing system, such as… Figure 2As shown, the on-grid computing system includes network devices and multiple computing nodes. The network devices can be switch devices containing switching chips and on-grid computing accelerators. The on-grid computing accelerators are connected to the switching chips to accelerate aggregated communication. The on-grid computing accelerators can be field-programmable gate arrays (FPGAs). Each computing node can be a computing node containing a graphics processing unit (GPU) accelerator card. Multiple computing nodes synchronize data and state information through aggregated communication primitives; for example, the aggregated communication primitive can be an all-reduce primitive. Multiple computing nodes form a distributed computing cluster to jointly complete the computational tasks of an artificial intelligence model (such as training tasks). Specifically, each computing node performs partial data computation of the computational task and constructs the computational results into data packets, which are then sent to the network devices. The network devices utilize the on-grid computing accelerators within them to perform accelerated aggregated computation on the data packets sent by all computing nodes, thereby obtaining the task execution result. The technical solution in this embodiment is mainly used to improve the efficiency of aggregated computation of data packets on the network devices.

[0052] Specifically, the method in this embodiment may include the following steps:

[0053] Step 101: Within the same polling cycle, obtain the current message sequence number of the data packets transmitted from each computing node in the communication group to the network device.

[0054] In the in-network computing system, a polling cycle for congestion control is pre-set, and the actual duration of the polling cycle can be set according to the business scenario. For example, the polling cycle can be 10 microseconds. In each polling cycle, the in-network computing system continuously records the latest message sequence number, i.e., the current message sequence number, of all uplink data packets received by all communication processes in the communication group (i.e., data packets transmitted from all computing nodes to network devices).

[0055] In addition, a communication group refers to a communication set consisting of multiple computing nodes participating in the same distributed computing task in a network computing system and needing to synchronize data and status information through collective communication.

[0056] It should be noted that in a communication group, each data packet transmitted by a computing node to a network device carries a packet sequence number (PSN), or message sequence number. The PSN identifies the sequential position of the data packet within the message sequence sent by its respective computing node. Based on this, network devices such as switches can receive data packets sent by each computing node and record the latest PSN for each computing node, i.e., the current message sequence number, which can be denoted as the latest packet sequence number (LPSN). The size of the PSN indicates the progress of the computing node in sending data packets to the network device; a larger PSN indicates that the corresponding computing node is ahead in transmitting data packets, and a smaller PSN indicates that the corresponding computing node is behind in transmitting data packets.

[0057] For example, there are four computing nodes in the communication group: node 1, node 2, node 3, and node 4. Each computing node sends data packets to the network device according to its own sending window. Each data packet carries a PSN. On the network device or its connected electronic device, the PSN of the most recently sent data packet from node 1 is 2356, the PSN of the most recently sent data packet from node 2 is 2354, the PSN of the most recently sent data packet from node 3 is 2359, and the PSN of the most recently sent data packet from node 4 is 2342. These PSNs represent the progress of the corresponding node in transmitting data packets. Thus, node 3 is the most advanced and node 4 is the least advanced.

[0058] Step 102: Based on the asynchronous ratio coefficient and the current message sequence number corresponding to each computing node, determine whether there is asynchronous message transmission between computing nodes in the communication group. If there is no asynchronous message transmission between computing nodes in the communication group, continue to execute step 101 until there is asynchronous message transmission between computing nodes in the communication group, and then execute step 103.

[0059] The asynchronous scaling factor, represented by ASYNC_RATIO, characterizes the degree to which the upper limit of the allowable time difference in data packet transmission between computing nodes within a communication group can be adjusted. Specifically, it can be understood as adjusting the sensitivity to the asynchronous nature of packet transmission between computing nodes within the communication group. For example, a smaller asynchronous scaling factor means greater sensitivity to asynchronous packet transmission between computing nodes within the communication group, making it easier to trigger subsequent filtering of leading and lagging nodes and congestion control operations. Conversely, a larger asynchronous scaling factor indicates a higher tolerance for asynchronous packet transmission between computing nodes within the communication group.

[0060] Specifically, the value of the asynchronous proportional coefficient can be set according to the network conditions of the on-network computing system and the performance requirements of the network equipment.

[0061] In one implementation, after obtaining the current message sequence number of the latest data packet transmitted to the network device by each computing node within the same polling cycle, the message transmission progress gap between each computing node in the communication group is calculated based on the current message sequence number corresponding to each computing node. This progress gap is measured by comparing the sequence number differences between the current message sequence numbers of different computing nodes. For example, the difference between the current message sequence numbers of any two computing nodes can reflect the degree of lead or lag of these two computing nodes in message transmission. Based on this, this embodiment compares the obtained progress gap with the asynchronous tolerance level of message transmission represented by the asynchronous ratio coefficient to determine whether the current asynchronous level of message transmission has exceeded the acceptable asynchronous range represented by the asynchronous ratio coefficient.

[0062] Step 103: Determine the leading and lagging nodes in the communication group based on the leading ratio coefficient, the lagging ratio coefficient, and the current message sequence number corresponding to each computing node;

[0063] The leading ratio coefficient, represented by `LEADING_RATIO`, ranges from 0 to 1, such as 0.75. It is used to determine the leading threshold. The lagging ratio coefficient, represented by `LAGGING_RATIO`, also ranges from 0 to 1, such as 0.25. It is used to determine the lagging threshold. The values ​​of both the leading and lagging ratio coefficients determine the number of computing nodes marked as leading or lagging nodes.

[0064] For example, the larger the value of LEADING_RATIO, the fewer computing nodes are marked as leading nodes; the smaller the value of LAGGING_RATIO, the fewer computing nodes are marked as lagging nodes.

[0065] Correspondingly, the leading threshold can characterize the lower limit of the progress gap in message transmission between the leading nodes in the communication group and the lagging threshold can characterize the upper limit of the progress gap in message transmission between the lagging nodes in the communication group.

[0066] Specifically, in this embodiment, computing nodes whose message transmission progress exceeds the leading threshold within the communication group are identified as leading nodes, and computing nodes whose message transmission progress has not reached the lagging threshold are identified as lagging nodes.

[0067] Step 104: Perform the corresponding message transmission congestion control operations for the leading node and the lagging node respectively.

[0068] The message transmission congestion control operations for the leading node are different from those for the lagging node.

[0069] Specifically, in this embodiment, congestion control messages (or congestion control instructions) can be sent to the leading node and the lagging node respectively to instruct the corresponding computing nodes to perform corresponding message transmission congestion control operations.

[0070] As can be seen from the above technical solution, in the asynchronous message transmission processing method provided by the embodiments of this application, by monitoring whether there are computing nodes in the communication group with asynchronous message transmission, it is possible to filter out leading nodes and lagging nodes when asynchronous transmission is detected, and then perform corresponding congestion control operations on the leading and lagging nodes respectively. It is evident that the congestion control operation in this application can prevent excessive differences in the progress of data packet transmission between computing nodes in the communication group, thereby ensuring a high degree of alignment between data packets received on the network device. This prevents the network device from spending excessive time waiting for data packets transmitted by lagging nodes, allowing for faster aggregation and computation of data packets on the network device, thus improving the efficiency of data packet aggregation and computation on the network device.

[0071] In one implementation, step 102 can determine whether asynchronous message transmission exists between computing nodes within the communication group in the following way: Figure 3 As shown:

[0072] Step 301: Select the maximum and minimum message sequence numbers from the current message sequence numbers corresponding to all computing nodes.

[0073] The maximum message sequence number can be represented by LPSN_MAX, and the minimum message sequence number can be represented by LPSN_MIN.

[0074] For example, there are 4 computing nodes in the communication group: node 1 (latest PSN is 2356), node 2 (latest PSN is 2354), node 3 (latest PSN is 2359) and node 4 (latest PSN is 2342). The largest PSN is selected as 2359 (node ​​3) and the smallest PSN is selected as 2342 (node ​​4).

[0075] The largest message sequence number corresponds to the computing node with the fastest current transmission progress in the communication group, while the smallest message sequence number corresponds to the computing node with the slowest current transmission progress in the communication group.

[0076] Specifically, the sequence number difference between the maximum and minimum message sequence numbers can characterize the timing offset between the fastest and slowest computing nodes in the communication group, and can comprehensively reflect the overall asynchronous nature of message transmission within the communication group.

[0077] Step 302: Determine whether there is asynchronous message transmission between computing nodes in the communication group based on the asynchronous ratio coefficient, the maximum message sequence number, and the minimum message sequence number.

[0078] In one implementation, the asynchronous scaling factor can be a value between 0 and 1. In this embodiment, the maximum value of the sending window corresponding to the communication group can be used as a basis, and the maximum value of the sending window can be adjusted using the asynchronous scaling factor. The adjusted value can then be used as the tolerance threshold (such as a first threshold) for asynchronous message transmission.

[0079] Based on this, in this embodiment, the sequence number difference between the maximum and minimum message sequence numbers is compared with the above tolerance threshold, and then the comparison result is used to determine whether there is asynchronous message transmission between computing nodes in the communication group.

[0080] As can be seen, by selecting the two extreme nodes with the fastest and slowest transmission progress within the communication group in this embodiment, the overall asynchronous degree of the communication group can be fully reflected with minimal computational overhead. This avoids the computational complexity caused by comparing the progress differences between all pairs of computing nodes one by one, which is beneficial for quickly completing asynchronous judgment during the execution of distributed training tasks and improving the efficiency of congestion control.

[0081] In one implementation, step 302 can be achieved in the following way:

[0082] First, obtain the sequence number difference between the maximum and minimum message sequence numbers, which can be represented as LPSN_MAX-LPSN_MIN. Then, compare the sequence number difference with the first threshold corresponding to the asynchronous scaling factor. If the sequence number difference is greater than the first threshold, it is determined that there is asynchronous message transmission between computing nodes in the communication group; if the sequence number difference is less than or equal to the first threshold, it is determined that there is no asynchronous message transmission between computing nodes in the communication group.

[0083] The first threshold serves as the tolerance threshold for asynchrony, and its value is jointly determined by the asynchrony scaling factor and the maximum value of the sending window. The smaller the asynchrony scaling factor, the smaller the first threshold, and the more sensitive the on-network computing system is to the asynchrony of message transmission; the larger the asynchrony scaling factor, the higher the first threshold, and the greater the tolerance of the on-network computing system for asynchrony in message transmission.

[0084] Specifically, the first threshold can be determined based on the asynchronous scaling factor and the maximum sending window value set for the communication group. The maximum sending window value can be represented by WIN_MAX. This maximum sending window value can be set by the software controller in the network device for each communication group during the initialization of the on-grid computing system. The maximum sending window value refers to the largest sending window size allowed for any computing node within the communication group. The sending window of a computing node refers to the maximum number of data packets that a computing node is allowed to send to the network device before receiving an acknowledgment (ACK) message from the network device. The maximum sending window value can be configured based on factors such as network bandwidth, transmission latency, and buffer depth in the on-grid computing system.

[0085] For example, the first threshold is the product of the asynchronous scaling factor and the maximum value of the sending window. Specifically, the first threshold can be ASYNC_RATIO × WIN_MAX.

[0086] For example, taking an asynchronous ratio coefficient of 0.5 as an example, the maximum value of the sending window in the communication group is 20, and the corresponding first threshold is 10. Based on this, in this embodiment, the difference between the maximum PSN and the minimum PSN is calculated to obtain a sequence number difference of 17. By comparing the sequence number difference of 17 with the first threshold of 10, it can be determined that the progress difference of data packet transmission between node 3 corresponding to the maximum PSN and node 4 corresponding to the minimum PSN exceeds the upper limit of 10, that is, there is asynchronous packet transmission in the communication group.

[0087] As can be seen, in this embodiment, the judgment threshold, such as the first threshold, can be dynamically determined by using the asynchronous proportional coefficient as an adjustable parameter. This allows the on-network computing system to flexibly adjust its sensitivity to asynchronicity, i.e., the asynchronous proportional coefficient, which is beneficial for implementing corresponding congestion control in different business scenarios and improves the reliability of congestion control in this embodiment.

[0088] In one implementation, step 103, when determining the leading and lagging nodes within the communication group, can be achieved in the following way: Figure 4 As shown:

[0089] Step 401: Compare the current message sequence number corresponding to the computing node with the leading threshold and the lagging threshold respectively; if the current message sequence number corresponding to the first node in the communication group is greater than or equal to the leading threshold, proceed to step 402; if the current message sequence number corresponding to the second node in the communication group is less than or equal to the lagging threshold, proceed to step 403.

[0090] Among them, the leading threshold is greater than the lagging threshold.

[0091] It should be noted that the threshold difference between the leading threshold and the lagging threshold limits the normal fluctuation range of message transmission progress among computing nodes within the communication group. In other words, if a third node within the communication group has a current message sequence number lower than the leading threshold but higher than the lagging threshold, this third node can be considered a computing node with normal message transmission progress, or simply a regular node.

[0092] Step 402: Determine the first node as the leading node;

[0093] Among them, the leading node refers to a computing node whose message transmission progress is significantly ahead of other computing nodes in the communication group, and whose data message sending rate is too high and needs to be limited to alleviate congestion.

[0094] Step 403: Determine that the second node is a lagging node;

[0095] Among them, a lagging node refers to a computing node whose message transmission progress is significantly behind that of other computing nodes in the communication group, whose data message sending rate is too low and needs to be accelerated to catch up with the transmission progress.

[0096] In one implementation, the specific values ​​of the leading and lagging thresholds can be adjusted according to the actual operating status of the communication group. A higher leading threshold results in stricter conditions for being marked as a leading node, and fewer leading nodes are marked; a lower leading threshold results in more lenient conditions for being marked as a leading node, and more leading nodes are marked. Conversely, a higher lagging threshold results in more lenient conditions for being marked as a lagging node, and more lagging nodes are marked; a lower lagging threshold results in more strict conditions for being marked as a lagging node, and fewer lagging nodes are marked.

[0097] As can be seen, this embodiment creates a normal message transmission progress range between the leading and lagging thresholds by setting the leading threshold to be greater than the lagging threshold. This allows computing nodes whose message transmission progress is within this range to avoid triggering differentiated congestion control operations, while computing nodes outside this range perform corresponding congestion control operations. This helps avoid unnecessary control interventions on normal computing nodes, reducing system overhead, while ensuring that only computing nodes that truly deviate from the normal transmission progress are marked and have their corresponding congestion control operations performed, thus improving the accuracy of congestion control.

[0098] Based on the above implementation schemes, in one specific implementation method, the leading threshold can be determined based on the leading ratio coefficient, the maximum window value, and the minimum message sequence number; the lagging threshold can be determined based on the lagging ratio coefficient, the maximum window value, and the minimum message sequence number.

[0099] The maximum sending window value is the maximum sending window value set for the communication group, i.e., WIN_MAX. The minimum message sequence number is the minimum current message sequence number corresponding to the computing node within the communication group, i.e., LPSN_MIN. The lead ratio coefficient is greater than the lag ratio coefficient, which in turn makes the lead threshold greater than the lag threshold.

[0100] Specifically, in this embodiment, the maximum value of the sending window is adjusted based on the minimum message sequence number and superimposed on the minimum message sequence number to obtain the leading threshold; and the maximum value of the sending window is adjusted based on the minimum message sequence number and superimposed on the minimum message sequence number to obtain the lagging threshold.

[0101] For example, the leading threshold can be LEADING_RATIO × WIN_MAX + LPSN_MIN. Similarly, the lagging threshold can be LAGGING_RATIO × WIN_MAX + LPSN_MIN.

[0102] As can be seen, by establishing both the leading threshold and the lagging threshold on the basis of the minimum message sequence number in this embodiment, the two thresholds can dynamically change with the message transmission progress of the computing node with the slowest transmission progress in the communication group. This can help adapt to the fluctuations in the overall message transmission progress of the communication group in different polling cycles, avoid misjudging or missing the leading / lagging node in different polling cycles due to fixed thresholds, and improve the accuracy of marking leading and lagging nodes.

[0103] In one implementation, when performing the corresponding message transmission congestion control operation on the leading node in step 104, the first message can be sent to the leading node according to the sending period.

[0104] The first message is used to instruct the leading node to reduce its sending window. The sending period is longer than the polling period.

[0105] It should be noted that in this embodiment, for the computing node marked as the leading node, the first message is periodically sent to the leading node according to a preset sending period. This sending period is longer than the aforementioned polling period, that is, the first message is sent only once every multiple polling periods. This is to avoid introducing additional network overhead due to the first message being sent too frequently. At the same time, it allows the leading node sufficient time to adjust the sending window, thereby improving the reliability of congestion control.

[0106] Specifically, the first message is used to instruct the leading node to reduce its sending window. Upon receiving the first message, the leading node reduces its sending window in response to the first message, thereby reducing its sending bandwidth. Thus, by periodically sending the first message, the leading node's sending window can be continuously reduced until its transmission progress is detected to have fallen back to the range of normal message transmission progress in a new polling cycle, that is, it changes from a leading node to a normal node.

[0107] For example, the first message can be a congestion notification packet (CNP). In this embodiment, a minimum interval for sending CNP messages can be set as the sending period. For example, the sending period can be 50 microseconds. In this embodiment, a timer can be used to implement the function of sending CNP messages periodically according to the sending period. For example, when the timer reaches the sending period, a CNP message is sent to the leading node. The CNP message is used to notify the leading node that there is congestion in the current network. After receiving the CNP message, the leading node reduces its sending window in response to the CNP message.

[0108] As can be seen, by sending the first message only once every multiple polling cycles in this embodiment, the rate of message transmission to the leading node can be effectively suppressed, while avoiding the introduction of additional network bandwidth and computing resource consumption due to the excessive frequency of sending the first message. This is conducive to achieving a good balance between congestion control effect and network overhead, and improving the overall operating efficiency of the on-network computing system.

[0109] Based on the above implementation scheme, in the specific implementation method, the first message includes a first control field; the first control field records the instructions for the leading node to reduce the sending window.

[0110] As can be seen, in this embodiment, by carrying a control field to reduce the sending window in the first message, the leading node can reduce the sending window according to its own control logic, which is beneficial to ensuring the data packet transmission service on the leading node side.

[0111] In another specific implementation, the first message includes a first control field; the first control field records the value that instructs the leading node to reduce the sending window.

[0112] Based on this, after the leading node receives the first message, it parses the first control field in the first message to obtain the value of the sending window that needs to be reduced, and then reduces its own sending window according to the value.

[0113] For example, the first control field can be carried in the payload portion of a CNP message. The CNP message can adopt the congestion notification message encapsulation format of the corresponding transport protocol; for example, a CNP message may include fields such as an Ethernet header, CNP payload, and frame check sequence (FCS). In the CNP payload portion, one or more reserved fields or custom fields can be set as the first control field to carry the value of the transmission window that needs to be reduced.

[0114] In one implementation, the value recorded in the first control field can be the absolute target value of the sending window. After the leading node receives the first message, it directly adjusts the sending window to this target value.

[0115] In another implementation, the value recorded in the first control field can be the reduction step size of the sending window. After the leading node receives the data, it subtracts this step size from the current sending window.

[0116] In another implementation, the value recorded in the first control field can be a reduction ratio, and the leading node reduces the sending window according to this ratio after receiving it.

[0117] As can be seen, in this embodiment, by explicitly carrying the specific value of the sending window to be reduced in the first message, the leading node can accurately know the adjustment range of the sending window. This can avoid the problem that may be caused by different computing nodes having inconsistent understandings of the reduction range and inconsistent control effects (i.e., control logic) due to only qualitative instructions (such as only indicating "reduce"). It is beneficial to achieve precise control of the sending rate of the leading node and improve the fineness of congestion control.

[0118] In one implementation, when performing the corresponding message transmission congestion control operation on the lagging node in step 104, a second message can be sent to the lagging node.

[0119] The second message is used to instruct the lagging node to reduce the time interval of the sliding transmission window.

[0120] It should be noted that the sliding of the lagging node's transmission window depends on receiving downlink result packets (i.e., the result data packets obtained after aggregation calculation) from the network device. In normal operation, the lagging node only triggers the sending of an uplink ACK packet to the network device after receiving a certain number of downlink result packets. The network device responds to this uplink ACK packet by sending a downlink ACK packet to the lagging node, and the lagging node only slides its transmission window after receiving the downlink ACK packet. In this process, because the lagging node needs to accumulate a certain number of downlink result packets before triggering ACK feedback, the frequency of transmission window sliding is low, limiting the transmission rate. Therefore, in this embodiment, a second packet is sent to the lagging node, instructing it to reduce the time interval of the sliding transmission window, i.e., instructing the lagging node to trigger transmission window sliding within a shorter time interval, thereby improving its transmission rate.

[0121] As can be seen, by reducing the time interval of the sliding sending window in this embodiment, the lagging node can trigger the sliding of the sending window at a shorter period, thereby sending uplink data packets to the network device more frequently. This avoids the transmission progress from continuously falling behind due to the long stagnation time of the sending window, which helps to improve the sending rate of the lagging node as soon as possible and catch up with the packet transmission progress of other computing nodes, thereby improving the overall packet transmission synchronization of the communication group.

[0122] Based on the above implementation scheme, in the specific implementation method, the second message is the downlink result message obtained by the network device through the aggregation processing of data packets (i.e., uplink data packets sent by the computing node to the network device).

[0123] The downlink result message includes a valid response identifier; this identifier instructs lagging nodes to reduce the accumulated message count. The accumulated message count triggers the compute node to send a corresponding uplink reply message to the network device for each downlink result message received. In response to the uplink reply message, the network device sends a downlink reply message to the compute node, which triggers the compute node to continue sending data packets to the network device.

[0124] For example, the second message can reduce the time interval of the sliding transmission window by instructing the lagging node to reduce the number of accumulated messages required to trigger the uplink ACK message feedback. The lower the number of accumulated messages, the more frequently the lagging node triggers the uplink ACK message feedback, the higher the frequency of the transmission window sliding, and the faster the transmission rate.

[0125] It should be noted that the lagging node maintains the parameter of the cumulative message count locally. After the second message is received by the lagging node, the lagging node reduces the cumulative message count locally to the target value or by one adjustment compensation, according to the reduction of the cumulative message count indicated by the valid acknowledgment flag in the second message. This reduces the cumulative message count, and the lagging node triggers the transmission of uplink reply messages when it receives a smaller number (e.g., each) of downlink result messages, thereby obtaining more frequent downlink reply messages and achieving a more timely sliding of the transmission window.

[0126] For example, the valid acknowledgment flag in the second message can be implemented by setting the Acknowledgment Request bit in the second message. For instance, setting the Acknowledgment Request bit to 1 allows the lagging node to trigger an uplink ACK message for each downlink result message received, i.e., triggering the sending of uplink ACK messages and the return of downlink ACK messages packet by packet, thereby enabling timely sliding of the lagging node's sending window. If the Acknowledgment Request bit is 0, it means that the lagging node can follow the default strategy, accumulating multiple downlink result messages before sending an uplink ACK message, instead of immediately sending an uplink ACK message.

[0127] For example, by using the valid acknowledgment flag in the second message, the lagging node can reduce its waiting time from 5 downlink result messages before sending an uplink ACK message to waiting for 2 downlink result messages before sending an uplink ACK message. The network device can send back a downlink ACK message as soon as it receives an uplink ACK message. As a result, the compute node, which is the lagging node, can send an uplink data message as soon as it receives two downlink ACK messages, thereby speeding up the transmission of uplink data messages and enabling faster sliding of the transmission window.

[0128] As can be seen, in this embodiment, a valid acknowledgment identifier is carried in the downlink result message of the network device to instruct the lagging node to reduce the number of accumulated messages. Thus, by reusing the existing downlink result message to carry the indication information of congestion control operation, there is no need to design new signaling messages, which helps to reduce the complexity of transmission protocol modification and signaling overhead.

[0129] Moreover, by reducing the number of accumulated messages in this embodiment, the lagging node can trigger the transmission of uplink reply messages after receiving fewer downlink result messages, thereby obtaining more frequent downlink reply messages to achieve more timely sliding of the transmission window. This is beneficial for fine-grained control of the transmission window sliding frequency of the lagging node, enabling it to catch up with the transmission progress as soon as possible.

[0130] refer to Figure 5This is a schematic diagram of a processing device for asynchronous message transmission in a network computing system provided in an embodiment of this application. The device can be deployed on a network device or an electronic device connected to the network device, and may include the following structure:

[0131] The sequence number acquisition unit 501 is used to obtain the current message sequence number of the data packets transmitted from each computing node in the communication group to the network device within the same polling cycle;

[0132] The asynchronous determination unit 502 is used to determine whether there is asynchronous message transmission between computing nodes in the communication group based on the asynchronous ratio coefficient and the current message sequence number corresponding to each computing node; the asynchronous ratio coefficient represents the degree of adjustment of the upper limit of the progress gap of data message transmission between computing nodes in the communication group.

[0133] The device determination unit 503 is used to determine the leading node and the lagging node in the communication group based on the leading ratio coefficient, the lagging ratio coefficient and the current message sequence number corresponding to each computing node if there is asynchronous message transmission between computing nodes in the communication group.

[0134] Wherein, the leading ratio coefficient is used to determine the leading threshold, and the lagging ratio coefficient is used to determine the lagging threshold;

[0135] The congestion control unit 504 is used to perform corresponding message transmission congestion control operations on the leading node and the lagging node respectively.

[0136] As can be seen from the above technical solution, in the processing device for asynchronous message transmission in a network computing system provided in this application embodiment, by monitoring whether there are computing nodes in the communication group with asynchronous message transmission, it is possible to filter out leading nodes and lagging nodes when asynchronous transmission is detected, and then perform corresponding congestion control operations on the leading and lagging nodes respectively. It is evident that the congestion control operation in this application can prevent excessive differences in the progress of data packet transmission between computing nodes in the communication group, thereby ensuring a high degree of alignment between data packets received on the network device. This reduces the time spent waiting for data packets transmitted by lagging nodes on the network device, allowing for faster aggregation and computation of data packets, thus improving the efficiency of data packet aggregation and computation on the network device.

[0137] In one implementation, the asynchronous determination unit 502 is specifically used to: select the maximum message sequence number and the minimum message sequence number from the current message sequence numbers corresponding to all the computing nodes; and determine whether there is asynchronous message transmission between the computing nodes in the communication group based on the asynchronous ratio coefficient, the maximum message sequence number, and the minimum message sequence number.

[0138] Specifically, when the asynchronous determination unit 502 determines whether there is asynchronous message transmission among computing nodes in the communication group based on the asynchronous ratio coefficient, the maximum message sequence number, and the minimum message sequence number, it is specifically used to: obtain the sequence number difference between the maximum message sequence number and the minimum message sequence number; if the sequence number difference is greater than a first threshold, determine that there is asynchronous message transmission among computing nodes in the communication group; if the sequence number difference is less than or equal to the first threshold, determine that there is no asynchronous message transmission among computing nodes in the communication group; wherein, the first threshold is determined based on the asynchronous ratio coefficient and the maximum value of the sending window set for the communication group.

[0139] In one implementation, the device determination unit 503 is specifically configured to: compare the current message sequence number corresponding to the computing node with the leading threshold and the lagging threshold respectively; if the current message sequence number corresponding to a first node in the communication group is greater than or equal to the leading threshold, determine the first node as the leading node; if the current message sequence number corresponding to a second node in the communication group is less than or equal to the lagging threshold, determine the second node as the lagging node; wherein the leading threshold is greater than the lagging threshold.

[0140] Wherein, the leading threshold is determined based on the leading ratio coefficient, the maximum sending window value, and the minimum message sequence number; the lagging threshold is determined based on the lagging ratio coefficient, the maximum sending window value, and the minimum message sequence number; the maximum sending window value is the maximum sending window value set for the communication group; the minimum message sequence number is the minimum value among the current message sequence numbers corresponding to the computing node within the communication group; and the leading ratio coefficient is greater than the lagging ratio coefficient.

[0141] In one implementation, when the congestion control unit 504 performs a corresponding message transmission congestion control operation on the leading node, it is specifically configured to: send a first message to the leading node according to a sending period; the first message is used to instruct the leading node to reduce its sending window; wherein the sending period is greater than the polling period.

[0142] The first message includes a first control field; the first control field records the value of instructing the leading node to reduce the sending window.

[0143] In one implementation, when the congestion control unit 504 performs the corresponding message transmission congestion control operation on the lagging node, it is specifically used to: send a second message to the lagging node, the second message being used to instruct the lagging node to reduce the time interval of the sliding transmission window.

[0144] Wherein, the second message is a downlink result message obtained by the network device after aggregating the data packets; the downlink result message includes a valid response identifier; the valid response identifier is used to instruct the lagging node to reduce the number of accumulated packets; the number of accumulated packets is used to trigger the computing node to send a corresponding uplink reply message to the network device every time it receives the downlink result message of the number of accumulated packets, and the network device sends a downlink reply message to the computing node in response to the uplink reply message, and the downlink reply message is used to trigger the computing node to continue sending the data packets to the network device.

[0145] It should be noted that the specific implementation of each unit in this embodiment can be referred to the corresponding content above, and will not be described in detail here.

[0146] by Figure 2 Taking the online computing system shown below as an example, the technical solution of this application will be described in detail below:

[0147] First, such as Figure 6 The diagram shown illustrates an example of gradient aggregation performed by an on-grid computing system using an on-grid computing accelerator to accelerate all-reduce set communication. The specific steps are as follows:

[0148] (1) Each computing node (Worker) packages the computation result data generated in one round of training into n uplink data packets (Request packets), namely packets P1 to Pn, each packet having k 32-bit floating point numbers (FP32).

[0149] (2) Each worker sends s data packets (P1 to) to the switch (specifically the switching chip in the switch), which then forwards them to the on-network computing accelerator (FPGA accelerator).

[0150] (3) The on-net computing accelerator receives and records all worker data packets and performs aggregate calculations.

[0151] (4) The on-net computing accelerator aggregates the Worker's data packets to generate s new downlink result messages (Response data packets).

[0152] (5) The on-net computing accelerator sends the newly generated s Response data packets to the switch, which then broadcasts them to all Workers.

[0153] (6) All Workers run synchronously, and after completing the aggregation calculation of a batch of s data packets, they perform the aggregation calculation of the next batch of s data packets.

[0154] Based on the above process, such as Figure 7 This diagram illustrates the interaction of the end-to-end protocol stack in a network computing system. Distributed training processes across multiple computing nodes form a communication group. When a member of the communication group calls All Reduce, the workflow is as follows:

[0155] (1) The training process calls the All Reduce interface to pass the starting address and length of the data to be transmitted and reduced to the protocol stack.

[0156] (2) The protocol stack divides the data into several messages and organizes them into a message sequence.

[0157] (3) The protocol stack maintains a sliding window and calls the Remote Direct Memory Access Verbs Interface (RDMA Verbs) to divide the data in the sliding window into packets and send them out, such as transmitting to the RDMA network card of the computing node via RDMA write.

[0158] (4) The RDMA network card of the computing node divides the message passed by RDMA write into uplink data packets and sends them to the switch. Each uplink data packet has its own PSN.

[0159] (5) When the on-grid computing accelerator receives an uplink data packet, it performs an aggregation operation and writes the aggregation result back to form a downlink result packet. The packet is then handed over to the switching chip for forwarding. The switching chip copies the packet and sends it to each computing node. In addition, the on-grid computing accelerator determines whether it is a leading or lagging node based on the latest PSN in the uplink data packets sent by each computing node. If it is a leading node, it determines the CNP sending frequency, such as by sending CNP to the computing node to reduce the sending window. If it is a lagging node, it determines the ACK bit setting, such as setting it to 1, so that the lagging node triggers (reflects) the uplink ACK packet sending packet by packet.

[0160] (6) The RDMA network card of the computing node receives the downlink result message and sends an uplink ACK message.

[0161] (7) After receiving the uplink ACK message, the on-grid computing accelerator returns along the same path and generates a downlink ACK message, which is called ACK reflection. Before returning, the source Internet Protocol (IP), destination IP, and destination queue pair (QP) are modified so that the original ACK sender receives the new ACK.

[0162] (8) After receiving the downlink ACK message, the compute node performs window sliding and acknowledgment message reception (such as the downlink ACK message of the last packet in the message triggers the generation of a completion queue entry (CQE)) and writes the all reduce aggregation result to the result buffer.

[0163] Based on the above process, such as Figure 8 The diagram illustrates the workflow of congestion control in a network computing system. When congestion occurs in the network, and the packet transmission progress of some computing nodes falls behind that of other nodes, a congestion event is triggered. Figure 8 The congestion control method shown is as follows:

[0164] The network computing system pre-defines a congestion control polling period (e.g., 10 microseconds) to poll and record the current PSN (Passive Number Subsequent Name) of uplink data packets received by all communication processes within the communication group, and performs asynchronous determination of the all-reduce aggregation calculation operation. Specifically, the maximum LPSN value for each communication process within the communication group is recorded as LPSN_MAX, and the minimum LPSN value is recorded as LPSN_MIN. For example... Figure 2 The software controller of the on-grid computing system presets a maximum transmission window value for a computing node during system initialization, denoted as WIN_MAX.

[0165] Based on this, if LPSN_MAX - LPSN_MIN is less than or equal to ASYNC_RATIO × WIN_MAX, then the aggregation asynchrony is determined to be "No", and the traditional DCQCN algorithm is used for congestion control. If LPSN_MAX - LPSN_MIN is greater than ASYNC_RATIO × WIN_MAX, then the aggregation asynchrony is determined to be "Yes", and further marking of leading and lagging nodes is required.

[0166] The specific process for marking leading and lagging nodes is as follows:

[0167] If the current PSN (denoted as LPSN_CURRENT) of the current communication process is greater than or equal to LEADING_RATIO×WIN_MAX+LPSN_MIN, then it is determined to be the leading node;

[0168] If the LPSN_CURRENT of the current communication process is less than or equal to LAGGING_RATIO×WIN_MAX+LPSN_MIN, it is judged as a lagging node.

[0169] If the compute node is neither a leading node nor a lagging node, then congestion control is performed using an algorithm based on Data Center Quantized Congestion Notification.

[0170] Based on this, for the leading node, a minimum interval (e.g., 50 microseconds) for sending Congestion Notification Messages (CNP) is preset, and CNP messages are sent periodically using a timer. By rapidly reducing the sending window of the leading node, the transmission bandwidth is reduced.

[0171] For lagging nodes, the Acknowledge Request bit in the header of the RDMA message is set (e.g., set to 1), so that the lagging node triggers the uplink ACK message to be sent packet by packet, so as to realize the timely sliding of the sending window.

[0172] In summary, the technical solution of this application has the following advantages:

[0173] 1. This application can improve the congestion control response accuracy and control granularity of on-grid computing accelerators, and the collected indicators are inherent during transmission, and the measurement method is simple, enabling real-time control.

[0174] 2. This application can improve the efficiency of on-net computing to accelerate all reduce aggregation, and ultimately improve the performance of the overall distributed training system.

[0175] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0176] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0177] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0178] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for processing asynchronous message transmission in a network computing system, the method comprising: Within the same polling cycle, obtain the current message sequence number of the data packets transmitted from each computing node in the communication group to the network device; Based on the asynchronous ratio coefficient and the current message sequence number corresponding to each computing node, it is determined whether there is asynchronous message transmission between computing nodes in the communication group; The asynchronous scaling factor characterizes the degree of adjustment of the upper limit of the progress gap between computing nodes in the communication group. If there is asynchronous message transmission among the computing nodes in the communication group, the leading node and the lagging node in the communication group are determined according to the leading ratio coefficient, the lagging ratio coefficient, and the current message sequence number corresponding to each computing node; Wherein, the leading ratio coefficient is used to determine the leading threshold, and the lagging ratio coefficient is used to determine the lagging threshold; Perform corresponding message transmission congestion control operations on the leading node and the lagging node respectively.

2. The method according to claim 1, determining whether there is asynchronous message transmission between computing nodes in the communication group based on the asynchronous ratio coefficient and the current message sequence number corresponding to each computing node, includes: Select the maximum and minimum message sequence numbers from the current message sequence numbers corresponding to all the computing nodes; Based on the asynchronous ratio coefficient, the maximum message sequence number, and the minimum message sequence number, it is determined whether there is asynchronous message transmission between computing nodes within the communication group.

3. The method according to claim 2, determining whether there is asynchronous message transmission between computing nodes within the communication group based on the asynchronous ratio coefficient, the maximum message sequence number, and the minimum message sequence number, includes: Obtain the sequence number difference between the maximum message sequence number and the minimum message sequence number; If the sequence number difference is greater than a first threshold, it is determined that there is asynchronous message transmission between computing nodes within the communication group; If the sequence number difference is less than or equal to the first threshold, it is determined that there is no asynchronous message transmission between the computing nodes in the communication group; The first threshold is determined based on the asynchronous scaling factor and the maximum value of the sending window set for the communication group.

4. The method according to claim 1, wherein determining the leading node and lagging node within the communication group based on the leading ratio coefficient, the lagging ratio coefficient, and the current message sequence number corresponding to each computing node, includes: The current message sequence number corresponding to the computing node is compared with the leading threshold and the lagging threshold respectively; If the current message sequence number of a first node in the communication group is greater than or equal to the leading threshold, the first node is determined to be the leading node. If the current message sequence number of a second node in the communication group is less than or equal to the lagging threshold, the second node is determined to be a lagging node. Wherein, the leading threshold is greater than the lagging threshold.

5. The method according to claim 4, wherein the leading threshold is determined based on the leading ratio coefficient, the maximum value of the sending window, and the minimum message sequence number; and the lagging threshold is determined based on the lagging ratio coefficient, the maximum value of the sending window, and the minimum message sequence number. in, The maximum value of the sending window is the maximum value of the sending window set for the communication group; the minimum message sequence number is the minimum value among the current message sequence numbers corresponding to the computing node within the communication group; the leading ratio coefficient is greater than the lagging ratio coefficient.

6. The method according to claim 1, wherein the corresponding message transmission congestion control operation is performed on the leading node, comprising: According to the transmission cycle, the first message is sent to the leading node; The first message is used to instruct the leading node to reduce its transmission window; The sending period is longer than the polling period.

7. The method according to claim 6, wherein the first message includes a first control field; the first control field records a value instructing the leading node to decrease the sending window.

8. The method according to claim 1, wherein for the lagging node, a corresponding message transmission congestion control operation is performed, comprising: A second message is sent to the lagging node, the second message being used to instruct the lagging node to reduce the time interval of the sliding transmission window.

9. The method according to claim 8, wherein the second message is a downlink result message obtained by the network device through aggregation processing of the data packets; the downlink result message includes a valid acknowledgment identifier; the valid acknowledgment identifier is used to instruct the lagging node to reduce the number of accumulated messages; in, The cumulative number of packets is used to trigger the computing node to send a corresponding uplink reply packet to the network device every time it receives a downlink result packet of the cumulative number of packets. In response to the uplink reply packet, the network device sends a downlink reply packet to the computing node. The downlink reply packet is used to trigger the computing node to continue sending the data packets to the network device.

10. A processing apparatus for asynchronous message transmission in a network computing system, deployed on a network device or an electronic device connected to the network device, the apparatus comprising: The sequence number acquisition unit is used to obtain the current sequence number of the data packets transmitted from each computing node in the communication group to the network device within the same polling cycle; An asynchronous determination unit is used to determine whether there is asynchronous message transmission between computing nodes in the communication group based on the asynchronous ratio coefficient and the current message sequence number corresponding to each computing node; The asynchronous scaling factor characterizes the degree of adjustment of the upper limit of the progress gap between computing nodes in the communication group. The device determination unit is used to determine the leading node and the lagging node in the communication group based on the leading ratio coefficient, the lagging ratio coefficient and the current message sequence number corresponding to each computing node if there is asynchronous message transmission between computing nodes in the communication group. Wherein, the leading ratio coefficient is used to determine the leading threshold, and the lagging ratio coefficient is used to determine the lagging threshold; The congestion control unit is used to perform corresponding message transmission congestion control operations on the leading node and the lagging node, respectively.

Images