Protocol computing method and device for collective communication, computing card and storage medium

Abstract

The application discloses a protocol calculation method and device of collective communication, a computing card and a storage medium. In a first data transmission period, a communication unit receives first protocol sub-data sent by a first node; a protocol information processing unit determines a protocol type, a data address and first data sent by the first node according to the first protocol sub-data; a protocol calculation unit reads local data corresponding to the data address from a storage unit according to the data address; the protocol calculation unit performs protocol calculation on the first data and the local data according to the protocol type, and obtains second data; in a second data transmission period, the protocol information processing unit reads the second data in the storage unit, generates second protocol sub-data according to the second data, and the communication unit sends the second protocol sub-data to a second node; through multiple data transmission periods, until data corresponding to the data address complete protocol calculation of all nodes. The balance between protocol calculation and chip self-computing tasks in computing power and storage is realized, and the overall performance is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of chip set communication, and particularly relates to a protocol calculation method and device for set communication, a computing card and a storage medium. BACKGROUND

[0002] With the continuous development of large language models (LLM), the size of the model is getting larger and larger, and it is more and more popular to build large-scale clusters through distributed systems. At the algorithm level of large language models, multiple parallel splitting strategies are involved. The interaction involving multiple nodes in parallel training is called set communication. Reduction calculation is an operation in set communication.

[0003] At present, the reduction calculation task of the chip shares the computing unit and the storage unit with the chip itself computing task, thereby increasing the pressure of computing power and storage bandwidth. The competition of reduction calculation and chip itself computing task for computing power and storage bandwidth leads to the decline of overall performance. How to effectively balance the reduction calculation and the chip itself computing task in terms of computing power and storage has become a problem to be solved in reduction calculation in set communication. SUMMARY

[0004] The purpose of the present application is to provide a reduction calculation method and device for set communication, a computing card and a storage medium, which can improve the balance of computing power and storage and improve the overall performance.

[0005] According to a first aspect of the present application, a reduction calculation method for set communication is provided, which is applied to a target node in a preset topology structure, the target node being any one node in the preset topology structure, the target node comprising a computing unit, a communication unit, a storage unit, a reduction information processing unit of a near communication unit and a reduction calculation unit of a near storage unit, and the target node being connected to a first node in the preset topology structure through the communication unit.

[0006] The method comprises:

[0007] In the first data transmission period, the communication unit receives first reduction sub-data sent by the first node, the first node being a previous hop node of the target node determined according to the preset topology structure;

[0008] The reduction information processing unit determines a reduction type, a data address and first data sent by the first node according to the first reduction sub-data, the data address being an address corresponding to each reduction sub-data obtained by fragmenting the data to be reduced, and the reduction result of the reduction sub-data being stored;

[0009] The reduction information processing unit sends the reduction type, the data address and the first data sent by the first node to the reduction calculation unit;

[0010] The protocol computing unit reads local data corresponding to the data address from the storage unit according to the data address, and performs protocol calculation on the first data and the local data according to the protocol type, to obtain second data, which is a protocol calculation result of a current data transmission period, and writes the second data into the data address;

[0011] In a second data transmission period, the protocol information processing unit reads second data in the storage unit, generates second protocol sub-data according to the second data, and the communication unit sends the second protocol sub-data to a second node, which is a next hop node of a target node determined according to a preset topology structure; the second data transmission period is a next data transmission period of the first data transmission period; after a plurality of data transmission periods, until data corresponding to the data address complete protocol calculation of all nodes.

[0012] According to a second aspect of the present application, a protocol calculation device for collective communication is provided, which is applied to a target node in a preset topology structure, the target node being any one node in the preset topology structure, and the target node comprising a computing unit, a communication unit, a storage unit, a protocol information processing unit of a near communication unit and a protocol computing unit of a near storage unit, and the device comprising:

[0013] The receiving module is configured to, in a first data transmission period, receive first protocol sub-data sent by a first node by the communication unit, the first node being a previous hop node of the target node determined according to the preset topology structure.

[0014] The protocol information acquisition module is configured to determine, by the protocol information processing unit, a protocol type, a data address and first data sent by the first node according to the first protocol sub-data, the data address being an address corresponding to each protocol sub-data obtained by fragmenting to-be-protocol data, and the protocol result of the protocol sub-data being used for storing.

[0015] The on-board transmission module is configured to send, by the protocol information processing unit, the protocol type, the data address and the first data sent by the first node to the protocol computing unit.

[0016] The writing module is configured to read, by the protocol computing unit, local data corresponding to the data address from the storage unit according to the data address, and perform protocol calculation on the first data and the local data according to the protocol type, to obtain second data, which is a protocol calculation result of a current data transmission period, and write the second data into the data address.

[0017] The sending module is configured to, in a second data transmission period, read second data in the storage unit by the protocol information processing unit, generate second protocol sub-data according to the second data, and send the second protocol sub-data to a second node by the communication unit, the second node being a next hop node of the target node determined according to the preset topology structure; the second data transmission period being a next data transmission period of the first data transmission period; and the data corresponding to the data address completing protocol calculation of all nodes through multiple data transmission periods.

[0018] According to a third aspect of the present application, a computing card is provided, the computing card serving as a target node in a preset topology structure, the target node being any one of nodes in the preset topology structure, the computing card comprising a computing unit, a communication unit, a storage unit, a protocol information processing unit of a near communication unit, and a protocol calculation unit of a near storage unit.

[0019] The communication unit is configured to, in a first data transmission period, receive first protocol sub-data sent by a first node, the first node being a previous hop node of the target node determined according to the preset topology structure.

[0020] The protocol information processing unit is configured to determine a protocol type, a data address, and first data sent by the first node according to the first protocol sub-data, the data address being an address corresponding to each protocol sub-data obtained by fragmenting to-be-protocol data, and used for storing a protocol result of the protocol sub-data; and send the protocol type, the data address, and the first data sent by the first node to the protocol calculation unit.

[0021] The protocol calculation unit is configured to read local data corresponding to the data address from the storage unit according to the data address; perform protocol calculation on the first data and the local data according to the protocol type to obtain second data, the second data being a protocol calculation result of a current data transmission period, and write the second data to the data address.

[0022] The protocol information processing unit is further configured to, in a second data transmission period, read second data in the storage unit, and generate second protocol sub-data according to the second data.

[0023] The communication unit is further configured to send the second protocol sub-data to a second node, the second node being a next hop node of the target node determined according to the preset topology structure; the second data transmission period being a next data transmission period of the first data transmission period; and the data corresponding to the data address completing protocol calculation of all nodes through multiple data transmission periods.

[0024] According to a fourth aspect of the present application, there is provided a distributed system, characterized in that comprising: a plurality of the computing cards according to any one of the embodiments of the present application; the plurality of the computing cards are connected to an inter-network through respective communication units.

[0025] According to a fifth aspect of the present application, there is provided a computer readable storage medium storing computer instructions for causing a processor to implement the protocol computation method of the collective communication according to any one of the embodiments of the present application when executed.

[0026] The technical scheme provided by the application comprises the following steps: in a first data transmission period, a communication unit receives first protocol sub-data sent by a first node, the first node being a previous hop node of a target node determined according to a preset topology structure; a protocol information processing unit determines a protocol type, a data address and first data sent by the first node according to the first protocol sub-data, the data address being an address corresponding to each protocol sub-data obtained by fragmenting to-be-protocol data, and used for storing protocol results of the protocol sub-data; the protocol information processing unit sends the protocol type, the data address and the first data sent by the first node to a protocol calculation unit; the protocol calculation unit reads local data corresponding to the data address from the storage unit according to the data address; performs protocol calculation on the first data and the local data according to the protocol type, to obtain second data, the second data being a protocol calculation result of a current data transmission period, and the second data being written into the data address; in a second data transmission period, the protocol information processing unit reads the second data in the storage unit, generates second protocol sub-data according to the second data, and the communication unit sends the second protocol sub-data to a second node, the second node being a next hop node of the target node determined according to the preset topology structure; the second data transmission period is a next data transmission period of the first data transmission period; after a plurality of data transmission periods, protocol calculation of data corresponding to the data address is completed on all nodes. At present, protocol calculation needs to be participated by a calculation unit, so that the calculation unit needs to complete protocol calculation by interacting with a storage unit in addition to performing its own calculation task. In comparison with the above protocol calculation scheme, the application does not need the calculation unit to participate in protocol calculation, but in the data transmission process of protocol calculation, a protocol information processing unit of a nearby communication unit analyzes the protocol type, the data address and the first data sent by the first node required by protocol calculation, and then a protocol calculation unit reads local data corresponding to the data address from the storage unit according to the data address, performs protocol calculation on the first data and the local data according to the protocol type, to obtain second data, and completes protocol calculation. In the target node, the calculation unit does not need to participate in protocol calculation, so that the calculation unit can be liberated. In the data transmission process, data is subjected to protocol calculation, so that protocol calculation can be quickly completed. In the protocol calculation process, the protocol calculation unit accesses the storage unit once to obtain local data, and then writes the obtained second data into the data address. Protocol calculation can be completed through one reading and writing. In comparison with the prior art in which received data is first written into the storage unit, and then the calculation unit reads the data to perform protocol calculation, the reading and writing times of the storage unit can be reduced, and then the storage bandwidth can be liberated to serve the self calculation task of the calculation unit.Since the application can reduce the storage bandwidth occupied in the reduction calculation process and does not need the calculation unit to participate in the reduction calculation, the calculation unit can retain more computing power to perform its own task, and the storage bandwidth can be used as little as possible for reduction calculation, realizing the balance of reduction calculation and chip self-computing task in computing power and storage, and improving the overall performance.

[0027] It should be understood that the content described in this part is not intended to identify the key or important features of the embodiments of the application, nor is it used to limit the scope of the application. Other features of the application will become apparent from the following description. BRIEF DESCRIPTION OF DRAWINGS

[0028] In order to more clearly illustrate the technical solutions in the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments description. Obviously, the drawings in the following description are only some embodiments of the application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0029] Figure 1 is a flowchart of a reduction calculation method of a collective communication provided by an embodiment of the application;

[0030] Figure 2 is a schematic diagram of a distributed system architecture provided by an embodiment of the application;

[0031] Figure 3 is a reduction calculation data flow diagram provided by an embodiment of the application;

[0032] Figure 4 is a reduction calculation schematic diagram of a ring topology provided by an embodiment of the application;

[0033] Figure 5 is a reduction calculation schematic diagram of a flat butterfly topology provided by an embodiment of the application;

[0034] Figure 6 is a reduction calculation schematic diagram of a parameter server topology provided by an embodiment of the application;

[0035] Figure 7 is a reduction calculation schematic diagram of another parameter server topology provided by an embodiment of the application;

[0036] Figure 8 is a structural schematic diagram of a reduction calculation device of a collective communication provided by an embodiment of the application;

[0037] Figure 9 is a structural schematic diagram of a computing card provided by an embodiment of the application. DETAILED DESCRIPTION

[0038] In the following, the technical solutions in the embodiments of the present application will be described clearly and completely in conjunction with the drawings in the embodiments of the present application, so that those skilled in the art can better understand the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should be within the scope of protection of the present application.

[0039] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0040] With the continuous development of large language models (LLM), the size of the model is getting larger and larger, and building large-scale clusters through distributed systems is becoming more and more popular. There are many parallel splitting strategies involved in the algorithm level of large language models. The interaction involving multiple nodes in parallel training is called collective communication. Reduction computation is an operation in collective communication.

[0041] In artificial intelligence chips, the computing unit and the storage unit are two key indicators. The computing unit is used to meet the demand for computing power; the storage unit is used to solve the storage of data, and the storage also needs to provide large bandwidth capability to match the computing power of the computing unit. In the distributed system scenario, interconnection has become a new challenge. Taking the allreduce operation as an example, on the one hand, it needs to complete the communication between multiple nodes for data interaction, and on the other hand, it needs to complete the reduction operation (reduce) of data after communication. Efficient processing of communication and computation requires careful stream arrangement, which has high complexity. At the same time, the reduction operation consumes valuable computing resources in the chip, so the reduction computation and the computing unit form a competition. In addition to the interconnection between chips, the reduction operation inside the chip also needs to frequently access the storage unit, further increasing the requirement for storage bandwidth.

[0042] It can be seen that the current chip protocol computing task shares the computing unit and the storage unit with the chip itself computing task, thereby increasing the pressure of computing power and storage bandwidth. The competition of protocol computing and chip itself computing task for computing power and storage bandwidth causes the decline of overall performance. How to effectively balance the protocol computing and chip itself computing task in terms of computing power and storage has become a problem to be solved in the reduction of collection communication.

[0043] Figure 1 is a flowchart of a reduction computing method of collection communication provided by an embodiment of the present application. The embodiment can be applied to the case of reduction computing when multiple compute cards in a distributed system perform collection communication. The method can be executed by a reduction computing device of collection communication, which can be realized in the form of hardware and / or software. If executed in the form of hardware, it can be applied to a compute card (General Compute Unit, GCU). The method is applied to a target node in a preset topology. The target node is any one node in the preset topology. The target node includes a computing unit, a communication unit, a storage unit, a reduction information processing unit of a near communication unit, and a reduction computing unit of a near storage unit. The method includes the following steps.

[0044] S101, in the first data transmission period, the communication unit receives the first reduction sub-data sent by the first node.

[0045] The first node is the last hop node of the target node determined according to the preset topology.

[0046] The first data transmission period is one data transmission period in the reduction computing. In each data transmission period, the compute card performs data transmission in a predetermined direction according to the preset topology. In the present application, the compute card in the distributed system is referred to as a node. The communication unit of the target node is connected to other nodes through an interconnection network. In the first data transmission period, the target node receives the first reduction sub-data sent by the first node. The preset topology includes but is not limited to a ring topology, a flat butterfly topology, or a parameter server topology.

[0047] S102, the reduction information processing unit determines the reduction type, the data address, and the first data sent by the first node according to the first reduction sub-data.

[0048] The data address is the address corresponding to each reduction sub-data obtained by sharding the data to be reduced, and is used to store the reduction result of the reduction sub-data.

[0049] Figure 2A schematic diagram of a distributed system architecture provided by the present application, which includes a plurality of computing cards and an interconnection network, each computing card including a control unit, a computing unit, a communication unit, a storage unit, a protocol information processing unit of a near storage unit, and a protocol computing unit of a near storage unit. The plurality of computing cards are connected to the interconnection network through the respective communication units.

[0050] The control unit is configured to complete the control of data flow in the computing card and the scheduling of other parts. The computing unit is configured to complete the operation of matrices, vectors, and the like. The storage unit is configured to complete the storage of data during the computing process, including the storage of input and output data and intermediate process data. The communication unit is configured to complete the movement of data between the computing cards. The communication unit can have one or more groups. The communication unit is connected to other computing cards through the interconnection network. The communication unit can be RoCE (RDMA over Converged Ethernet), Infiniband (IB), or PCIe, etc. The interconnection network is configured to interconnect the computing cards after the communication unit is connected to the interconnection network. The interconnection network can be a direct cable connection or an indirect connection based on a switch. The protocol information processing unit is configured to complete the extraction and insertion of protocol information across the computing cards, and cooperate with the protocol computing unit to complete the operation of the protocol. The protocol computing unit is independent of the computing unit and is configured to complete the calculation of the protocol during the data transmission process.

[0051] In the distributed system, a plurality of computing cards distributedly arranged are used to perform protocol calculation on data in parallel. Specifically, in each data transmission cycle, the first node sends first protocol sub-data containing a data shard processed by the target node to the target node. During the transmission of the first protocol sub-data, the target node determines, by the protocol information processing unit, a protocol type, a data address, and first data sent by the first node according to the first protocol sub-data, the first data being the data shard processed by the target node. Then, the protocol computing unit of the near storage unit performs protocol calculation, so as to avoid the participation of the computing unit in the protocol calculation of the data shard, and further avoid the occupation of the computing unit by the protocol calculation. The protocol type can be addition, subtraction, minimum value, maximum value, etc.

[0052] Optionally, the protocol information processing unit determines the protocol type, the data address, and the first data sent by the first node according to the first protocol sub-data, which can be implemented in the following manner:

[0053] If the protocol information processing unit determines to perform protocol calculation according to the protocol sub-data, the protocol information processing unit determines the protocol type, the data address, and the first data sent by the first node according to the first protocol sub-data.

[0054] If the protocol information processing unit determines not to perform the protocol calculation according to the protocol sub-data, the protocol information processing unit transmits the first data to the storage unit in a transparent manner.

[0055] After the first protocol sub-data is parsed, the protocol type can be obtained. According to the protocol type, it can be determined whether the protocol calculation is needed and the specific type of the protocol calculation. If the data in the field indicating the protocol type does not belong to any protocol type, it is determined that the protocol calculation is not performed.

[0056] If the protocol calculation is performed, the protocol calculation is performed according to the protocol calculation manner provided by the embodiment of the present application. If the protocol calculation is not performed, the first protocol sub-data received is transmitted in a transparent manner, and the first data contained therein is written into the data address.

[0057] The above embodiment can distinguish between the protocol calculation and the non-protocol calculation. When the protocol calculation is not performed, the data is written into the storage unit in a transparent manner, so that the computing unit performs other calculation tasks according to the written data, ensures the normal operation of the non-protocol calculation, and improves the reliability.

[0058] Further, the protocol type, the data address, and the first data sent by the first node can be determined according to the first protocol sub-data.

[0059] The protocol type and the data address are determined according to the message header of the protocol sub-data, and the first data sent by the first node is determined according to the message body of the protocol sub-data.

[0060] Different fields in the message header are predefined to represent different contents. The protocol type can be determined according to the data in the field indicating the protocol type. The first protocol sub-data includes the message header and the message body. In the protocol calculation, the data slice for performing the protocol calculation, i.e., the first data, is located in the message body, and the data address related to the storage of the protocol calculation type is located in the message header.

[0061] The protocol information processing unit is arranged at the near communication unit. After the first protocol sub-data is received by the communication unit, the protocol information processing unit can parse the first protocol sub-data to obtain the message header and the message body. The protocol type and the data address can be determined according to the message header, and the first data sent by the first node can be obtained according to the message body.

[0062] The above embodiment can quickly determine the protocol type and the data address according to the preset field in the message header, and determine the first data according to the message body. The protocol type, the data address, and the first data are determined during the message transmission process, and the protocol calculation efficiency is improved.

[0063] S103, the protocol information processing unit sends the protocol type, data address and the first data sent by the first node to the protocol calculation unit.

[0064] Optionally, if the data amount of the first data transmitted in the first data transmission period reaches the preset data amount, it is indicated that the protocol calculation can be performed, and the protocol type, data address and the first data sent by the first node are sent to the protocol calculation unit, so that the protocol calculation unit performs protocol calculation according to the above data.

[0065] Optionally, the protocol information processing unit sends the protocol type, data address and the first data sent by the first node to the protocol calculation unit, and can also be implemented in the following manner:

[0066] The protocol information processing unit sends the protocol type, data address and the first data sent by the first node to the flow data cache. When the data amount stored in the flow data cache reaches the preset data amount, the protocol calculation unit reads the cached data from the flow data cache.

[0067] In collective communication, data transmission is performed according to clock cycles. Since the data amount transmitted in one clock cycle is limited, the data amount transmitted in one clock cycle may be less than the data amount of one data segment. In order to complete the calculation of one data segment at one time or to improve the calculation amount of the protocol calculation performed by each calculation card at one time, the data received by the communication unit can be cached.

[0068] The flow data cache can be located in the protocol calculation unit, and the protocol information processing unit sends the obtained first data, data address and protocol type to the flow data cache. The protocol calculation unit includes the flow data cache and the protocol operation unit, and when the data amount cached reaches the preset data amount, the protocol operation unit reads the cached data from the flow data cache. The cached data can be the first data, or the first data, data address and protocol type.

[0069] When the data size in the flow data cache reaches the preset data amount, such as 16 KB, the protocol calculation is triggered. At this time, the protocol operation unit reads the cached data from the flow data cache, and performs a read operation on the data in the storage unit according to the data address. When the storage unit returns the read operation result, that is, the local data in the data address, the protocol operation unit performs protocol operation according to the returned local data and the first data, and obtains the second data.

[0070] The protocol operation unit performs protocol operation once every time the data size in the flow data cache accumulates to the preset data amount. In the protocol operation process, the segments transmitted between the calculation cards form a pipeline, and continuously work until the data processing of other calculation cards is completed.

[0071] The above embodiment can cache the data received by the communication module into the pipeline data cache, when the amount of cached data reaches the preset data amount, it indicates that the required data amount of one reduction operation has been received, at this time the reduction operation unit reads the first data in the pipeline data cache for reduction calculation, which can improve the calculation amount of single reduction calculation, thereby reducing the reduction calculation times and reducing the bandwidth occupation of the storage unit, and improving the computing unit computing power.

[0072] S104, the reduction calculation unit reads the local data corresponding to the data address from the storage unit according to the data address; and performs reduction calculation on the first data and the local data according to the reduction type to obtain second data.

[0073] The second data is the reduction calculation result of the current data transmission period, and the second data is written into the data address.

[0074] Figure 3 The reduction calculation data flow diagram provided by the embodiment of the present application is shown in FIG. 1. Figure 3 As shown in FIG. 1, the reduction information processing unit includes a pipeline data cache and a reduction operation unit. The reduction information processing unit sends the reduction type, the data address and the first data obtained by unpacking to the pipeline data cache.

[0075] If it is non-reduction calculation, it is transmitted, and the data is written into the local data storage space in the storage unit according to the data address provided by the communication unit. If it is reduction calculation, it is not transmitted, the reduction operation unit reads the first data and the reduction type in the pipeline data cache, reads the local data participating in the reduction calculation from the local data of the storage unit according to the data address. According to the reduction type, the first data and the local data are reduced to obtain second data, and the second data is written into the local data storage space of the storage unit.

[0076] The calculation result of each shard is sequentially written into the local data corresponding to the shard position in the storage unit. Since the shards are processed in sequence, and the local data is invalid after the reduction is completed, the calculation result can be directly written into the original position, i.e., the data address.

[0077] S105, in the second data transmission period, the reduction information processing unit reads the second data in the storage unit, generates second reduction sub-data according to the second data, and the communication unit sends the second reduction sub-data to a second node, which is the next hop node of the target node determined according to the preset topology structure; the second data transmission period is the next data transmission period of the first data transmission period; after a plurality of data transmission periods, until the data corresponding to the data address completes the reduction calculation of all nodes.

[0078] In the first data transmission cycle, the reduction information processing unit encapsulates the reduction calculation result obtained in the previous data transmission cycle of the first data transmission cycle and sends to the second node.

[0079] In the second data transmission cycle, the reduction information processing unit reads the second data written by the reduction calculation unit to the storage unit, generates the second reduction sub-data according to the second data and the current reduction progress, and sends the second reduction sub-data to the second node.

[0080] Optionally, the second reduction sub-data generated according to the second data can be implemented in the following manner: generating a message body of the second reduction sub-data according to the second data, generating a message header of the second reduction sub-data according to the reduction operation type and the data address, and generating the second reduction sub-data according to the message body and the message header.

[0081] The above manner can quickly complete the encapsulation of the second data by the reduction information processing unit in the second data transmission cycle to obtain the second reduction sub-data, without the calculation unit participating in the reduction calculation and the reduction data transmission, thereby realizing the quick generation and sending of the second reduction sub-data and improving the calculation unit power.

[0082] Different reduction schemes can be set for different topology results, and in each reduction scheme, the first node and the second node corresponding to the target node can be fixed or can change with different reduction time nodes according to the different topology structures. The specific conditions are described in the following embodiments. In the embodiments of the present application, no matter what kind of topology structure and what kind of reduction scheme is adopted, the reduction calculation is completed in the reduction data transmission process, and the calculation unit does not need to participate in the reduction calculation, thereby improving the calculation unit power.

[0083] The technical scheme provided by the application, in the first data transmission period, the communication unit receives the first protocol sub-data sent by the first node, and the first node is the last hop node of the target node determined according to the preset topology structure; the protocol information processing unit determines the protocol type, the data address and the first data sent by the first node according to the first protocol sub-data, the data address is the address corresponding to each protocol sub-data obtained by fragmenting the data to be regulated, and is used for storing the protocol result of the protocol sub-data; the protocol information processing unit sends the protocol type, the data address and the first data sent by the first node to the protocol calculation unit; the protocol calculation unit reads the local data corresponding to the data address from the storage unit according to the data address; the first data and the local data are calculated according to the protocol type, and the second data is obtained, the second data is the protocol calculation result of the current data transmission period, and the second data is written into the data address; in the second data transmission period, the protocol information processing unit reads the second data in the storage unit, generates the second protocol sub-data according to the second data, and the communication unit sends the second protocol sub-data to the second node, and the second node is the next hop node of the target node determined according to the preset topology structure; the second data transmission period is the next data transmission period of the first data transmission period; after a plurality of data transmission periods, until the data corresponding to the data address completes the protocol calculation of all nodes. At present, the protocol calculation needs the participation of the calculation unit, so that the calculation unit not only performs its own calculation task, but also completes the protocol calculation by interacting with the storage unit. Compared with the above protocol calculation scheme, the application does not need the participation of the calculation unit in the protocol calculation, but in the data transmission process of the protocol calculation, the protocol information processing unit of the near communication unit analyzes the protocol type, the data address and the first data sent by the first node required for the protocol calculation, and then the protocol calculation unit reads the local data corresponding to the data address from the storage unit according to the data address, and calculates the first data and the local data according to the protocol type, to obtain the second data and complete the protocol calculation. In the target node, the calculation unit does not need to participate in the protocol calculation, so the calculation unit can be freed. In the data transmission process, the data is regulated and calculated, which can realize fast completion of the protocol calculation. In the process of protocol calculation, the local data is accessed once by the protocol calculation unit, and then the second data is written into the data address. The protocol calculation can be completed by reading and writing once. Compared with the prior art, the received data is first written into the storage unit, and then the calculation unit reads the data for protocol calculation, which can reduce the read-write times of the storage unit, and then release the storage bandwidth for the service of the calculation task of the calculation unit.Since the application can reduce the storage bandwidth occupied in the reduction calculation process and does not require the calculation unit to participate in the reduction calculation, the calculation unit can retain more computing power to perform its own tasks, and the storage bandwidth can be used as little as possible for reduction calculation, achieving the balance of reduction calculation and chip self-computing tasks in computing power and storage, and improving the overall performance.

[0084] In the embodiment of the application, the reduction calculation is accompanied by the writing of data from the communication unit, and the processing of the reduction calculation unit only increases the RTT (round trip time) time of the storage unit, that is, the reduce operation can be completed with the transmission of data. The whole processing delay is low, and the bandwidth consumption of storage is significantly reduced. From the perspective of pipeline processing, the number of pipeline stages of the collection communication only needs two stages through the scheme of the application, which has significant benefits in terms of control complexity and time consumption of the pipeline. The above two stages are: first stage: in the first data transmission period, the first data sent by the first node is received, and the data is written to the corresponding local storage unit through the communication unit, and the reduction calculation unit completes the reduction calculation during the writing process. Second stage: in the second data transmission period, the second data (i.e. the reduction calculation result) is read from the local data address and sent to the second node by the communication unit.

[0085] On the basis of the above embodiment, optionally, the preset topology can be a ring topology. The ring topology includes a plurality of nodes, which are node A, node B, node C, and node N in turn; the previous hop node of node A is node N, and the next hop node is node B, and so on, forming a ring topology; if the target node is node A, the first node is node N, and the second node is node B.

[0086] Figure 4 The reduction calculation schematic diagram of the ring topology provided by the embodiment of the application is shown in Figure 4 As shown in the figure, it is assumed that the distributed system includes four nodes, and the above node N is node D. Node A, node B, node C and node D form a closed loop.

[0087] Taking the reduction operation in the ring topology of the above four nodes as an example, according to the number of nodes, the data of each node is divided into 4 shards. Each data transmission period can be called a step, and in one data transmission period, the node transmits one data and receives one data. Assuming that node A is the target node, the first node is node D, and the second node is node B.

[0088] For the received data, the target node performs a reduction calculation (reduce) operation on the received data and the local data. After 3 hops, each node obtains 1 / 4 of the entire reduce data. Then, after one week of transmission, each node obtains the complete data.

[0089] For each node (also known as a computing card), each hop process includes several parts of operations: sending data; receiving data, and performing reduction operations.

[0090] In the above example, the data is divided into four shards. Taking the fourth data shard as an example: in step 1, node A receives the D3 data shard sent by node D, and performs a reduction operation to obtain the data shard content of A3+D3. In step 2, node B receives the A3+D3 data shard content sent by node A, and performs a reduction operation to obtain the data shard content of A3+D3+B3. In step 3, node C receives the A3+D3+B3 data shard content sent by node B, and performs a reduction operation to obtain the data shard content of A3+D3+B3+C3. At this time, the fourth data shard completes the reduction operation on the four nodes, and obtains the operation result of A3+D3+B3+C3.

[0091] The above embodiment can efficiently complete the reduction operation among multiple nodes in a ring topology scenario, while improving the computing power of the computing units in each computing card.

[0092] Figure 5 A reduction calculation diagram of a flat butterfly topology provided for an embodiment of the present application is shown in FIG. 1. Figure 5 As shown in FIG. 1, the preset topology structure is a flat butterfly topology. The flat butterfly topology includes a plurality of groups, and each group includes two nodes.

[0093] In the first data transmission period, the two nodes in each group perform reduction sub-data calculation to obtain a first reduction result; in the second data transmission period, the nodes in each group respectively send the first reduction result to a node in an adjacent group, and in the node in the adjacent group, a second reduction result of two groups of nodes is obtained.

[0094] Suppose the flat butterfly topology is composed of four nodes, node A and node B form a group, and node C and node D form a group. In the first data transmission period, the two nodes in each group are first nodes to each other. For example, node A receives data B sent by node B as the first node, and performs a reduction calculation on data B and local data A of node A to obtain data A+B. Similarly, node B receives data A sent by node A as the first node, and performs a reduction calculation on data A and local data B of node B to obtain data A+B.

[0095] In the second data transmission period, the two nodes in the two groups are second nodes to each other in the two groups. Specifically, node C is a second node to node A, and node D is a second node to node B. At the same time, node A is a second node to node C, and node B is a second node to node D.

[0096] For example, in the second data transmission cycle, node A sends the second data obtained in the first data transmission cycle, i.e., data A+B, to node C. Node C obtains data A+B+C+D after reduction. In this way, node A, node B and node D all obtain the reduced data A+B+C+D.

[0097] In the above embodiment, after grouping, the reduce operation is performed in sequence, and finally the complete reduced result is obtained on each node. For the operation of each node, it still includes: sending data; receiving data, and reduce operation.

[0098] The above embodiment can realize efficient reduce operation among multiple nodes in a flat butterfly topology scenario, and improve the computing power of the computing unit in each computing card.

[0099] Figure 6 The reduce calculation schematic diagram of the parameter server topology provided by the embodiment of the application is shown in Figure 6 The preset topology structure is: a parameter server topology, the parameter server topology includes a server node and a plurality of regular nodes; the server node receives reduce sub-data sent by the regular nodes, obtains a reduce result according to the reduce sub-data; and the server node sends the reduce result to each regular node.

[0100] For example, the parameter server topology includes one server and four nodes. In the first data transmission cycle, each node sends the data held by itself to the parameter server, and the parameter server performs reduce calculation according to the four pieces of shard data received to obtain a settlement result. In the second data transmission cycle, the parameter server sends the reduce calculation result obtained to the four nodes respectively, so that each node holds the reduce settlement result. The parameter server is also called a server.

[0101] In the above topology structure, the four nodes send the data shards prepared for reduce calculation to one parameter server, and the parameter server completes the reduce calculation (allreduce) accumulation and sends the reduce result to each node in the form of broadcast. At this time, each computing card is not responsible for calculation, and the calculation is completed by the parameter server. Each node only completes data sending and receiving.

[0102] The above embodiment can realize efficient reduce operation among multiple nodes in a parameter server topology scenario with the help of a parameter server node, and can improve the computing power of the computing unit in each computing card.

[0103] Further, Figure 7 Another reduce calculation schematic diagram of the parameter server topology provided by the embodiment of the application is shown inFigure 7 Optionally, one node is selected from the plurality of nodes as the server node in the above embodiment, and the rest of the nodes are the regular nodes. In the plurality of data transmission periods, the regular nodes only perform data transmission, and the server node performs the reduction calculation.

[0104] Optionally, one node is selected from the plurality of nodes as the server node in the above embodiment, and the rest of the nodes are the regular nodes. In the plurality of data transmission periods, the regular nodes only perform data transmission, and the server node performs the reduction calculation.

[0105] Optionally, one node is selected from the plurality of nodes as the server node in the above embodiment, and the rest of the nodes are the regular nodes. In the plurality of data transmission periods, the regular nodes only perform data transmission, and the server node performs the reduction calculation.

[0106] The above embodiment can make each node assume the role of the server node. For a topology structure with N nodes, each node completes 1 / N of the data. At this time, for each node, N-1 groups of data are received, the reduction operation of N-1 groups of data and local data is performed, and the data is transmitted.

[0107] The above embodiment can complete the reduction calculation of all data shards by taking the node as the server node and through the data interaction between the server node and the regular node. The reduction operation is efficiently completed among the plurality of nodes, and the computing power of the computing unit in each computing card can be improved.

[0108] Figure 8 The structure diagram of the reduction calculation device for collective communication provided by the embodiment of the application is used to implement the reduction calculation method for collective communication. The device is applied to a target node in a preset topology structure, the target node is any one node in the preset topology structure, and the target node includes a computing unit, a communication unit, a storage unit, a reduction information processing unit of a near communication unit, and a reduction calculation unit of a near storage unit. The device includes:

[0109] The receiving module 21 is configured to receive, in a first data transmission period, first reduction sub-data transmitted by a first node, the first node being a previous hop node of the target node determined according to the preset topology structure;

[0110] The reduction information acquisition module 22 is configured to determine, by the reduction information processing unit, a reduction type, a data address, and first data transmitted by the first node according to the first reduction sub-data, the data address being an address corresponding to each reduction sub-data obtained by dividing the data to be reduced, and the reduction result of the reduction sub-data.

[0111] a board transmission module 23, configured to transmit, by the protocol information processing unit, the protocol type, the data address and the first data transmitted by the first node to the protocol calculation unit;

[0112] a writing module 24, configured to read, by the protocol calculation unit, local data corresponding to the data address from the storage unit according to the data address; perform protocol calculation on the first data and the local data according to the protocol type to obtain second data, the second data being a protocol calculation result of a current data transmission period, and write the second data into the data address;

[0113] a sending module 25, configured to read, by the protocol information processing unit, the second data in the storage unit in a second data transmission period, generate second protocol sub-data according to the second data, and transmit, by the communication unit, the second protocol sub-data to a next hop node of a target node determined according to a preset topology structure; the second data transmission period is a next data transmission period of the first data transmission period; after a plurality of data transmission periods, protocol calculation of data corresponding to the data address on all nodes is completed.

[0114] On the basis of the above-mentioned embodiments, optionally, the protocol information acquisition module 22 is configured to:

[0115] if the protocol information processing unit determines to perform protocol calculation according to the protocol sub-data, the protocol information processing unit determines the protocol type, the data address and the first data transmitted by the first node according to the first protocol sub-data;

[0116] if the protocol information processing unit determines not to perform protocol calculation according to the protocol sub-data, the protocol information processing unit transmits the first data to the storage unit in a transparent manner.

[0117] On the basis of the above-mentioned embodiments, optionally, the protocol information acquisition module 22 is configured to:

[0118] determine the protocol type and the data address according to a packet header of the protocol sub-data, and determine the first data transmitted by the first node according to a packet body of the protocol sub-data.

[0119] On the basis of the above-mentioned embodiments, optionally, the sending module 25 is configured to:

[0120] generate a packet body of second protocol sub-data according to the second data, generate a packet header of the second protocol sub-data according to the protocol operation type and the data address, and generate the second protocol sub-data according to the packet body and the packet header.

[0121] On the basis of the above-mentioned embodiment, optionally, the in-board transmission module 23 is used for:

[0122] The protocol information processing unit sends the protocol type, data address and first data sent by the first node to the flow data cache;

[0123] When the amount of data stored in the flow data cache reaches the preset data amount, the protocol calculation unit reads the cached data from the flow data cache.

[0124] On the basis of the above-mentioned embodiment, optionally, the preset topology is a ring topology; the ring topology comprises a plurality of nodes, sequentially being node A, node B, node C and node N; the previous hop node of node A is node N, and the next hop node of node A is node B, and the same is true for the other nodes, thereby forming a ring topology.

[0125] If the target node is node A, the first node is node N, and the second node is node B.

[0126] On the basis of the above-mentioned embodiment, optionally, the preset topology is a flat butterfly topology; the flat butterfly topology comprises a plurality of groups, each group comprising two nodes,

[0127] In the first data transmission period, the two nodes in each group perform calculation on the protocol sub-data, and obtain a first protocol result.

[0128] In the second data transmission period, the nodes in each group respectively send the first protocol result to a node in an adjacent group, and in the node in the adjacent group, a second protocol result of two nodes is obtained.

[0129] On the basis of the above-mentioned embodiment, optionally, the preset topology is a parameter server topology, and the parameter server topology comprises a server node and a plurality of regular nodes.

[0130] The server node receives protocol sub-data sent by the regular nodes, obtains a protocol result according to the protocol sub-data, and sends the protocol result to each regular node.

[0131] On the basis of the above-mentioned embodiment, optionally, one node is selected from the plurality of nodes as a server node, and the remaining nodes are regular nodes.

[0132] The protocol calculation device for collective communication provided by the embodiment of the application comprises a receiving module 21, a protocol information acquisition module 22, an on-board transmission module 23, a writing module 24 and a sending module 25. In a first data transmission period, the communication unit receives first protocol sub-data sent by a first node, and the first node is a previous hop node of a target node determined according to a preset topology structure. The protocol information processing unit determines a protocol type, a data address and first data sent by the first node according to the first protocol sub-data, the data address is an address corresponding to each protocol sub-data obtained by fragmenting to-be-protocol data, and is used for storing a protocol result of the protocol sub-data. The protocol information processing unit sends the protocol type, the data address and the first data sent by the first node to the protocol calculation unit. The protocol calculation unit reads local data corresponding to the data address from the storage unit according to the data address. The protocol calculation unit performs protocol calculation on the first data and the local data according to the protocol type, obtains second data, writes the second data into the data address, and performs protocol calculation on the first data and the local data according to the protocol type. The second data is a protocol calculation result of a current data transmission period. In a second data transmission period, the protocol information processing unit reads the second data in the storage unit, generates second protocol sub-data according to the second data, and the communication unit sends the second protocol sub-data to a second node, and the second node is a next hop node of the target node determined according to the preset topology structure. The second data transmission period is a next data transmission period of the first data transmission period. After a plurality of data transmission periods, the data corresponding to the data address completes protocol calculation of all nodes. At present, protocol calculation needs to be participated in by a calculation unit, so that the calculation unit needs to complete protocol calculation by interacting with a storage unit in addition to performing a calculation task of the calculation unit. In the protocol calculation scheme, the calculation unit does not participate in protocol calculation, but the protocol information processing unit of a near communication unit analyzes the protocol type, the data address and the first data sent by the first node required for protocol calculation in a data transmission process of protocol calculation, then the protocol calculation unit reads local data corresponding to the data address from the storage unit according to the data address, performs protocol calculation on the first data and the local data according to the protocol type, obtains second data, and completes protocol calculation. In the target node, the calculation unit does not participate in protocol calculation, so that the calculation unit can be liberated. In the data transmission process, data is subjected to protocol calculation, so that protocol calculation can be quickly completed. In the protocol calculation process, the protocol calculation unit accesses the storage unit once to obtain local data, and then writes the obtained second data into the data address. Protocol calculation can be completed by reading and writing once. Compared with the prior art that first writes received data into the storage unit and then reads the data by the calculation unit to perform protocol calculation, the number of reading and writing times of the storage unit can be reduced, and then the storage bandwidth can be liberated to serve the calculation task of the calculation unit.The application can reduce the storage bandwidth occupied in the reduction calculation process, and the calculation unit does not need to participate in the reduction calculation, so that the calculation unit can retain more computing power to perform its own tasks, and the storage bandwidth can be used as little as possible for reduction calculation, achieving the balance of reduction calculation and chip self-computing tasks in computing power and storage, and improving the overall performance.

[0133] The reduction calculation device for collective communication provided by the embodiment of the application can execute the reduction calculation method for collective communication provided by any embodiment of the application, and has the corresponding function modules and beneficial effects of the execution method.

[0134] Figure 9 is a structural schematic diagram of a computing card provided by the embodiment of the application. The computing card serves as a target node in a preset topology structure, the target node is any one node in the preset topology structure, and the computing card comprises a computing unit 31, a communication unit 32, a storage unit 33, a reduction information processing unit 34 of a near communication unit, and a reduction calculation unit 35 of a near storage unit 31.

[0135] The communication unit 32 is configured to receive first reduction sub-data sent by a first node in a first data transmission period, the first node being a last hop node of the target node determined according to the preset topology structure.

[0136] The reduction information processing unit 34 is configured to determine a reduction type, a data address, and first data sent by the first node according to the first reduction sub-data, the data address being an address corresponding to each reduction sub-data obtained by fragmenting the data to be reduced, and used for storing a reduction result of the reduction sub-data; and send the reduction type, the data address, and the first data sent by the first node to the reduction calculation unit 35.

[0137] The reduction calculation unit 35 is configured to read local data corresponding to the data address from the storage unit 33 according to the data address; perform reduction calculation on the first data and the local data according to the reduction type to obtain second data, the second data being a reduction calculation result of a current data transmission period; and write the second data to the data address.

[0138] The reduction information processing unit 34 is further configured to read second data in the storage unit 33 in a second data transmission period, and generate second reduction sub-data according to the second data.

[0139] The communication unit 32 is further configured to send the second reduction sub-data to a second node, the second node being a next hop node of a target node determined according to a preset topology; the second data transmission period being a next data transmission period of the first data transmission period; and the data address corresponding data completing reduction calculation of all nodes through a plurality of data transmission periods.

[0140] The computing card can be applied to various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The computing card can also be applied to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart phones, wearable devices (e.g., headsets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not intended to limit the implementations of the applications described and / or claimed in this document.

[0141] The storage unit 33 can be a read-only memory (ROM), a random access memory (RAM), etc., in which a computer program executable by the at least one processor is stored. The computing unit 31 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) or loaded from the storage unit 33 into the random access memory (RAM). Various programs and data required for the operation of the computing card can also be stored in the RAM. The processor, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0142] The computing unit 31 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 31 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The computing unit 31 performs various methods and processes described above, such as the reduction calculation method of collective communication.

[0143] In some embodiments, the protocol computation method of collective communication can be implemented as a computer program tangibly embodied in a computer readable storage medium, e.g., storage unit 33. In some embodiments, portions or all of the computer program can be loaded and / or installed onto the computing card via the ROM and / or communication unit 32. When the computer program is loaded onto the RAM and executed by the computing unit 31, one or more steps of the protocol computation method of collective communication described above can be performed. Alternatively, in other embodiments, the computing unit 31 can be configured to perform the protocol computation method of collective communication by any other suitable means, e.g., by means of firmware.

[0144] Various implementations of the systems and techniques described above can be realized in digital electronic circuitry, integrated circuitry, a field programmable gate array (FPGA), an application specific integrated circuit (ASIC), a system on a chip (SOC), a programmable logic device (PLD), a computer hardware, firmware, software, and / or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0145] Computer programs used to implement the protocol computation method of collective communication of the present application can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the computer program running on the processor implements the functions / operations specified in the flowcharts and / or the block diagrams. The computer program can be executed entirely on a machine, partially on a machine, partially on a machine as a stand-alone software package, and partially on a remote machine or a server.

[0146] Embodiments of the present application also provide a computer readable storage medium, which stores computer instructions, and the computer instructions can be used to make a processor execute a protocol computation method of collective communication, and the method comprises:

[0147] In the first data transfer period, the communication unit receives a first protocol sub-data sent by a first node, and the first node is a previous hop node of a target node determined according to a preset topology structure;

[0148] The reduction information processing unit determines a reduction type, a data address, and first data sent by the first node according to the first reduction sub-data, the data address being an address corresponding to each reduction sub-data obtained by fragmenting the data to be reduced, and used for storing a reduction result of the reduction sub-data;

[0149] The reduction information processing unit sends the reduction type, the data address, and the first data sent by the first node to the reduction calculation unit;

[0150] The reduction calculation unit reads local data corresponding to the data address from the storage unit according to the data address, and performs reduction calculation on the first data and the local data according to the reduction type to obtain second data, which is a reduction calculation result of the current data transmission period, and writes the second data to the data address;

[0151] In the second data transmission period, the reduction information processing unit reads the second data in the storage unit, generates second reduction sub-data according to the second data, and the communication unit sends the second reduction sub-data to a second node, which is a next hop node of a target node determined according to a preset topology structure; the second data transmission period is a next data transmission period of the first data transmission period; after a plurality of data transmission periods, the data corresponding to the data address completes reduction calculation of all nodes.

[0152] On the basis of the above-mentioned embodiments, the reduction information processing unit determines a reduction type, a data address, and first data sent by the first node according to the first reduction sub-data, comprising:

[0153] If the reduction information processing unit determines to perform reduction calculation according to the reduction sub-data, the reduction information processing unit determines a reduction type, a data address, and first data sent by the first node according to the first reduction sub-data;

[0154] If the reduction information processing unit determines not to perform reduction calculation according to the reduction sub-data, the reduction information processing unit transmits the first data to the storage unit.

[0155] On the basis of the above-mentioned embodiments, determining a reduction type, a data address, and first data sent by the first node according to the first reduction sub-data, comprising:

[0156] Determining a reduction type and a data address according to a message header of the reduction sub-data, and determining first data sent by the first node according to a message body of the reduction sub-data.

[0157] On the basis of the above-mentioned embodiments, generating second reduction sub-data according to the second data, comprising:

[0158] generate a message body of the second reduction sub-data according to the second data, generate a message header of the second reduction sub-data according to a reduction operation type and a data address, and generate the second reduction sub-data according to the message body and the message header.

[0159] In the above embodiment, the reduction information processing unit sends the reduction type, the data address, and the first data sent by the first node to the reduction calculation unit, and the reduction information processing unit comprises:

[0160] The reduction information processing unit sends the reduction type, the data address, and the first data sent by the first node to the pipeline data cache.

[0161] When the amount of data stored in the pipeline data cache reaches a preset data amount, the reduction calculation unit reads the cached data from the pipeline data cache.

[0162] In the above embodiment, the preset topology is a ring topology, and the ring topology comprises a plurality of nodes, sequentially being a node A, a node B, a node C, and a node N; a previous hop node of the node A is the node N, and a next hop node of the node A is the node B, and the same applies to the other nodes, so as to form the ring topology.

[0163] If the target node is the node A, the first node is the node N, and the second node is the node B.

[0164] In the above embodiment, the preset topology is a flat butterfly topology, and the flat butterfly topology comprises a plurality of groups, each group comprising two nodes.

[0165] In a first data transmission period, the two nodes in each group perform calculation of the reduction sub-data, and obtain a first reduction result.

[0166] In a second data transmission period, the nodes in each group respectively send the first reduction result to a node in an adjacent group, and in the node in the adjacent group, a second reduction result of two nodes is obtained.

[0167] In the above embodiment, the preset topology is a parameter server topology, and the parameter server topology comprises a server node and a plurality of regular nodes.

[0168] The server node receives the reduction sub-data sent by the regular nodes, obtains a reduction result according to the reduction sub-data, and sends the reduction result to each regular node.

[0169] In the above embodiment, one node is selected from the plurality of nodes as the server node, and the remaining nodes are selected as the regular nodes.

[0170] In the context of the present application, a computer-readable storage medium can be a tangible medium that can contain or store a computer program for use by or in connection with an instruction execution system, apparatus, or device. A computer-readable storage medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of a machine-readable storage medium will include one or more lines of a program of instructions in a transitory signal, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0171] To provide for interaction with a user, the systems and techniques described here can be implemented on a computing card having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computing card. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

[0172] The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a user computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0173] The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system, to solve the defects of large management difficulty and weak business scalability in traditional physical host and VPS service.

[0174] It should be understood that the various forms of flow shown above can be reordered, added to, or have steps deleted. For example, the steps described in the present application can be performed in parallel, in series, or in a different order, as long as the desired results of the technical solutions of the present application can be achieved, which are not limited herein.

[0175] The above detailed description does not constitute a limitation on the protection scope of the present application. Those skilled in the art should understand that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A protocol calculation method of collective communication, characterized by, The application is applied to a target node in a preset topology structure, the target node is any node in the preset topology structure, the target node comprises a computing unit, a communication unit, a storage unit, a protocol information processing unit of a near storage unit and a protocol computing unit of a near communication unit, and the method comprises the following steps: In a first data transmission period, the communication unit receives first protocol sub-data sent by a first node, and the first node is a previous hop node of the target node determined according to the preset topology structure; The protocol information processing unit determines a protocol type, a data address and first data sent by the first node according to the first protocol sub-data, the data address is an address corresponding to each protocol sub-data obtained by fragmenting to-be-protocol data, and the protocol information processing unit determines a protocol result of the protocol sub-data; The protocol information processing unit sends the protocol type, the data address and the first data sent by the first node to the protocol computing unit; The protocol computing unit reads local data corresponding to the data address from the storage unit according to the data address, performs protocol calculation on the first data and the local data according to the protocol type, obtains second data, the second data is a protocol calculation result of a current data transmission period, and writes the second data into the data address; In a second data transmission period, the protocol information processing unit reads the second data in the storage unit, generates second protocol sub-data according to the second data, and the communication unit sends the second protocol sub-data to a second node, and the second node is a next hop node of the target node determined according to the preset topology structure; the second data transmission period is a next data transmission period of the first data transmission period; after a plurality of data transmission periods, until data corresponding to the data address complete protocol calculation of all nodes.

2. The method of claim 1, wherein, The protocol information processing unit determines a protocol type, a data address and first data sent by a first node according to the first protocol sub-data, and comprises the following steps: If the protocol information processing unit determines to perform protocol calculation according to the protocol sub-data, the protocol information processing unit determines a protocol type, a data address and first data sent by a first node according to the first protocol sub-data; If the protocol information processing unit determines not to perform protocol calculation according to the protocol sub-data, the protocol information processing unit transmits the first data to the storage unit.

3. The method of claim 2, wherein, According to the first protocol sub-data, the protocol type, the data address and the first data sent by the first node are determined, and comprise the following steps: According to the message header of the protocol sub-data, the protocol type and the data address are determined, according to the message body of the protocol sub-data, the first data sent by the first node is determined.

4. The method of claim 3, wherein, According to the second data, the second protocol sub-data is generated, and comprises the following steps: According to the message body of the second protocol sub-data, according to the protocol operation type and the data address, the message header of the second protocol sub-data is generated, and according to the message body and the message header, the second protocol sub-data is generated.

5. The method of claim 1, wherein, The protocol information processing unit sends the protocol type, the data address and the first data sent by the first node to the protocol computing unit, and comprises the following steps: The protocol information processing unit sends the protocol type, the data address, and the first data sent by the first node to the pipeline data cache; When the amount of data stored in the pipeline data cache reaches a preset data amount, the protocol calculation unit reads the cached data from the pipeline data cache.

6. The method of claim 1, wherein, The preset topology is a ring topology; the ring topology includes a plurality of nodes, sequentially being a node A, a node B, a node C, and a node N; a previous hop node of the node A is the node N, and a next hop node of the node A is the node B, and the like, to form a ring topology structure; If the target node is the node A, the first node is the node N, and the second node is the node B.

7. The method of claim 1, wherein, The preset topology is a flat butterfly topology; the flat butterfly topology includes a plurality of groups, each group including two nodes, In a first data transmission period, the two nodes in each group perform protocol sub-data calculation to obtain a first protocol result; In a second data transmission period, the nodes in each group respectively send the first protocol result to a node in an adjacent group, and in the node in the adjacent group, a second protocol result of two nodes is obtained.

8. The method of claim 1, wherein, The preset topology is a parameter server topology, and the parameter server topology includes a server node and a plurality of regular nodes; The server node receives protocol sub-data sent by the regular nodes, obtains a protocol result according to the protocol sub-data, and sends the protocol result to each regular node.

9. The method of claim 8, wherein, One node is selected from the plurality of nodes as a server node, and the remaining nodes are regular nodes.

10. A protocol computing apparatus for collective communication, characterized by, The target node is applied to a preset topology structure, the target node is any one node in the preset topology structure, and the target node includes a calculation unit, a communication unit, a storage unit, a protocol information processing unit of a near communication unit, and a protocol calculation unit of a near storage unit. The receiving module is configured to, in a first data transmission period, receive first protocol sub-data sent by a first node by the communication unit, the first node being a previous hop node of the target node determined according to the preset topology structure; The protocol information acquisition module is configured to determine, by the protocol information processing unit, a protocol type, a data address, and first data sent by the first node according to the first protocol sub-data, the data address being an address corresponding to each protocol sub-data obtained by sharding to-be-protocol data, and the protocol result used for storing the protocol sub-data; The on-board transmission module is configured to send, by the protocol information processing unit, the protocol type, the data address, and the first data sent by the first node to the protocol calculation unit; The writing module is configured to read, by the protocol calculation unit, local data corresponding to the data address from the storage unit according to the data address, perform protocol calculation on the first data and the local data according to the protocol type to obtain second data, the second data being a protocol calculation result of a current data transmission period, and write the second data to the data address. The sending module is configured to, in a second data transmission period, read second data in the storage unit by the protocol information processing unit, generate second protocol sub-data according to the second data, and send the second protocol sub-data to a second node by the communication unit, the second node being a next hop node of a target node determined according to a preset topology structure; the second data transmission period is a next data transmission period of the first data transmission period; after a plurality of data transmission periods, the data corresponding to the data address completes protocol calculation of all nodes.

11. A computing card, comprising: The computing card serves as a target node in the preset topology structure, the target node being any one node in the preset topology structure, and the computing card comprises a computing unit, a communication unit, a storage unit, a protocol information processing unit of a near communication unit, and a protocol calculation unit of a near storage unit. The communication unit is configured to, in a first data transmission period, receive first protocol sub-data sent by a first node, the first node being a previous hop node of a target node determined according to a preset topology structure. The protocol information processing unit is configured to determine a protocol type, a data address, and first data sent by the first node according to the first protocol sub-data, the data address being an address corresponding to each protocol sub-data obtained by fragmenting to-be-protocol data, and the protocol result of the protocol sub-data being used for storing protocol results of protocol sub-data. The protocol information processing unit is configured to send the protocol type, the data address, and the first data sent by the first node to the protocol calculation unit. The protocol calculation unit is configured to read local data corresponding to the data address from the storage unit according to the data address. The protocol calculation unit is configured to perform protocol calculation on the first data and the local data according to the protocol type to obtain second data, the second data being a protocol calculation result of a current data transmission period, and write the second data into the data address. The protocol information processing unit is further configured to, in a second data transmission period, read second data in the storage unit, and generate second protocol sub-data according to the second data. The communication unit is further configured to send the second protocol sub-data to a second node, the second node being a next hop node of a target node determined according to a preset topology structure; the second data transmission period being a next data transmission period of the first data transmission period; after a plurality of data transmission periods, the data corresponding to the data address completes protocol calculation of all nodes.

12. A distributed system, characterized by The method comprises the following steps: A plurality of computing cards as claimed in claim 11; the plurality of computing cards are connected with an interconnection network through respective communication units.

13. A storage medium containing computer executable instructions for performing the method of any one of claims 1-9 when executed by a computer processor.

Images