Training state determination method, training platform, device, equipment, medium and product

Abstract

The embodiment of the invention provides a training state determination method, a training platform, a device, equipment, a medium and a product, and the method comprises the steps: in the training process of an artificial intelligence cluster, a monitoring node can obtain and analyze each feature data packet in a feature data set. Afterwards, the monitoring node can integrate the analyzed gradient data according to the sequence of the feature data packets in the feature data set. In the method, the gradient data obtained after the monitoring nodes are integrated can be regarded as the global gradient data of the artificial intelligence cluster, and the global gradient data can describe the training state of the artificial intelligence cluster more completely and more accurately. And if so, when the artificial intelligence cluster training fails, the monitoring node can recover the artificial intelligence cluster to a normal state by using the global gradient data so as to facilitate subsequent continuous training.

Description

Technical Field

[0001] This invention relates to the field of network technology, and in particular to a training state determination method, training platform, device, equipment, medium, and product. Background Technology

[0002] Artificial Intelligence (AI) clusters refer to clusters of nodes with massive parameters, complex computational structures, and the need for training. AI clusters typically consist of multiple interconnected computing nodes. The collaborative work between these nodes can process massive amounts of data to complete various complex tasks, such as natural language processing and image recognition.

[0003] During the training of an AI swarm, it is necessary to periodically analyze and record the training status of the AI ​​swarm. This training status includes the parameters of the AI ​​swarm obtained after different training epochs, which can also reflect the degree of convergence of the AI ​​swarm. In the event of a training failure, the AI ​​swarm can be restored to the most recently recorded training status so that swarm training can continue based on this status.

[0004] To ensure the effectiveness of AI cluster training, accurately obtaining the training status of the AI ​​cluster has become an urgent problem to be solved. Summary of the Invention

[0005] In view of this, embodiments of the present invention provide a training state determination method, training platform, device, equipment, medium, and product for accurately obtaining the training state of an artificial intelligence cluster.

[0006] In a first aspect, embodiments of the present invention provide a training state determination method, comprising:

[0007] Obtain a feature dataset, wherein the feature data package in the feature dataset contains gradient data generated during the training of the artificial intelligence cluster;

[0008] The feature data packets in the feature dataset are parsed;

[0009] If the gradient data of the artificial intelligence cluster is obtained by parsing, the parsed gradient data is integrated according to the order of the feature data package in the feature dataset to obtain global gradient data.

[0010] The training state of the artificial intelligence cluster is recorded, and the training state includes the global gradient data.

[0011] In a second aspect, embodiments of the present invention provide an electronic device, including: a direct memory access engine and a computing chip;

[0012] The direct memory access engine is used to acquire a feature dataset, wherein the feature data package in the feature dataset contains gradient data generated during the training of the artificial intelligence cluster.

[0013] The computing chip is used to parse the feature data packets in the feature dataset; if the gradient data of the artificial intelligence cluster is obtained by parsing, the parsed gradient data is integrated according to the order of the feature data packets in the feature dataset to obtain global gradient data; the training state of the artificial intelligence cluster is recorded, and the training state includes the global gradient data.

[0014] Thirdly, embodiments of the present invention provide an electronic device, including a direct memory access engine and a processing module;

[0015] The direct memory access engine is used to acquire a feature dataset, wherein the feature data package in the feature dataset contains gradient data generated during the training of the artificial intelligence cluster.

[0016] The processing module is used to parse the feature data packets in the feature dataset; if the gradient data of the artificial intelligence cluster is obtained by parsing, the parsed gradient data is integrated according to the order of the feature data packets in the feature dataset to obtain global gradient data; and the training state of the artificial intelligence cluster is recorded, the training state including the global gradient data.

[0017] Fourthly, embodiments of the present invention provide a training platform for an artificial intelligence cluster, comprising: an artificial intelligence cluster including computing nodes and monitoring nodes;

[0018] The computing node is used to generate feature data packets containing gradient data during the training process of the artificial intelligence cluster.

[0019] The monitoring node acquires a feature dataset containing the feature data packet;

[0020] The feature data packet is parsed;

[0021] If the gradient data of the artificial intelligence cluster is obtained by parsing, the parsed gradient data is integrated according to the order of the feature data package in the feature dataset to obtain global gradient data.

[0022] The training state of the artificial intelligence cluster is recorded, and the training state includes the global gradient data.

[0023] Fifthly, embodiments of the present invention provide a non-transitory machine-readable storage medium storing executable code, which, when executed by a processor of an electronic device, enables the processor to at least implement the training state determination method as described in the first aspect.

[0024] In a sixth aspect, embodiments of the present invention provide a computer program product, the computer program product including a computer program or instructions, which, when executed by a processor, cause the processor to implement the training state determination method as described in the first aspect above.

[0025] The method for determining the training state of an artificial intelligence cluster provided in this invention allows a monitoring node to acquire a feature dataset during the training process, and then parse the feature data packets within the feature dataset. For the gradient data parsed from the feature data packets, the monitoring node can also integrate the parsed gradient data according to the order of the feature data packets in the feature dataset to obtain global gradient data. This global gradient data can also be recorded by the monitoring node as part of the training state of the artificial intelligence cluster.

[0026] In the above method, the monitoring node integrates the gradient data from different data packets to obtain the global gradient data of the AI ​​cluster. This global gradient data can more completely describe the training state of the AI ​​cluster. Furthermore, the monitoring node considers the order of the data packets when integrating the gradient data; therefore, the integrated global gradient data can more accurately describe the training state of the AI ​​cluster. Thus, when the AI ​​cluster experiences a training failure, this global gradient data can be used to restore the AI ​​cluster to a normal state, facilitating subsequent training. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the structure of an artificial intelligence cluster training platform provided in an embodiment of the present invention;

[0029] Figure 2 A flowchart of a training state determination method provided in an embodiment of the present invention;

[0030] Figure 3A flowchart of another training state determination method provided in an embodiment of the present invention;

[0031] Figure 4a This is a schematic diagram of a feature data packet synchronization process provided in an embodiment of the present invention;

[0032] Figure 4b This is a schematic diagram of a feature data packet sending mode provided in an embodiment of the present invention;

[0033] Figure 5 A schematic diagram illustrating the pipeline operation process of a node according to an embodiment of the present invention;

[0034] Figure 6a This is a schematic diagram illustrating the storage state of feature data packets in different computing nodes, as provided in an embodiment of the present invention.

[0035] Figure 6b This is a schematic diagram illustrating the redundancy state of feature data packets in a monitoring node when it monitors two computing nodes, as provided in an embodiment of the present invention.

[0036] Figure 6c This is a schematic diagram illustrating the redundancy state of feature data packets in a monitoring node when the monitoring node monitors two or more computing nodes, as provided in an embodiment of the present invention.

[0037] Figure 7 This is a schematic diagram illustrating the working process of a monitoring node according to an embodiment of the present invention;

[0038] Figure 8 This is a schematic diagram illustrating the process of a monitoring node intercepting characteristic data packets, provided in an embodiment of the present invention.

[0039] Figure 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;

[0040] Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention;

[0041] Figure 11 This is a schematic diagram of the structure of another electronic device provided in an embodiment of the present invention;

[0042] Figure 12 This is a schematic diagram of the structure of another electronic device provided in an embodiment of the present invention;

[0043] Figure 13 This is a schematic diagram of a training state determination device provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0045] The terminology used in the embodiments of this invention is for the purpose of describing particular embodiments only and is not intended to limit the invention. The singular forms “a,” “the,” and “the” used in the embodiments of this invention and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. “Multiple” generally includes at least two, but does not exclude the inclusion of at least one.

[0046] It should be understood that the term "and / or" used in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0047] Depending on the context, the words “if” or “suppose” as used here can be interpreted as “when” or “in response to determination” or “in response to identification.” Similarly, depending on the context, the phrases “if determination” or “if identification (of the condition or event of the statement)” can be interpreted as “when determination” or “in response to determination” or “when identification (of the condition or event of the statement)” or “in response to identification (of the condition or event of the statement).”

[0048] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this invention are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0049] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a product or system comprising a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a product or system. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the product or system that includes said element.

[0050] Before describing the various embodiments provided by the present invention in detail, the application scenarios of the various embodiments provided by the present invention can also be introduced in conjunction with the background technology.

[0051] In practice, AI clusters can achieve convergence through multiple rounds of iterative training. The training parameters obtained after each iteration can be considered as the training state of the AI ​​cluster after that iteration. This training state can also reflect the degree of convergence of the AI ​​cluster, and the training state obtained after each iteration also corresponds to a checkpoint.

[0052] In the process of multi-round iterative training, when a failure occurs in the Nth round of training, in order to ensure training efficiency, the training state of the artificial intelligence cluster is often restored to any round before the Nth round, that is, the artificial intelligence cluster is restored to a previous checkpoint, and training continues on this basis.

[0053] Therefore, to restore the training state, the training state corresponding to each training round can be recorded during the training process. It is easy to understand that the completeness and accuracy of the recorded training state directly affect the subsequent training effect of the AI ​​cluster. To ensure the training effect of the AI ​​cluster, the methods provided in the following embodiments of the present invention can be used to accurately and completely record and save the training state of the AI ​​cluster.

[0054] Based on the above description, some embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Where there is no conflict between the embodiments, the following embodiments and features can be combined with each other. Furthermore, the timing of the steps in the following method embodiments is merely an example and not a strict limitation.

[0055] Figure 1 This is a schematic diagram of the structure of an artificial intelligence cluster training platform provided in an embodiment of the present invention. Essentially, this platform is a node cluster, such as... Figure 1 As shown, there are two types of nodes in this node cluster: compute nodes and listener nodes.

[0056] Multiple computing nodes can form an AI cluster to be trained. The computing nodes can execute training logic on the training dataset to obtain a feature dataset for the AI ​​cluster. This feature dataset can consist of feature data packets containing feature values. The feature values ​​in the data packets can be calculated during the backpropagation phase of the AI ​​cluster training by processing the training data packets in the training dataset. These feature values ​​reflect the training state of the AI ​​cluster.

[0057] Optionally, the feature values ​​may include gradient data and / or activation values. Activation values ​​can represent the degree of response of each neuron in the cluster to the input data. Gradient data is used to quantify the difference between the actual output of the AI ​​cluster and the supervisory information of the training data (i.e., the reference output). Therefore, gradient data can also guide the adjustment of cluster parameters to optimize cluster performance.

[0058] The monitoring node does not need to participate in the training of the AI ​​cluster. It can directly obtain the feature dataset generated by the computing nodes and analyze the feature values ​​to determine the cluster's training state. The monitoring node can also record the analyzed training state so that it can be used in case of training failures in the AI ​​cluster.

[0059] Alternatively, in practice, a specific instance of an artificial intelligence cluster could be a large model.

[0060] Optionally, the compute node can be a GPU node that utilizes the computing power provided by a Graphics Processing Unit (GPU) to perform cluster training. Optionally, the monitoring node can be a CPU node that utilizes the computing power provided by a Central Processing Unit (CPU) to perform data analysis. Optionally, depending on actual needs, the compute node and monitoring node can be a single type of compute node such as a GPU node or a CPU node, or they can be heterogeneous compute nodes.

[0061] Optionally, the transmission of various data between compute nodes and listening nodes can be facilitated by a data center network; the transmission of various data between compute nodes can also be facilitated by a data center network.

[0062] The working process of the monitoring node will be described in detail below with reference to the method embodiments provided by the present invention.

[0063] Figure 2 This is a flowchart illustrating a training state determination method provided in an embodiment of the present invention. This method provided in the embodiment of the present invention can be executed by a monitoring node. Figure 2 As shown, the method may include the following steps:

[0064] S101, Obtain the feature dataset. The feature data package in the feature dataset contains gradient data, which is generated during the training process of the artificial intelligence cluster.

[0065] according to Figure 1 As shown in the illustrated embodiment, the feature dataset can be calculated by the computing node, and the listening node can receive the feature dataset sent by the computing node. At this point, the listening node has completed the interception of the data packets. Furthermore, the acquired feature dataset can be composed of ordered feature data packets, and each feature data packet can include feature values. For a detailed introduction to feature datasets, feature data packets, and feature values, please refer to [link to relevant documentation]. Figure 1 The relevant descriptions in the illustrated embodiments will not be repeated here.

[0066] Optionally, different communication modes can be used between the monitoring node and the computing node to transmit the feature dataset. These communication modes may include peer-to-peer (P2P) communication, point-to-multipoint (P2MP) communication, and so on. Alternatively, the feature dataset can also be transmitted using various network communication devices such as switches in the data center network between the monitoring node and the computing node.

[0067] S102, parse the feature data packets in the feature dataset.

[0068] S103, if the gradient data of the artificial intelligence cluster is obtained by parsing, the parsed gradient data is integrated according to the order of the feature data packet in the feature dataset to obtain the global gradient data.

[0069] S104, record the training state of the artificial intelligence cluster, the training state includes the global gradient data.

[0070] Furthermore, the monitoring node performs the analysis of the feature data packets by executing steps S102 to S104.

[0071] Specifically, the listening node can parse the feature data packets sequentially according to the order in which the feature data packets are in the feature dataset.

[0072] For any feature data packet in the feature dataset, in one scenario, if an activation value is parsed from that feature data packet, since the activation value is not recorded as the training state of the AI ​​cluster, the listening node can ignore this parsed activation value and continue parsing the remaining unparsed data packets in the feature dataset according to the packet order. In another scenario, if gradient data is parsed from that feature data packet, the listening node can save this gradient data and continue parsing the remaining data packets in order.

[0073] After parsing the feature data packets in the feature dataset, the listening node can further integrate the parsed gradient data packets according to their respective order in the feature dataset to obtain global gradient data. Optionally, the global gradient data can be recorded and saved to the corresponding storage space as the training state of the artificial intelligence cluster. The recorded training state can also correspond to a checkpoint.

[0074] Optionally, gradient data integration can specifically involve concatenating gradient data. For example, suppose the feature dataset contains three feature data packets, and the storage order of these three data packets in the feature dataset is data packet 1 → data packet 2 → data packet 3. Then, for gradient data extracted from data packets 1 to 3, the listening node can concatenate the gradient data extracted from the three data packets to obtain the global gradient data of the artificial intelligence cluster.

[0075] In this embodiment, during the training process of the artificial intelligence cluster, the monitoring node can acquire a feature dataset and then parse the feature data packets within the feature dataset. For the gradient data parsed from the feature data packets, the monitoring node can also integrate the parsed gradient data according to the order of the feature data packets in the feature dataset to obtain global gradient data. This global gradient data can also be recorded by the monitoring node as part of the training state of the artificial intelligence cluster.

[0076] In the above method, the monitoring node integrates the gradient data from different data packets to obtain the global gradient data of the AI ​​cluster. This global gradient data can more completely describe the training state of the AI ​​cluster. Furthermore, the monitoring node considers the order of the data packets when integrating the gradient data; therefore, the integrated global gradient data can more accurately describe the training state of the AI ​​cluster. Thus, when the AI ​​cluster experiences a training failure, this global gradient data can be used to restore the AI ​​cluster to a normal state, facilitating subsequent training.

[0077] Based on the above, the technical effects achieved by the methods provided by the present invention and the following embodiments can also be understood in conjunction with the following:

[0078] according to Figure 1 and Figure 2 As can be seen from the description in the illustrated embodiment, the data transmission method and content between the computing node and the listening node are complex during the training process of the artificial intelligence cluster. That is, the listening node can receive the feature dataset sent by the computing node through different communication modes, and the feature values ​​in the feature dataset are also of various types, such as gradient data or activation values.

[0079] The methods provided in the various embodiments of this invention are well-suited for scenarios with complex transmission methods and content. Furthermore, using the methods provided in the various embodiments of this invention eliminates the need to rewrite data packets or perform complex configurations of the various hardware devices used by monitoring nodes such as switches to obtain feature datasets, thereby reducing the implementation cost and difficulty of global gradient data recording.

[0080] Furthermore, regarding the iterative training process of the AI ​​cluster, specifically, after inputting the training dataset, different layers within the AI ​​cluster can be trained sequentially in a pre-defined training order for one round. After this round of training is completed, the AI ​​cluster can continue to be trained in the next round according to the aforementioned pre-defined training order. It is evident that any round of training can actually include two granularities: layer-level training and cluster-level training. The AI ​​cluster can include an input layer, at least one intermediate layer, and an output layer. The intermediate layer can specifically include convolutional layers, pooling layers, fully connected layers, etc.

[0081] Corresponding to cluster training, the feature dataset computed by the computing nodes can also include two granularities: layer-level feature datasets and cluster-level feature datasets. These two levels of feature datasets can also be obtained by the monitoring nodes. The layer-level feature dataset can be used as a subset to form the cluster-level feature dataset. Corresponding to the feature datasets, the global gradient data ultimately analyzed by the monitoring nodes can also include two granularities: layer-level global gradient data and cluster-level global gradient data. Easily understood, the monitoring nodes can first analyze the received layer-level feature dataset to obtain the layer-level global gradient data. After obtaining the global gradient data corresponding to each layer, the monitoring nodes also obtain the cluster-level global gradient data, which can be used in case of training failures.

[0082] Although the above Figure 2 The embodiments shown do not limit the granularity of the dataset and gradient data obtained by the monitoring nodes, but it is easy to understand that the feature dataset and global gradient data obtained by the monitoring nodes can be at the same level, such as the same layer level or cluster level.

[0083] When the listening node receives a layer-level feature dataset, the listening node can further... Figure 3 The method shown determines the global gradient data at the layer level. Since the feature dataset at the layer level can be a subset of the feature dataset at the cluster level, for the sake of brevity in the following embodiments, the feature dataset at the cluster level can be referred to as the feature dataset, and the feature dataset at the layer level can be referred to as the subset.

[0084] but Figure 3 A flowchart illustrating another training state determination method provided in an embodiment of the present invention. Figure 3 As shown, the method may include the following steps:

[0085] S201, Obtain the target subset. The feature data package contained in the target subset corresponds to one round of training of the target layer in the artificial intelligence cluster. The target layer is any layer in the artificial intelligence cluster.

[0086] The target subset can be computed by the computing nodes, and the listening nodes can receive this target subset sent by the computing nodes. At this point, the listening nodes have completed the interception of the data packets. The feature data packets contained in the target subset can correspond to one round of training in the target layer, which can be any layer in the artificial intelligence cluster.

[0087] S202, parse the feature data packets in the target subset.

[0088] S203. If the gradient data of the artificial intelligence cluster is obtained by parsing, the parsed gradient data is integrated according to the order of the feature data packet in the feature dataset to obtain the global gradient data.

[0089] S204 records the training state of the artificial intelligence cluster, which includes global gradient data.

[0090] S205, if the activation value of the artificial intelligence cluster is obtained by parsing, then parse the remaining feature data packets in the feature dataset.

[0091] Furthermore, the monitoring node performs the analysis of the feature data packets by executing steps S202 to S205.

[0092] Specifically, the monitoring node can sequentially parse data packets with different characteristics within the target subset. According to... Figure 1 As can be seen from the description in the illustrated embodiment, the feature data packet may include gradient data or activation values. Therefore, the parsing result of any feature data packet in the target subset may include the following two cases: parsing gradient data or parsing activation values ​​from any feature data packet.

[0093] in, Figure 3In the illustrated embodiment, the feature data packets in the target subset are intercepted by the monitoring node. Specifically, during the training of an AI cluster using a distributed training method, there is a process where computing nodes synchronize their locally stored feature data packets to other different computing nodes. During this synchronization process, the feature data packets stored locally on different computing nodes may partially overlap, until they become completely identical. Therefore, during this feature data packet synchronization process, the monitoring node can also intercept overlapping feature data packets transmitted between different computing nodes in real time. The reason for the overlap of feature data packets can be explained by... Figure 4a The characteristic data packet synchronization process shown is understood.

[0094] like Figure 4a As shown, assume there are four computing nodes in the AI ​​cluster: computing node A, computing node B, computing node C, and computing node D. The feature data packets generated by computing node A include packets a0 to d0; those generated by computing node B include packets a1 to d1; those generated by computing node C include packets a2 to d2; and those generated by computing node D include packets a3 to d3. Furthermore, the computing nodes can use Ring All Reduce to synchronize the feature data packets. In this case, the feature data packet synchronization process can be combined with... Figure 4a The illustrated embodiments are for understanding. As... Figure 4a As shown, in the process of executing Ring all reduce to achieve feature data packet synchronization, communication primitives such as Reduce-scatter and Algather are used sequentially. That is to say, in... Figure 4a In the feature data packet synchronization process shown, the all-reduce computation can include two stages: Reduce-scatter computation and Algather computation. After performing Reduce-scatter computation, the feature data packets corresponding to the Reduce-scatter computation are obtained; similarly, after performing Allgather computation, the feature data packets corresponding to the Algather computation are obtained. Therefore, the feature data packets contained in the target subset can correspond to different types of communication primitives, and the communication primitives can reflect whether the feature data packets contain gradient data.

[0095] For parsing any given feature data packet, optionally, after receiving any feature data packet, the listening node can determine the corresponding communication primitive. Based on this communication primitive, it can be determined whether the feature data packet contains gradient data. The method for determining the communication primitive corresponding to the feature data packet can be combined with the following... Figure 4a The illustrated examples are for understanding.

[0096] The communication primitives can include non-all reduce, reduce-scatter, all-collector, etc. The feature data packets corresponding to non-all reduce do not contain gradient data, while those corresponding to reduce-scatter and all-collector features contain gradient data. However, the listening node needs to parse the gradient data contained in the feature data packets corresponding to all-collector features. Therefore, when the listening node determines that the communication primitive corresponding to any feature data packet is all-collector, it can parse the packet to extract the gradient data. Even when the listening node determines that the communication primitive corresponding to any feature data packet is not all-collector, it can still parse the packet to obtain the activation values ​​contained within it.

[0097] After parsing all feature data packets in the target subset, for feature data packets containing parsed gradient data, the listening node can concatenate the parsed gradient data according to the order of these feature data packets in the target subset to obtain the global gradient data of the target layer. Optionally, the order of feature data packets in the target subset can be represented by the index of the feature data packet. The index of the feature data packet can be calculated based on the sequence number of the feature data packet, or the sequence number of the feature data packet can be directly used as the index of the feature data packet.

[0098] In this embodiment, the monitoring node can obtain layer-level global gradient data by parsing data packets and integrating gradient data. Furthermore, any content not described in detail in this embodiment and the technical effects achieved can be found in the relevant descriptions of the above embodiments, and will not be repeated here.

[0099] For the process of determining the communication primitive corresponding to any feature data packet, optionally, after receiving any feature data packet sent by the computing node, the listening node can first count the cumulative number of data packets received by the current listening node. Simultaneously, the listening node can also obtain the first number of feature data packets generated during cluster training, which is the total number of data packets in the cluster-level feature dataset. The listening node can also obtain the second number of feature data packets generated by different layers in the artificial intelligence cluster, which is the total number of data packets in the layer-level feature dataset. Optionally, the aforementioned first and second numbers can be preset. Furthermore, the listening node can also obtain the preset training order of different layers in the artificial intelligence cluster and the preset sending order of feature data packets.

[0100] The computing nodes can send feature data packets according to a preset sending order. The feature data packet sending mode can be found in [reference needed]. Figure 4a .like Figure 4a As shown, the preset sending order may include first sending the feature data packet corresponding to Non-all reduce, then sending the feature data packet corresponding to Reduce-scatter, and finally sending the feature data packet corresponding to Alligather.

[0101] Finally, the listening node can determine whether any feature data packet corresponds to the target layer and the communication primitive corresponding to that feature data packet based on the cumulative received amount, the first quantity, the second quantity, the preset training order, and the preset sending order obtained above.

[0102] In this embodiment, during the training process of the artificial intelligence cluster, since the training order of the cluster and the sending order of data packets are fixed, after receiving a feature data packet, the listening node determines which layer of the artificial intelligence cluster the feature data packet belongs to and the corresponding communication primitive based on the cumulative number of data packets received.

[0103] Based on the feature data packet transmission pattern shown in Figure 4, the indices of different feature data packets corresponding to Algather, which contain gradient data in the target subset, should be continuous. However, during the integration of gradient data, if the monitoring node determines that the indices of the feature data packets are not continuous, it can determine that feature data packets have been lost, i.e., gradient data has been lost. In this case, alternatively, the monitoring node can send a packet loss notification message to the computing node, which, in response, sends its own computed global gradient data to the monitoring node. Therefore, using this method, even if packet loss occurs, the monitoring node can still obtain the global gradient data.

[0104] In this process, the computing nodes execute training logic on the training dataset to obtain the feature dataset, and at the same time, they also execute the same logic as the listening nodes. That is, the computing nodes also parse the feature data packets they generate and integrate the parsed gradient data to finally obtain the global gradient data.

[0105] Alternatively, when gradient data is lost and the listening node also fails to receive global gradient data sent by the computing node, the listening node can further send a packet loss notification message to the control platform corresponding to the artificial intelligence cluster, so that the control platform can handle the computing node that has lost packets.

[0106] In addition, as Figure 1As described in the illustrated embodiment, both the computing node and the listening node can be CPU nodes, GPU nodes, or heterogeneous nodes. Optionally, the computing node can also be a multi-core node. In this case, the steps of parsing the feature data packet and calculating the index can be performed by the first core of the computing node; the step of integrating gradient data can be performed by the second core of the computing node; and the steps of recording, sending, and updating the global gradient data to the artificial intelligence cluster can be performed by the third core of the computing node. This pipelined operation of the computing node can also be combined with... Figure 5 (a) Understanding in the text.

[0107] In this embodiment, each step is assigned to a different core for execution, which improves the competition for computing resources among nodes. Furthermore, this pipelined processing method allows the next core to begin processing immediately after the previous core has finished, thus improving the efficiency of the computing nodes.

[0108] Similar to compute nodes, listener nodes can also be multi-core nodes. Therefore, for... Figure 2 and Figure 3 In the illustrated embodiment, the packet analysis process of the monitoring node can be executed by different cores in a multi-core node. Specifically, the steps of parsing feature packets and calculating indices can be executed by the first core of the monitoring node; the step of integrating gradient data can be executed by the second core of the monitoring node; and the recording of global gradient data can be executed by the third core of the monitoring node. This pipelined operation of the monitoring node can also be combined with... Figure 5 (b) Understanding in the text.

[0109] In this embodiment, each step is assigned to a different core for execution, which improves the competition for computing resources among nodes. Furthermore, this pipeline approach allows the next core to process data immediately after the previous core finishes, thus improving the efficiency of the monitoring nodes. This enables the monitoring nodes to determine global gradient data within the same time window, mitigating the negative impact of the slower processing speed of the monitoring nodes on the overall training efficiency of the AI ​​cluster.

[0110] When packet loss occurs, the above embodiments have already provided a solution: the listening node can directly obtain global gradient data from the computing node. Alternatively, this embodiment also provides the following alternative solution.

[0111] Specifically, for AI clusters composed of multiple computing nodes, a distributed training approach is typically used to improve training efficiency. This involves dividing the training dataset according to the number of computing nodes, allowing each node to execute training logic on its assigned portion to obtain feature data. Since each node uses a portion of the training dataset, the gradient data parsed from the feature data generated by any node and then integrated yields local gradient data, not global gradient data. To ensure each node receives global gradient data, during the backpropagation phase of the AI ​​cluster training, any node can synchronize its locally stored feature data to other nodes. During this synchronization process, feature data from different nodes may partially overlap, eventually becoming identical. Further details can be found later. Figure 3 The relevant descriptions in the illustrated embodiments will not be repeated here.

[0112] Assuming there are N computing nodes in the AI ​​cluster, the process of synchronizing feature data packets between the computing nodes can also be combined with... Figure 4a The illustrated embodiments are for understanding. As... Figure 4a As shown, any computing node can send the feature data packets it generates, corresponding to Reduce-scatter, to the remaining N-1 computing nodes respectively.

[0113] Furthermore, after each round of feature data packet transmission, the computing node performs an all-reduce calculation. Therefore, the feature data packets corresponding to Reduce-scatter and those corresponding to Algather will each undergo N-1 all-reduce calculations. The all-reduce calculation can include two stages: performing Reduce-scatter calculation on the feature data packets and performing Algather calculation on the feature data packets. After performing Reduce-scatter calculation on the feature data packets, the feature data packets corresponding to Reduce-scatter are obtained; similarly, after performing Algather calculation on the feature data packets, the feature data packets corresponding to Algather are obtained.

[0114] Furthermore, during the synchronization of feature data packets between different computing nodes, due to the partial overlap between these packets, it's possible to configure one listening node in the node cluster to monitor at least two computing nodes. This can mitigate the problem of listening nodes failing to obtain global gradient data due to packet loss. The specific reasons can be understood through the following example.

[0115] For example, suppose the node set contains compute node 1, compute node 2, ..., compute node N. Compute node 1 generates feature data packet 1, compute node 2 generates feature data packet 2, ..., compute node N generates feature data packet N. Furthermore, a listening node can listen to at least two compute nodes.

[0116] Based on the above premise, during the process of computing nodes synchronizing the feature data packets they generate, at a certain moment, just as Figure 6a As shown, feature data packets 1 and 2 are stored locally on compute node 1, feature data packets 2 and 3 are stored locally on compute node 2, feature data packets 1 and 3 are stored locally on compute node 3, and compute nodes 1 to 3 also each store feature data packets 4 to N generated by compute nodes 4 to N.

[0117] In the above scenario, when a listening node simultaneously listens to two compute nodes, i.e., the listening node listens to compute node 1 and compute node 2, as follows: Figure 6b As shown, the listening node can receive different feature data packets stored locally by computing nodes 1 to 3. At this time, feature data packets 2 and 4 to N are redundant, while feature data packets 1 and 3 are not redundant. Therefore, when at least one of the feature data packets 2, 4 to N is lost during transmission, it will not affect the listening node's acquisition of global gradient data.

[0118] In this embodiment, when a listening node listens to two computing nodes, the feature data packets may have some redundancy. This redundancy can, to some extent, mitigate the impact of packet loss on the acquisition of global gradient data.

[0119] In the above scenario, when a listening node simultaneously listens to two or more compute nodes, for example, when a listening node simultaneously listens to compute nodes 1 through 3, such as... Figure 6c As shown, the listening node can receive different feature data packets stored locally by computing nodes 1 to 3. At this time, feature data packets 4 to N have 3 redundancies, and feature data packets 1 to 3 have 2 redundancies. Therefore, if any of the feature data packets 1 or N is lost, it will not affect the listening node's acquisition of global gradient data.

[0120] Compared to Figure 6b As shown in this embodiment, when a listening node listens to two or more computing nodes, the feature data packets are redundant. Therefore, the impact of packet loss on global gradient data acquisition can be improved.

[0121] In addition, in practice, if packet loss still occurs after the monitoring node monitors multiple computing nodes, it is usually because the computing node is faulty. Optionally, a fallback solution is that the control node in the node cluster can also send a notification message to the computing node that generated the lost characteristic data packet in order to troubleshoot the computing node.

[0122] for Figure 2 and Figure 3 In the illustrated embodiment, upon receiving a feature data packet at any level, the listening node optionally writes it into its memory before parsing it. Therefore, for the feature data packet writing process, the listening node can optionally receive the feature data packets sequentially sent by the computing node using the communication link between itself and the computing node, and write these feature data packets into its buffer. The listening node can also sequentially read the feature data packets from the buffer into its memory. Optionally, the listening node's buffer can be a circular buffer. Afterward, the listening node can read the feature data packets from memory and parse them.

[0123] Optionally, to improve the speed of writing data packets to memory, the listening node can also utilize at least one of its configured Direct Memory Access (DMA) engines to write feature data packets from the buffer to memory. This increased data packet writing speed can also indirectly improve the speed of subsequent gradient data parsing and integration.

[0124] Based on the descriptions in the above embodiments, the working process of the monitoring node can also be combined with... Figure 7 Understanding. For example... Figure 7 As shown, this working process is essentially the same as Figure 5 The content shown in (b) is the same.

[0125] In this embodiment, the configuration of the hardware engine, namely the DMA engine, in the monitoring node can improve the speed of writing feature data packets into memory, and can also further improve the speed of data packet parsing and gradient data integration.

[0126] And in Figure 1 In the AI ​​cluster shown, the compute nodes can complete the training logic within a specified time window, thus completing one round of training for the AI ​​cluster. Furthermore, the compute nodes and the monitoring nodes operate independently; that is, while the compute nodes execute the training logic, the monitoring nodes can autonomously determine the global gradient data. The use of the DMA engine allows the monitoring nodes to also determine the global gradient data within the same time window, thereby mitigating the negative impact on the overall training efficiency of the AI ​​cluster caused by the slow processing speed of the monitoring nodes.

[0127] As described in the aforementioned embodiments, the computing node sends feature data packets sequentially. However, considering the stability of the communication link and the network environment, the feature data packets received by the monitoring node may be out of order. To improve this situation, optionally, the monitoring node can write the received feature data packets into a contiguous storage space of the buffer in an orderly manner according to the data packet sequence number. Optionally, the ordered feature data packets can be written into a contiguous storage space of the buffer according to consecutive virtual addresses.

[0128] After the feature data packets are stored in the buffer in an orderly manner, in order to ensure that the feature data packets are written to memory in an orderly manner, the listening node can optionally use the Data Plane Development Kit (DPDK) to convert the virtual address of the feature data packets into a physical address. Then the listening node can write the feature data packets from the buffer to memory in an orderly manner according to the converted physical address.

[0129] The above-mentioned ordered reading and writing process of feature data packets can also be combined Figure 8 understand.

[0130] In this embodiment, the monitoring node can ensure that the feature data packets are written to memory in an orderly manner through address translation.

[0131] The above embodiments have described the working process of the monitoring node from a methodological perspective. The monitoring node can also be described from a structural perspective below. As can be seen from the descriptions in the above method embodiments, the working process of the monitoring node mainly includes two parts: interception of feature data packets and analysis of feature data packets. Specifically, as... Figure 7 and Figure 8 As described in the illustrated embodiment, the interception of feature packets can be achieved using a hardware engine, namely a DMA engine. Alternatively, the analysis of feature packets can be implemented using software or hardware.

[0132] When the analysis of characteristic data packets can be implemented in hardware, the listening node mentioned in the embodiments of this invention can be manifested as a fully hardware-based electronic device. Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 9 As shown, electronic devices may include a direct memory access engine (i.e., a DMA engine) and a computing chip.

[0133] Alternatively, the computing chip can be a Field Programmable Gate Array (FPGA), an Application Specific Integrated Circuit (ASIC), or something similar.

[0134] The DAM engine is used to acquire the feature dataset, which contains gradient data packets generated during the training of the AI ​​cluster. The computing chip parses the different feature packets in the feature dataset. If gradient data for the AI ​​cluster is obtained, it is integrated according to the order of the different feature packets in the feature dataset to obtain global gradient data. The feature state of the AI ​​cluster is recorded; the training state includes the global gradient data.

[0135] The specific working process of the DMA engine and computing chip can be found in the relevant descriptions in the above method embodiments.

[0136] In this embodiment, the DMA engine in the electronic device can acquire a feature dataset. After the DMA engine writes the feature data packets from the feature dataset into memory, the computing chip in the electronic device can parse the feature data packets. For the gradient data parsed from the feature data packets, the computing chip can also integrate the parsed gradient data according to the order of the feature data packets in the feature dataset to obtain global gradient data. This global gradient data can also be recorded by the monitoring nodes as the training state of the artificial intelligence cluster.

[0137] In the above method, the computing chip integrates gradient data from different data packets to obtain the global gradient data of the AI ​​cluster. This global gradient data can more completely describe the training state of the AI ​​cluster. Furthermore, the listening node considers the order of the data packets when integrating the gradient data; therefore, the integrated global gradient data can more accurately describe the training state of the AI ​​cluster. Thus, when a training failure occurs in the AI ​​cluster, this global gradient data can be used to restore the AI ​​cluster to a normal state, facilitating subsequent training.

[0138] This embodiment provides a fully hardware-based monitoring node, which improves the speed of acquiring global feature data and reduces the processing load on the CPU. Furthermore, any details not described in this embodiment, as well as the technical effects achieved, can be found in the relevant descriptions in the above method embodiments, and will not be repeated here.

[0139] Optionally, Figure 9The electronic devices shown can also be integrated into electronic devices. Figure 10 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Figure 10 As shown, electronic devices may include: Figure 9 The diagram shows electronic components, a network interface card (NIC), and memory. The DMA engine within the electronic components can be integrated into the NIC. Since the electronic device allocates corresponding buffers and memory to the NIC, the DMA engine can read feature data packets from the feature dataset from the buffer and write them to memory.

[0140] In this embodiment, setting the DMA engine in the network card can reduce the occupation of kernel CPU resources during the processing of feature data packets, thereby reducing CPU utilization.

[0141] Alternatively, the DMA engine can be set up in the electronic device independently of the CPU and network card.

[0142] Figure 11 This is a schematic diagram of another electronic device provided in an embodiment of the present invention. (See attached diagram.) Figure 11 As shown, the electronic device may include a DMA engine and a processing module.

[0143] The DMA engine is used to acquire a feature dataset, in which feature data packets contain gradient data generated during the training of the AI ​​cluster. The processing module parses the feature data packets in the feature dataset. If gradient data for the AI ​​cluster is obtained, it integrates the parsed gradient data according to the order of the feature data packets in the feature dataset to obtain global gradient data. The training state of the AI ​​cluster, including the global gradient data, is recorded.

[0144] The specific working process of the DMA engine and processing module can be found in the relevant descriptions in the above method embodiments.

[0145] In this embodiment, the DMA engine in the electronic device can acquire a feature dataset. After the DMA engine writes the feature data packets from the feature dataset into memory, the processing module in the electronic device can parse the feature data packets. For the gradient data parsed from the feature data packets, the processing module can also integrate the parsed gradient data according to the order of the feature data packets in the feature dataset to obtain global gradient data. This global gradient data can also be recorded by the monitoring nodes as the training state of the artificial intelligence cluster.

[0146] In the above method, the processing module integrates gradient data from different data packets to obtain global gradient data for the AI ​​cluster. This global gradient data can more completely describe the training state of the AI ​​cluster. Furthermore, the listening node considers the order of the data packets when integrating the gradient data; therefore, the integrated global gradient data can more accurately describe the training state of the AI ​​cluster. Thus, when a training failure occurs in the AI ​​cluster, this global gradient data can be used to restore the AI ​​cluster to a normal state, facilitating subsequent training.

[0147] Furthermore, any content not described in detail in this embodiment, as well as the technical effects that can be achieved, can be found in the relevant descriptions in the above method embodiments, and will not be repeated here.

[0148] Optionally, Figure 11 The processing module in the electronic device shown can be a software module or a hardware module.

[0149] When the processing module is a software module Figure 12 This is a schematic diagram of another electronic device structure provided in an embodiment of the present invention. For example... Figure 12 As shown, the processing module can be implemented using a processor and memory.

[0150] The memory is used to store information that supports the electronic device in performing the above-mentioned tasks. Figures 1 to 8 The training state determination method provided in the illustrated embodiment has a program in which a processor is configured to execute a program stored in memory. The program includes one or more computer instructions, wherein the one or more computer instructions, when executed by the processor, can perform the steps described in the above method embodiments.

[0151] The structure of the electronic device may also include a communication interface for the electronic device to communicate with other devices or communication networks.

[0152] Optionally, the processor described above includes multiple cores. The step of parsing different feature data packets in the feature dataset can be implemented by the first core of the processor. If gradient data of the artificial intelligence cluster is obtained through parsing, the step of integrating the parsed gradient data according to the order of the different feature data packets in the feature dataset can be implemented by the second core of the processor. The step of recording global gradient data as the training state of the artificial intelligence cluster can be implemented by the third core of the processor.

[0153] When the processing module is a hardware module, it can be represented as a computing chip. In this case, the structural diagram of the electronic device can be as follows: Figure 9 As shown.

[0154] The training state determination apparatus of one or more embodiments of the present invention will be described in detail below. Those skilled in the art will understand that the apparatus can be configured using commercially available hardware components through the steps taught in this solution.

[0155] Figure 13 This is a schematic diagram of a training state determination device provided in an embodiment of the present invention, as shown below. Figure 13 As shown, the device includes:

[0156] The acquisition module 11 is used to acquire a feature dataset, wherein the feature data package in the feature dataset contains gradient data, which is generated during the training process of the artificial intelligence cluster.

[0157] The parsing module 12 is used to parse the feature data packets in the feature dataset.

[0158] The integration module 13 is used to integrate the parsed gradient data according to the order of the feature data package in the feature dataset if the gradient data of the artificial intelligence cluster is parsed, so as to obtain global gradient data.

[0159] The recording module 14 is used to record the training state of the artificial intelligence cluster, the training state including the global gradient data.

[0160] Optionally, the artificial intelligence cluster includes computing nodes for executing cluster training logic, and the method is applied to a listening node that listens to the computing nodes.

[0161] The acquisition module 11 is used to receive the feature data packet sent by the computing node according to the transmission link between the monitoring node and the computing node;

[0162] The feature data packets sent by the computing node are written into the buffer of the listening node;

[0163] Using at least one direct memory access engine configured in the listening node, the feature data packet read from the buffer is written into the memory of the listening node.

[0164] Optionally, the acquisition module 11 is used to receive feature data packets sent by at least two computing nodes respectively, wherein the feature data packets sent by different computing nodes overlap.

[0165] Optionally, the acquisition module 11 is used to convert the virtual address of the feature data packet in the buffer into the physical address of the memory; and write the feature data packet into the memory according to the physical address.

[0166] Optionally, the parsing module 12 is used to parse the feature data packets in the target subset of the feature dataset, wherein the feature data packets contained in the target subset correspond to a round of training of the target layer in the artificial intelligence cluster, and the target layer is any layer in the artificial intelligence cluster; the feature data packets in the target subset correspond to different types of communication primitives.

[0167] Optionally, the parsing module 12 is configured to parse any feature data packet in the target subset if the communication primitive corresponding to any feature data packet includes a full collection, so as to obtain the gradient data in the any feature data packet.

[0168] Optionally, the apparatus further includes: a communication primitive determination module 15, configured to determine the cumulative number of data packets received by the monitoring node in response to the reception of any of the characteristic data packets;

[0169] Obtain a first number of feature data packets generated during the training of the artificial intelligence cluster, and a second number of feature data packets generated by each of the different layers in the artificial intelligence cluster;

[0170] Obtain the preset training order and the preset sending order of feature data packets for different layers in the artificial intelligence cluster;

[0171] Based on the cumulative received amount, the first quantity, the second quantity, the preset training order, and the preset sending order, determine whether any feature data packet corresponds to the target layer and the communication primitive corresponding to any feature data packet.

[0172] Optionally, the integration module 13 is used to determine the index of any feature data packet based on the sequence number of any feature data packet; and to concatenate the gradient data parsed from the different feature data packets contained in the target subset according to their respective indices.

[0173] Optionally, the acquisition module 11 is configured to send a packet loss notification message to the computing node if the indices of different feature data packets in the target subset are not consecutive.

[0174] The computing node receives the global gradient data sent by the computing node, wherein, in response to receiving the packet loss notification message, the computing node integrates the gradient data parsed from the feature data packets contained in the feature subset to obtain the global gradient data.

[0175] Optionally, the device further includes a sending module 16, configured to send a packet loss notification message to the control platform corresponding to the artificial intelligence cluster if the indexes of the feature data packets in the target subset are not continuous, so that the control platform can process the computing nodes that have experienced packet loss.

[0176] Optionally, the parsing module 12 is used to parse the remaining feature data packets in the feature dataset if the activation value of the artificial intelligence cluster is obtained through parsing.

[0177] Optionally, the parsing module 12 is used to parse the target feature data packet if the communication primitive corresponding to the target feature data packet does not include the full collection, so as to obtain the activation value of the target feature data packet.

[0178] Figure 13 The device shown can perform Figures 1 to 8 For the methods shown in the embodiments, the parts not described in detail in this embodiment can be referred to the following: Figures 1 to 8 The relevant descriptions of the illustrated embodiments are provided below. For the execution process and technical effects of this technical solution, please refer to [link / reference]. Figures 1 to 8 The descriptions in the illustrated embodiments will not be repeated here.

[0179] Furthermore, embodiments of the present invention provide a non-transitory machine-readable storage medium for storing computer software instructions used in the aforementioned electronic device, which includes instructions for executing the above-mentioned... Figures 1 to 8 The training status shown determines the procedures involved.

[0180] In addition, embodiments of the present invention provide a computer program product. This computer program product includes a computer program or instructions. When the computer program or instructions are executed by a processor, the processor is able to perform the above-described functions. Figures 1 to 8 The steps or functions of the training state determination method are shown.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for determining training states, characterized in that, include: Obtain a feature dataset, wherein the feature data package in the feature dataset contains gradient data, which is generated during the training process of the artificial intelligence cluster; The feature data packets in the feature dataset are parsed; If the gradient data of the artificial intelligence cluster is obtained by parsing, the parsed gradient data is integrated according to the order of the feature data package in the feature dataset to obtain global gradient data. The training state of the artificial intelligence cluster is recorded, and the training state includes the global gradient data.

2. The method according to claim 1, characterized in that, The artificial intelligence cluster includes computing nodes for executing cluster training logic, and the method is applied to a listening node that monitors the computing nodes; The process of obtaining a feature dataset containing ordered data packets includes: Based on the transmission link between the monitoring node and the computing node, receive the feature data packet sent by the computing node; The feature data packets sent by the computing node are written into the buffer of the listening node; Using at least one direct memory access engine configured in the listening node, the feature data packet read from the buffer is written into the memory of the listening node.

3. The method according to claim 2, characterized in that, The receiving of the feature data packet sent by the computing node includes: Receive feature data packets sent by at least two computing nodes, with some overlap between the feature data packets sent by different computing nodes.

4. The method according to any one of claims 1 to 3, characterized in that, The parsing of different feature data packets in the feature dataset includes: The feature data packets in the target subset of the feature dataset are parsed. The feature data packets contained in the target subset correspond to a round of training of the target layer in the artificial intelligence cluster, and the target layer is any layer in the artificial intelligence cluster; the feature data packets in the target subset correspond to different types of communication primitives.

5. The method according to claim 4, characterized in that, The parsing of the feature data packets in the target subset includes: If the communication primitive corresponding to any feature data packet in the target subset includes full collection, then the feature data packet is parsed to obtain the gradient data in the feature data packet.

6. The method according to claim 5, characterized in that, The method further includes: In response to the reception of any of the characteristic data packets, determine the cumulative number of data packets received by the monitoring node; Obtain a first number of feature data packets generated during the training of the artificial intelligence cluster, and a second number of feature data packets generated by each of the different layers in the artificial intelligence cluster; Obtain the preset training order and the preset sending order of feature data packets for different layers in the artificial intelligence cluster; Based on the cumulative received amount, the first quantity, the second quantity, the preset training order, and the preset sending order, determine whether any feature data packet corresponds to the target layer and the communication primitive corresponding to any feature data packet.

7. The method according to claim 5, characterized in that, The step of integrating the parsed gradient data according to the order of the different feature data packets in the feature dataset includes: Determine the index of any feature data packet based on its sequence number; Gradient data parsed from the feature data packets of the target subset are concatenated according to the index of the feature data packets in the target subset.

8. The method according to claim 7, characterized in that, The method further includes: If the indices of the feature data packets in the target subset are not consecutive, a packet loss notification message is sent to the computing node. The computing node receives the global gradient data sent by the computing node, wherein, in response to receiving the packet loss notification message, the computing node integrates the gradient data parsed from the feature data packets contained in the feature subset to obtain the global gradient data.

9. The method according to claim 7, characterized in that, The method further includes: If the indexes of the feature data packets in the target subset are not continuous, a packet loss notification message is sent to the control platform corresponding to the artificial intelligence cluster so that the control platform can handle the computing nodes that have experienced packet loss.

10. The method according to claim 9, characterized in that, The parsing of the feature data packets in the target feature subset includes: If the communication primitive corresponding to the target feature data packet does not include the full collection, then the target feature data packet is parsed to obtain the activation value of the target feature data packet.

11. An artificial intelligence cluster training platform, characterized in that, include: This includes AI clusters with computing nodes and monitoring nodes; The computing node is used to generate feature data packets containing gradient data during the training process of the artificial intelligence cluster. The monitoring node acquires a feature dataset containing the feature data packet; The feature data packet is parsed; If the gradient data of the artificial intelligence cluster is obtained by parsing, the parsed gradient data is integrated according to the order of the feature data package in the feature dataset to obtain global gradient data. The training state of the artificial intelligence cluster is recorded, and the training state includes the global gradient data.

12. An electronic device, characterized in that, include: Direct memory access engine and computing chip; The direct memory access engine is used to acquire a feature dataset, wherein the feature data package in the feature dataset contains gradient data generated during the training of the artificial intelligence cluster. The computing chip is used to parse the feature data packets in the feature dataset; If the gradient data of the artificial intelligence cluster is obtained by parsing, the parsed gradient data is integrated according to the order of the feature data package in the feature dataset to obtain global gradient data; the feature state of the artificial intelligence cluster is recorded, and the training state includes the global gradient data.

13. The device according to claim 12, characterized in that, The electronic device is integrated in the electronic device, the direct memory access engine is integrated in the network card configured in the electronic device, and the electronic device allocates a buffer for the network card; The direct memory access engine is used to write the feature dataset in the buffer into the memory of the electronic device.

14. An electronic device, characterized in that, include: Direct memory access engine and processing module; The direct memory access engine is used to acquire a feature dataset, wherein the feature data package in the feature dataset contains gradient data generated during the training of the artificial intelligence cluster. The processing module is used to parse the feature data packets in the feature dataset; If the gradient data of the artificial intelligence cluster is obtained by parsing, the parsed gradient data is integrated according to the order of the feature data package in the feature dataset to obtain global gradient data. The training state of the artificial intelligence cluster is recorded, and the training state includes the global gradient data.

15. The device according to claim 14, characterized in that, The processing module includes a processor and a memory; the memory stores executable code, which, when executed by the processor, causes the processor to implement the training state determination method as described in any one of claims 1 to 10.

16. The device according to claim 15, characterized in that, The processor includes multiple cores; wherein the step of parsing the feature data packets in the feature dataset is implemented by the first core of the processor; If the gradient data of the artificial intelligence cluster is obtained by parsing, the step of integrating the parsed gradient data according to the order of the feature data package in the feature dataset is implemented by the second core of the processor; The step of recording the characteristic state of the artificial intelligence cluster is implemented by the third core of the processor.

17. The device according to claim 14, characterized in that, The processing module includes a computing chip for executing the training state determination method as described in any one of claims 1 to 10.

18. The device according to claim 14, characterized in that, The device includes a network interface card (NIC) and memory, the direct memory access engine is integrated into the NIC, and the electronic device allocates a buffer for the NIC. The direct memory access engine is used to write the feature dataset in the buffer into the memory of the electronic device.

19. A non-transitory machine-readable storage medium, characterized in that, The non-transitory machine-readable storage medium stores executable code that, when executed by a processor of an electronic device, causes the processor to perform the training state determination method as described in any one of claims 1 to 10.

20. A computer program product, characterized in that, The computer program product includes a computer program or instructions that enable the computer program or instructions to perform the steps in the training state determination method according to any one of claims 1 to 10.

Images