Method for detecting silent data corruption in training process of ai model, and related device

Abstract

A method for detecting a silent data corruption in a training process of an AI model (110), and a related device. The AI model (110) comprises a plurality of network layers, and a first network layer is one network layer or part of one network layer among the plurality of network layers. In a training process of the AI model (110), an activation value gradient of the first network layer belonging to the AI model (110) and running on a computing node (120A, 120B) is collected; then the norm of the collected activation value gradient of the first network layer is compared with a preset threshold to determine whether the norm of the activation value gradient of the first network layer is abnormal; when it is determined that the norm of the activation value gradient of the first network layer is abnormal, it is inversely deduced that the activation value gradient of the first network layer is abnormal; and on the basis of the activation value gradient of the first network layer being abnormal, it can be inversely deduced that a training process of the first network layer is abnormal, so that it is inversely deduced that a silent data corruption has occurred in the computing node (120A, 120B) on which the first network layer is deployed, so as to determine that a silent data corruption has occurred in the training process of the AI model (110).

Description

Methods and related equipment for silent data error detection during AI model training

[0001] This application claims priority to Chinese Patent Application No. 202410963936.8, filed on July 17, 2024, with the Chinese National Intellectual Property Administration, entitled “Method and Related Device for Detecting Silent Data Errors in AI Model Training Process”, the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and in particular to a method and related equipment for detecting silent data corruption (SDC) during AI model training. Background Technology

[0003] With the development of AI technology, AI models (also known as neural networks) have been widely applied in fields such as computer vision (CV), natural language processing (NLP), graph learning, and recommender systems. Before being put into use, AI models typically require training. During the training process, silent data errors may occur. These errors refer to the corruption of AI model data during transmission, processing, or storage within the AI ​​model training system. However, the probability of such corruption is relatively random and it is difficult to detect.

[0004] Because silent data errors are rare and difficult to detect, if a silent data error occurs during the training process of an AI model, its impact will spread until the end of the AI ​​model training, resulting in very low accuracy or even non-convergence of the final AI model, causing great losses to users.

[0005] Therefore, there is an urgent need to provide a method to detect whether silent data errors occur during the training of AI models, so that when silent data errors are identified, timely measures can be taken to prevent the impact of silent data errors from continuing to spread during the training of AI models. Summary of the Invention

[0006] This application provides a method and related equipment for detecting silent data errors during the training process of an AI model. It is used to detect whether silent data errors occur during the training process of an AI model, so that when a silent data error is determined to have occurred in the AI ​​model, timely measures can be taken to prevent the impact of the silent data error from continuing to spread in the AI ​​model.

[0007] Firstly, a method for detecting silent data errors during AI model training is provided. The AI ​​model includes multiple network layers, and the first network layer is one of the multiple network layers or a part of a network layer. The first network layer is deployed on a first computing node. The method specifically includes the following steps: obtaining the activation value gradient of the first network layer during the Nth iteration of AI model training, and determining the norm of the activation value gradient of the first network layer; then, comparing the norm of the activation value gradient of the first network layer with a first threshold; if the norm of the activation value gradient of the first network layer is determined to be greater than the first threshold, a silent data error is determined to have occurred on the first computing node.

[0008] The above scheme, during AI model training, collects the activation gradients of the network structure (i.e., the first network layer, which is one of the multiple network layers or a part of a network layer in the AI ​​model) running on computing nodes. Then, it compares the norm of the collected activation gradients with a preset threshold (i.e., the first threshold) to determine if the norm of the network gradient is abnormal. If the norm of the activation gradient is abnormal, it can be inferred that the activation gradient itself is abnormal. Based on this abnormality, it can be further inferred that the training process of the network structure is abnormal, and ultimately, based on this abnormality, it can be inferred that the computing node where the network structure is deployed has a silent data error. Since the computing node participates in the AI ​​model training process, determining that the computing node has a silent data error is equivalent to determining that the AI ​​model training process has a silent data error. It can be seen that this scheme can detect silent data errors during AI model training simply by collecting the activation gradients of the network structure running on computing nodes, and the scheme is very simple and efficient.

[0009] In one possible implementation, the first network layer is the last network layer running on the first computing node during the backpropagation process of the Nth iteration.

[0010] Since the first computing node may not have experienced a silent data error after the Nth iteration of AI model training but before backpropagation to the first network layer, and the silent data error occurs during backpropagation to the first network layer, if the determination of whether the first computing node has experienced a silent data error is based on the activation value gradient of the non-last network layer running on the first computing node, it will mistakenly determine that the first computing node has not experienced a silent data error. Therefore, in the above implementation, determining whether the first computing node has experienced a silent data error based on the activation value gradient of the last network layer running on the first computing node can improve the accuracy of the determination result.

[0011] In one possible implementation, after determining that a silent data error has occurred in the first computing node, the above method further includes the following steps:

[0012] Implement either measure 1 or measure 2. Measure 1 is to replace the activation value gradient of the first network layer with 0 and issue a fault notification to the maintenance personnel. The replaced activation value gradient is used to be transmitted to the next network layer of the first network layer during backpropagation. Measure 2 is to suspend the training of the AI ​​model and issue a fault notification to the maintenance personnel.

[0013] In measure 1, since the activation gradient of the first network layer is used to update the weights of neurons in the first network layer, it can be understood that after replacing the activation gradient of the first network layer with 0, the weights of neurons in the first network layer will not be updated during the update phase of the first network layer in the Nth iteration, thereby avoiding the abnormal activation gradient of the first network layer caused by the silent data error of the first computing node, which would affect the first network layer.

[0014] The activation gradient of the first network layer, which is replaced with 0, is used to propagate to the next network layer during backpropagation. The next network layer then calculates its own activation gradient based on the activation gradient of the first network layer, which is replaced with 0. Since the calculated activation gradient of the next network layer is also 0, the weights of the neurons in the next network layer will not be updated during the update phase of the next network layer in the Nth iteration. This avoids the abnormal activation gradient of the first network layer caused by the silent data error of the first computing node, which would affect the next network layer.

[0015] Similarly, the activation gradient of the next network layer is 0. This activation gradient will be transmitted to the next network layer after the next network layer in the backpropagation process. Following the above process, the weights of the network layers after the first network layer will not be updated in the backpropagation process. This avoids the abnormal activation gradient of the first network layer caused by the silent data error in the first computing node, which affects the network layers after the first network layer in the backpropagation process. This effectively prevents the impact of silent data error from continuing to spread in the training process of the AI ​​model.

[0016] In measure 2, suspending the training of the AI ​​model can also prevent the effects of silent data errors occurring at the first computing node from continuing to spread during the training process of the AI ​​model.

[0017] The fault notifications in Measures 1 and 2 can inform maintenance personnel of silent data errors on the first computing node, enabling timely maintenance of the first computing node, such as repairing the silent data errors or replacing it with a new one. Specifically, if the detection system is deployed on the first computing node, the fault notification can be an alarm signal; if the detection system is not deployed on the first computing node, the fault notification can carry the identifier of the first computing node, allowing maintenance personnel to locate the first computing node based on this identifier.

[0018] In one possible implementation, the second network layer is one of multiple network layers or a portion of a network layer. The second network layer is deployed on the second computing node. The method further includes the following steps: obtaining the activation gradient of the second network layer during the Mth iteration of AI model training, and determining the norm of the activation gradient of the second network layer; comparing the norm of the activation gradient of the second network layer with a first threshold and a second threshold; if the norm of the activation gradient of the second network layer is less than or equal to the first threshold and greater than the second threshold, determining that the second computing node is suspected of having a silent data error, and recording the suspected silent data error of the second computing node; if the number of times the suspected silent data error of the second computing node is recorded reaches a threshold, determining that the second computing node has a silent data error.

[0019] By implementing the above method, by comparing the norm of the activation value gradient of the network structure running on the computing node, the first threshold, and the second threshold, it is possible to detect computing nodes suspected of having silent data errors during the training process of the AI ​​model.

[0020] In one possible implementation, the norm of the activation gradient of the first network layer includes any one or more of the following: the maximum value among the absolute values ​​of the activation gradient of the first network layer; the standard deviation of the absolute values ​​of the activation gradient of the first network layer; and the average value of the absolute values ​​of the activation gradient of the first network layer.

[0021] In one possible implementation, the AI ​​model is a large language model.

[0022] In one possible implementation, the silent data error is any of the following: the storage cell storing the first network layer in the first computing node is damaged; the computing cell running the first network layer in the first computing node is subjected to voltage disturbance or high-energy particle impact; the circuit of the computing cell running the first network layer in the first computing node is aged or the transistor is etched.

[0023] Secondly, a detection system is provided, the system comprising:

[0024] The acquisition unit is used to acquire the activation gradient of the first network layer during the Nth iteration of AI model training. The AI ​​model includes multiple network layers, and the first network layer is one of the multiple network layers or a part of a network layer. The first network layer is deployed on the first computing node.

[0025] A detection unit is used to determine the norm of the activation value gradient of the first network layer;

[0026] The detection unit is also used to determine that a silent data error has occurred in the first computing node if the norm of the activation value gradient of the first network layer is greater than a first threshold.

[0027] In one possible implementation, the first network layer is the last network layer running on the first computing node during the backpropagation process of the Nth iteration.

[0028] In one possible implementation, the detection system further includes a processing unit, configured to: replace the activation value gradient of the first network layer with 0 and issue a fault notification to maintenance personnel, wherein the replaced activation value gradient is used to be transmitted to the next network layer of the first network layer during backpropagation; or, suspend training of the AI ​​model and issue a fault notification to maintenance personnel.

[0029] In one possible implementation, the second network layer is one of multiple network layers or a portion of a network layer, and the second network layer is deployed on the second computing node; the acquisition unit is further configured to acquire the activation value gradient of the second network layer during the Mth iteration of AI model training; the detection unit is further configured to determine the norm of the activation value gradient of the second network layer; the detection unit is further configured to determine that the second computing node is suspected of having a silent data error when the norm of the activation value gradient of the second network layer is less than or equal to a first threshold and greater than a second threshold, and record the suspected silent data error of the second computing node; the detection unit is further configured to determine that the second computing node has a silent data error when the number of times the suspected silent data error of the second computing node is recorded reaches a threshold.

[0030] In one possible implementation, the norm of the activation gradient of the first network layer includes any one or more of the following: the maximum value among the absolute values ​​of the activation gradient of the first network layer; the standard deviation of the absolute values ​​of the activation gradient of the first network layer; and the average value of the absolute values ​​of the activation gradient of the first network layer.

[0031] In one possible implementation, the AI ​​model is a large language model.

[0032] In one possible implementation, the silent data error is any of the following: the storage cell storing the first network layer in the first computing node is damaged; the computing cell running the first network layer in the first computing node is subjected to voltage disturbance or high-energy particle impact; the circuit of the computing cell running the first network layer in the first computing node is aged or the transistor is etched.

[0033] Thirdly, a computing device cluster is provided, the computing device cluster including at least one computing device, each of the at least one computing device including a processor and a memory, the processor of the at least one computing device being configured to execute instructions stored in the memory of the at least one computing device to cause the computing device cluster to implement the method as described in the first aspect.

[0034] Fourthly, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and the instructions are executed by a computing device or a cluster of computing devices to implement the method described in the first aspect.

[0035] Fifthly, a computer program product including instructions is provided, the computer program product including instructions that enable software or program products to run on a computing device or stored in any available medium, and when the computer program product runs on a computing device or a cluster of computing devices, cause the computing device or cluster of computing devices to implement the method described in the first aspect. Attached Figure Description

[0036] Figure 1 is a schematic diagram of the structure of an AI model training system according to this application;

[0037] Figure 2 is a schematic diagram of a Transformer model provided in this application;

[0038] Figure 3A is a schematic diagram of the deployment of an AI model in an AI model training system provided in this application;

[0039] Figure 3B is a schematic diagram of the deployment of another AI model provided in this application in an AI model training system;

[0040] Figure 4A is a schematic diagram of the architecture of a detection system provided in this application;

[0041] Figure 4B is a schematic diagram of the architecture of another detection system provided in this application;

[0042] Figure 4C is an example diagram of a detection system provided in this application deployed in a cloud environment;

[0043] Figure 5 is a flowchart illustrating a silent data error detection method during AI model training provided in this application.

[0044] Figure 6 is a flowchart illustrating another method for silent data error detection during AI model training provided in this application;

[0045] Figure 7A is an example diagram of the configuration interface in a silent data error detection method during AI model training provided in this application;

[0046] Figure 7B is a diagram of a fault notification display interface provided in this application;

[0047] Figure 8 is a schematic diagram of the structure of a detection system provided in this application;

[0048] Figure 9 is a schematic diagram of the structure of a computing device provided in this application;

[0049] Figure 10 is a schematic diagram of the structure of a computing device cluster provided in this application;

[0050] Figure 11 is a schematic diagram of another computing device cluster provided in this application. Detailed Implementation

[0051] The application scenarios involved in this application are described below.

[0052] This application relates to a scenario where silent data error detection is required during AI model training. The following section provides a detailed description of this scenario using the AI ​​model training system shown in Figure 1.

[0053] As shown in Figure 1, the AI ​​model training system 100 includes an AI model 110 and multiple computing nodes. In Figure 1, the multiple computing nodes include four computing nodes: computing node 120A, computing node 120B, computing node 120C, and computing node 120D, as an example.

[0054] AI model 110 can be a model comprising one or more network layers. In Figure 1, an example is taken where AI model 110 includes six network layers, from layer 1 to layer 6. AI model 110 can be a model used to implement any AI business, such as models used to implement face recognition, fingerprint recognition, object recognition, natural language translation, recommendation, etc. Examples include convolutional neural networks (CNN), recurrent neural networks (RNN), large language models, etc. This application does not specifically limit AI model 110. This application mainly relates to silent data error detection during the training process of large language models. The large language model is described below.

[0055] Large language models, also known as large-scale pre-trained language models, generally refer to language model training tasks designed using large-scale corpora (such as sentences, paragraphs, and other language training materials). A large-scale neural network algorithm structure is then trained to learn and implement these tasks, resulting in a large-scale neural network algorithm structure. Subsequent tasks can then be performed on this model for feature extraction or fine-tuning to achieve specific objectives. Through pre-training on large-scale corpora, large language models can learn powerful language representation capabilities, extracting rich syntactic and semantic information from text.

[0056] Large language models may include, but are not limited to, translator (Transformer) models, bidirectional encoder representations from transformers (BERT), sequence to sequence (Seq2Seq) models, generative pre-trained transformer (GPT) based on the Transformer architecture, end-to-end text generation models (text-to-text transfer transformer, T5), generalized autoregressive pretraining for language understanding (XLNet), and so on.

[0057] Most of the large language models mentioned above are based on the Transformer model structure. The following is a brief introduction to the Transformer model with reference to Figure 2. As shown in Figure 2, the Transformer model can be constructed by stacking multiple Transformer layers. A Transformer layer can include a multi-head attention layer and a feedforward neural network layer. The multi-head attention layer obtains corresponding weight values ​​by calculating the correlation between words, ultimately obtaining context-related word representations, and is the core part of the Transformer structure. The feedforward neural network layer can further transform the obtained word representations to obtain the final output of the Transformer layer. In addition to these two important modules, the Transformer layer can also include a residual layer and a linear normalization layer. The residual layer's role is to solve the gradient vanishing and gradient exploding problems during model training, thereby helping to improve the model's training effect and convergence speed. The linear normalization layer's role is to normalize the input or intermediate layer data, sometimes also called "standardization".

[0058] Computing nodes 120A, 120B, 120C, and 120D are used together to deploy the AI ​​model 110. The deployment method can be pipeline parallelism (PP), a combination of pipeline parallelism and tensor parallelism (TP), or tensor parallelism, etc., and this application does not specify a particular method. Pipeline parallelism is a strategy that distributes different network layers of the AI ​​model to different processing units (such as different computing nodes or different computing units) to perform parallel computation. Each processing unit is responsible for training a portion of the network layers of the entire AI model and then passing the intermediate results to the next processing unit to achieve parallel computation of the AI ​​model. Tensor parallelism is a strategy that decomposes large tensor operations (such as large matrix multiplications or deep convolution operations) in the AI ​​model and distributes them to multiple processing units for parallel computation. The computing units can be graphics processing units (GPUs), tensor processing units (TPUs), neural network processing units (NPUs), etc.

[0059] Taking the pipeline parallel approach as an example, the AI ​​model 110 shown in Figure 1, which contains network layers 1 to 6, can be divided into four parts according to the pipeline parallel approach: network layer 1, network layer 2 and network layer 3, network layer 4 and network layer 5, and network layer 6, as shown in Figure 3A. The first part, "network layer 1", is deployed on computing node 120A, the second part, "network layer 2 and network layer 3", is deployed on computing node 120B, the third part, "network layer 4 and network layer 5", is deployed on computing node 120C, and the fourth part, "network layer 6", is deployed on computing node 120D.

[0060] Taking the combination of pipeline parallelism and tensor parallelism as an example, we can first divide the network layers 1 to 6 of the AI ​​model 110 shown in Figure 1 into two parts according to the pipeline parallelism method: network layer 1 and network layer 2, and network layers 3 to network layer 6. Then, according to the tensor parallelism method, the first part "network layer 1 and network layer 2" is divided into two sub-parts: sub-network 11 + sub-network 21 and sub-network 12 + sub-network 22, and the second part "network layers 3 to network layer 6" is divided into two sub-parts: sub-network 31 + sub-network 41 + sub-network 51 + sub-network 61 and sub-network 32 + sub-network 42 + sub-network 52 + sub-network 62. Among them, sub-network 11 is a part of network layer 1, sub-network 12 is the remaining part of network layer 1 excluding sub-network 11, sub-network 21 is a part of network layer 2, sub-network 22 is the remaining part of network layer 2 excluding sub-network 21, and sub-network 31 is a part of network layer 2. Subnetwork 1 is a part of network layer 3, subnetwork 32 is the remaining part of network layer 3 excluding subnetwork 31, subnetwork 41 is a part of network layer 4, subnetwork 42 is the remaining part of network layer 4 excluding subnetwork 41, subnetwork 51 is a part of network layer 5, subnetwork 52 is the remaining part of network layer 5 excluding subnetwork 51, subnetwork 61 is a part of network layer 6, subnetwork 62 is the remaining part of network layer 6 excluding subnetwork 61, as shown in Figure 3B. Subnetwork 11 + subnetwork 21 is deployed on compute node 120A, subnetwork 12 + subnetwork 22 is deployed on compute node 120B, subnetwork 31 + subnetwork 41 + subnetwork 51 + subnetwork 61 is deployed on compute node 120C, and subnetwork 32 + subnetwork 42 + subnetwork 52 + subnetwork 62 is deployed on compute node 120D.

[0061] It should be noted that Figures 3A and 3B are merely examples of two parallel processing methods and should not be considered as specific limitations. For instance, in Figure 3A above, the example is dividing network layers 1 to 6 into four parts: network layer 1, network layer 2 and network layer 3, network layer 4, network layer 5, and network layer 6. In actual scenarios, it can also be divided into four parts: network layer 1 and network layer 2, network layer 3 and network layer 4, network layer 5, and network layer 6, or it can be divided in other ways.

[0062] As can be seen from Figure 3A, when the AI ​​model 110 is deployed on multiple computing nodes in a pipeline parallel manner, one or more complete network layers contained in the AI ​​model 110 are deployed on each computing node. When the AI ​​model 110 is deployed on multiple computing nodes in a combination of pipeline parallel and tensor parallel, one or more network layers contained in the AI ​​model 110 are deployed on each computing node.

[0063] In a specific implementation, computing nodes 120A, 120B, 120C, and 120D can be general-purpose physical servers, such as ARM servers or x86 servers, or other computing devices capable of deploying and training AI model 110, such as virtual machines or containers. This application does not specifically limit the computing nodes. Each computing node may include one or more computing units (not shown in Figure 1). The computing units may be GPUs, TPUs, NPUs, etc. Some or all of the computing units in each computing node can be used to run the network structure of the AI ​​model 110 deployed on each computing node.

[0064] In real-world scenarios, during the training of AI model 110 in the AI ​​model training system 100 shown in Figure 1, silent data corruption (SDC) may occur on the computing nodes where AI model 110 is deployed. Silent data corruption, also known as silent fault, refers to data corruption occurring during transmission, processing, or storage in a computer system (such as the AI ​​model training system 100 shown in Figure 1). This corruption occurs randomly and is difficult to detect, often leading to incorrect data processing results when using corrupted data (e.g., resulting in a low-accuracy or even non-convergent AI model 110). For example, if the storage unit in a computer system used to store data (such as the parameters of AI model 110) is damaged (e.g., the memory storing data experiences bit flipping due to electromagnetic interference or voltage disturbances, or the disk storing data ages or its magnetic properties deteriorates), the data stored in the storage unit may become corrupted. Consequently, the computer system may use the corrupted data stored in the storage unit to perform calculations and obtain incorrect results. Alternatively, during data calculation (e.g., during the training of AI model 110), the computing unit in the computer system may experience data corruption due to voltage disturbances or high-energy particle impacts, or due to circuit aging or transistor etching. This could lead to incorrect calculation results when using the corrupted data. Or, the data to be processed may be corrupted due to electromagnetic interference or other external interference during transmission, causing the computing unit to use the corrupted data to perform calculations and obtain incorrect results.

[0065] Because silent data errors have a low probability of occurrence and are not easily detected, if a silent data error occurs on a computing node where AI model 110 is deployed during the training process, the error will affect the training process of the network structure of AI model 110 deployed on that computing node, causing errors in the training results of the network structure on the computing node. Moreover, since the training process of AI model 110 involves multiple iterations, the impact of this error will spread with the increase of the number of iterations until the training of AI model 110 ends, resulting in very low accuracy or even non-convergence of the final AI model 110, causing great losses to users.

[0066] To address the aforementioned issues, this application provides a method and system for detecting silent data errors during AI model training. The method involves collecting the activation gradients of the network structure (one or a portion of a network layer within the AI ​​model's multiple network layers) of the AI ​​model running on a computing node during training. The norm of the collected activation gradients (e.g., the maximum absolute value of the activation gradient, the standard deviation of the absolute values ​​of the activation gradients, etc.) is then determined. This norm is compared with a preset threshold. If the norm exceeds the preset threshold, a silent data error is identified in the computing node. Since the computing node participates in the AI ​​model training process, identifying a silent data error in the computing node signifies a silent data error occurring during the AI ​​model's training process. The principle behind determining a silent data error in a computing node when the norm of the activation value gradient exceeds a preset threshold is as follows: if the norm of the activation value gradient exceeds the preset threshold, then the norm of the activation value gradient is abnormal. Based on the norm abnormality of the activation value gradient, we can infer the activation value gradient abnormality. Based on the activation value gradient abnormality, we can infer the training process abnormality of the network structure. Since the training process abnormality of the network structure is usually caused by silent data errors in the computing nodes deployed in the network structure, we can infer the silent data error in the computing nodes based on the training process abnormality of the network structure.

[0067] As can be seen, the solution provided in this application can achieve silent data error detection during the training process of AI models simply by collecting the activation value gradient of the network structure running on the computing node. The solution is very simple and therefore has high detection efficiency.

[0068] Furthermore, the solution provided in this application can take timely measures after detecting silent data errors during the training process of an AI model. For example, it can replace the activation value gradient of the network structure running on the computing node with the silent data error with 0 and issue a fault notification to the maintenance personnel, or suspend the training of the AI ​​model and issue a fault notification to the maintenance personnel. This allows the maintenance personnel to maintain the computing node with the silent data error in a timely manner and prevent the impact of the silent data error from continuing to spread during the training process of the AI ​​model, which can effectively reduce user losses.

[0069] The following sections will detail the detection system provided in the application and the method for detecting silent data errors during the AI ​​model training process, with reference to the corresponding attached diagrams.

[0070] Please refer to Figure 4A, which is a schematic diagram of the architecture of a detection system provided in this application. As shown in Figure 4A, the architecture includes an AI model training system 100 and a detection system 200, with the detection system 200 integrated into the AI ​​model training system 100. For details regarding the AI ​​model training system 100, please refer to the relevant content of the embodiments in Figures 1, 3A, and 3B, which will not be repeated here.

[0071] The detection system 200 is used to detect silent data errors during the training process of the AI ​​model 110.

[0072] In specific implementation, the detection system 200 can be a separate software or application running on a computing node (such as computing node 120A or computing node 120B) or each computing node in the AI ​​model training system 100, specifically designed to implement the silent data error detection function during the training process of the AI ​​model 110. Examples include silent data error detection tools and applications. Alternatively, it can be a silent data error detection function module or plugin within comprehensive software (such as fault detection software) running on a computing node or each computing node. This application does not impose specific limitations. In Figure 4A, the detection system 200 is taken as running on computing node 120E as an example.

[0073] Please refer to Figure 4B, which is a schematic diagram of the architecture of another detection system provided in this application. As shown in Figure 4B, the architecture includes an AI model training system 100 and a detection system 200, with the detection system 200 being independent of the AI ​​model training system 100.

[0074] For details regarding the AI ​​model training system 100, please refer to the embodiments shown in Figures 1, 3A, and 3B. These details will not be repeated here.

[0075] The detection system 200 can be invoked by the AI ​​model training system 100 to perform silent data error detection during the training process of the AI ​​model 110. The detection system 200 can establish a communication connection with the AI ​​model training system 100 via a network. This communication connection can be a wired connection or a wireless connection. The network can be the public Internet, an internal local area network (LAN), a virtual private network (VPN), a dedicated line such as fiber optic lines, copper wires, satellite connections, etc., or a wireless network such as wireless LAN (Wi-Fi), cellular network, etc. This application does not make specific limitations.

[0076] In specific implementation, the detection system 200 can be deployed on computing devices, or on a cluster of computing devices composed of multiple computing devices. The computing devices can be bare metal servers (BMS), virtual machines, or containers. The computing device cluster can be a cloud data center, an enterprise private cluster, or a hybrid cloud environment, i.e., a deployment mode that uses both public and private clouds simultaneously; this application does not impose specific limitations.

[0077] When actually deploying the detection system 200 shown in Figure 4B, it can be flexibly deployed according to specific application scenarios and business needs. The following is an example of the actual deployment method of the detection system 200 in a specific application scenario.

[0078] In one application scenario, the detection system 200 can be deployed on the user's own server or server cluster. Before the AI ​​model training system 100 starts training the AI ​​model 110, the user can configure the AI ​​model training system 100 to connect to the detection system 200 so that during the training of the AI ​​model 110, the detection system 200 can detect whether silent data errors occur during the training of the AI ​​model 110.

[0079] In another application scenario, the detection system 200 is provided by a cloud service provider, as shown in Figure 4C. Users can purchase relevant cloud services for the detection system 200 from the cloud service provider. The detection system 200 is then deployed on an instance within the cloud data center of the cloud service provider. After purchasing the relevant cloud services for the detection system 200, the user has access to the cloud services of the detection system 200. Before the AI ​​model training system 100 starts training the AI ​​model 110, the user can configure the AI ​​model training system 100 to connect to the detection system 200. This allows the user to use the relevant cloud services of the detection system 200 to detect whether silent data errors occur during the training of the AI ​​model during the training process of the AI ​​model 110.

[0080] It should be understood that the above application scenarios are for illustrative purposes only, and the detection system 200 can be flexibly deployed according to actual business needs.

[0081] In one possible embodiment, Figure 4B may further include a client 300 (not shown in Figure 4B). The client 300 can be deployed in the AI ​​model training system 100 to enable human-computer interaction. Users can use the client 300 to invoke the detection system 200 to perform silent data error detection during the training process of the AI ​​model 110. In specific implementation, the client 300 can be a separate software or application running on a computing node or each computing node in the AI ​​model training system 100, specifically designed to implement the silent data error detection function during the training process of the AI ​​model 110, such as a silent data error detection tool or application. Alternatively, it can be a silent data error detection function module or plugin within comprehensive software (such as fault detection software) running on a computing node or each computing node. This application does not impose specific limitations on this.

[0082] Optionally, client 300 can also be a cloud platform client, such as a cloud platform console. Specifically, it can be a web-based console or an application programming interface (API) based console; this application does not impose specific limitations. This console can provide users with a silent data error detection cloud service, and users can obtain access to the detection system 200 provided in this application by purchasing cloud services. Alternatively, the silent data error detection cloud service provided in this application can be a sub-service within the comprehensive cloud service provided by this console, available for purchase and use by users; this application does not impose specific limitations.

[0083] To facilitate a clearer understanding of the detection system provided in this application, and the specific process of silent data error detection during AI model training, a more detailed description is provided below with reference to the flowchart of a silent data error detection method during AI model training provided in this application, as shown in Figure 5.

[0084] As shown in Figure 5, the following steps may be included:

[0085] S501: The detection system obtains the activation gradient of the first network layer during the Nth iteration of AI model training. The first network layer is one of the multiple network layers in the AI ​​model or a part of a network layer. The first network layer is deployed on the first computing node.

[0086] First, let's go through the training process of the AI ​​model in detail.

[0087] The training process for AI models typically involves multiple iterations, each of which includes the following stages:

[0088] Forward propagation phase: The training data is input into the input layer of the AI ​​model, and forward propagation is performed layer by layer to calculate the predicted output of the output layer of the AI ​​model.

[0089] Loss calculation stage: Calculate the loss value between the predicted output and the true label.

[0090] Backpropagation Phase: Based on the loss value between the predicted output and the true label, the activation gradient of the AI ​​model's output layer is calculated, and the activation gradient is propagated layer by layer from the output layer to the input layer using the chain rule. Layer-by-layer propagation of the activation gradient means that after calculating the activation gradient of the current network layer, the activation gradient to be backpropagated to the next network layer is calculated based on the activation gradient of the current network layer, and then propagated to the next network layer. The activation gradient of a network layer includes the activation gradients of the neurons in the network layer; it is a collection of the activation gradients of the neurons in the network layer. For example, the activation gradient of the output layer includes the activation gradients of the neurons in the output layer.

[0091] Model update phase: The weight gradient of each network layer is calculated based on the activation value gradient of each network layer. Then, the weight gradient is sent to the optimizer of the network layer. The optimizer calculates the update amount required for the weight update of the network layer. After that, the weight of the network layer is updated according to the update amount to minimize the error of the entire AI model.

[0092] Taking the Transformer model shown in Figure 2 as an example, as shown in Figure 2, during the training process of the Transformer model, the direction from the input layer to the output layer is the forward propagation direction, and the direction from the output layer to the input layer is the backward propagation direction.

[0093] Taking Figure 3A as an example, assuming that the AI ​​model 110 deployed in the AI ​​model training system 100 shown in Figure 3A contains network layer 1 as the input layer and network layer 6 as the output layer, then as shown in Figure 3A, the direction from network layer 1 to network layer 6 is the forward propagation direction, and the direction from network layer 6 to network layer 1 is the backward propagation direction.

[0094] Taking Figure 3B as an example, assuming that the AI ​​model 110 deployed in the AI ​​model training system 100 shown in Figure 3B contains network layer 1 (including subnetwork 11 and subnetwork 12) as the input layer and network layer 6 (including subnetwork 61 and subnetwork 62) as the output layer, then as shown in Figure 3B, the direction from network layer 1 to network layer 6 is the forward propagation direction, and the direction from network layer 6 to network layer 1 is the backward propagation direction.

[0095] The Nth iteration can be any iteration in the multiple iterations of AI model training, and this application does not impose specific limitations.

[0096] The first network layer is either a single network layer or a portion of a network layer, depending on the network structure of the AI ​​model deployed on the first computing node. The first network layer will be explained in detail below, taking into account different network structures deployed on the first computing node.

[0097] Case (1): If the network structure deployed on the first computing node is a complete network layer, then the first network layer is the network layer deployed on the first computing node.

[0098] Taking computing node 120A as shown in Figure 3A as the first computing node, a complete network layer 1 is deployed on computing node 120A, then the first network layer is network layer 1.

[0099] Case (2): If the network structure deployed on the first computing node consists of at least two complete network layers, then the first network layer can be any one of the network layers deployed on the first computing node.

[0100] Taking computing node 120B as shown in Figure 3A as the first computing node, if a complete network layer 2 and network layer 3 are deployed on computing node 120B, then the first network layer can be either network layer 2 or network layer 3.

[0101] Case (3): If the network structure deployed on the first computing node is part of a network layer, then the first network layer can be part of the network layer deployed on the first computing node.

[0102] For example, if only a portion of the input layer of the AI ​​model is deployed on the first computing node, then the first network layer can be a portion of the input layer deployed on the first computing node.

[0103] Case (4): If the network structure deployed on the first computing node contains part of at least two network layers, then the first network layer can be part of any one of the at least two network layers deployed on the first computing node.

[0104] Taking computing node 120A as shown in Figure 3B as the first computing node, a part of network layer 1, "sub-network 11", and a part of network layer 2, "sub-network 21", are deployed on computing node 120A. Then the first network layer can be sub-network 11 or sub-network 21.

[0105] The activation gradient of the first network layer in the Nth iteration process, as can be seen from the above introduction to the training process of the AI ​​model, includes the activation gradient of neurons in the first network layer in the Nth iteration process, and is the set of activation gradients of neurons in the first network layer.

[0106] The following section details how the detection system obtains the activation gradient of the first network layer during the Nth iteration.

[0107] Method (1): If the detection system is integrated into the AI ​​model training system as shown in Figure 4A, the detection system can monitor the Nth iteration of the AI ​​model training process. During the monitoring of the Nth iteration, if the first computing node obtains the activation value gradient of the first network layer, the activation value gradient of the first network layer is directly collected.

[0108] Method (2): If the detection system is deployed independently of the AI ​​model training system as shown in Figure 4B, the first computing node in the AI ​​model training system can upload the activation gradient of the first network layer to the detection system after obtaining the activation gradient of the first network layer during the Nth iteration of AI model training.

[0109] It should be understood that the above methods (1) and (2) are merely examples of how the detection system obtains the activation value gradient of the first network layer, and should not be regarded as specific limitations.

[0110] S502: The detection system determines the norm of the activation gradient of the first network layer.

[0111] The norm of the activation gradient of the first network layer can be any one of the following three norms:

[0112] The first norm is the maximum absolute value of the gradient of the activation value of the first network layer.

[0113] The second norm is the standard deviation of the absolute value of the gradient of the activation values ​​of the first network layer.

[0114] The third norm is the average of the absolute values ​​of the gradients of the activation values ​​of the first network layer.

[0115] It should be noted that in real-world scenarios, the norm of the activation gradient of the first network layer can also be other, such as a combination of the above norms, the median of the absolute value of the activation gradient of the first network layer, the variance, etc. This application does not impose any specific limitations.

[0116] S503: The detection system compares the norm of the activation gradient of the first network layer with the first threshold.

[0117] The first threshold is used to determine whether the norm of the activation value gradient of the first network layer is abnormal. Specifically, if it is determined that the norm of the activation value gradient of the first network layer is greater than the first threshold, it can be determined that the norm of the activation value gradient of the first network layer is abnormal.

[0118] It is understandable that the first threshold will differ depending on the type of norm of the activation value gradient in the first network layer. In specific implementations, the first threshold can be obtained and set based on the actual scenario through expert experience or experimental statistics, and this application does not impose specific limitations. The first threshold is usually set relatively large. Taking the first norm of the activation value gradient in the first network layer as the first norm in S502 as an example, the first threshold can be set to 1,000,000, 990,000, etc.

[0119] S504: If the detection system determines that the norm of the activation value gradient of the first network layer is greater than the first threshold, then it determines that a silent data error has occurred in the first computing node.

[0120] If the detection system determines that the norm of the activation value gradient of the first network layer is greater than a first threshold, then it can be determined that the norm of the activation value gradient of the first network layer is abnormal. Based on the abnormal norm of the activation value gradient of the first network layer, it is possible to infer that the activation value gradient of the first network layer is abnormal. Furthermore, based on the abnormal activation value gradient of the first network layer, it is possible to infer that the training process of the first network layer is abnormal. Since the first computing node participates in the training process of the AI ​​model, determining that the first computing node has a silent data error means determining that the training process of the AI ​​model has a silent data error.

[0121] In a specific embodiment of this application, after the detection system determines that a silent data error has occurred in the first computing node, it can perform corresponding measures, such as the following measures 1 and 2, to prevent the impact of the silent data error from continuing to spread in the AI ​​model.

[0122] Measure 1: Replace the activation value gradient of the first network layer with 0 and send a fault notification to the maintenance personnel. The activation value gradient of the first network layer that is replaced with 0 is used to be transmitted to the next network layer of the first network layer during the backpropagation process.

[0123] The next network layer after the first network layer is a network layer or a part of a network layer. Taking network layer 1 on computing node 120A as shown in Figure 3A as an example, the next network layer after the first network layer is network layer 2 on computing node 120B. Taking network layer 2 on computing node 120B as an example, the next network layer after the first network layer is network layer 3 on computing node 120B. Taking sub-network 11 on computing node 120A as an example, the next network layer after the first network layer is sub-network 21 on computing node 120A. Taking sub-network 21 on computing node 120A as an example, the next network layer after the first network layer is sub-network 31 on computing node 120C.

[0124] Since the activation gradient of the first network layer is used to update the weights of neurons in the first network layer, it can be understood that after replacing the activation gradient of the first network layer with 0, the weights of neurons in the first network layer will not be updated during the update phase of the first network layer in the Nth iteration. This avoids the abnormal activation gradient of the first network layer caused by the silent data error of the first computing node, which would affect the first network layer.

[0125] The activation gradient of the first network layer, which is replaced with 0, is used to propagate to the next network layer during backpropagation. The next network layer then calculates its own activation gradient based on the activation gradient of the first network layer, which is replaced with 0. Since the calculated activation gradient of the next network layer is also 0, the weights of the neurons in the next network layer will not be updated during the update phase of the next network layer in the Nth iteration. This avoids the abnormal activation gradient of the first network layer caused by the silent data error of the first computing node, which would affect the next network layer.

[0126] Similarly, the activation gradient of the next network layer is 0. This activation gradient will be transmitted to the next network layer after the next network layer in the backpropagation process. Following the above process, the weights of the network layers after the first network layer will not be updated in the backpropagation process. This avoids the abnormal activation gradient of the first network layer caused by the silent data error in the first computing node, which affects the network layers after the first network layer in the backpropagation process. This effectively prevents the impact of silent data error from continuing to spread in the training process of the AI ​​model.

[0127] The fault notification issued by the detection system can inform maintenance personnel of silent data errors on the primary computing node, enabling timely maintenance of the primary computing node, such as repairing the silent data errors or replacing it with a new one. Specifically, if the detection system is deployed on the primary computing node, the fault notification can be an alarm signal; if the detection system is not deployed on the primary computing node, the fault notification can carry the identifier of the primary computing node, allowing maintenance personnel to locate the primary computing node based on this identifier.

[0128] Measure 2: Suspend training of the AI ​​model and send a fault notification to maintenance personnel.

[0129] Pausing AI model training can also prevent the effects of silent data errors occurring at the first computing node from continuing to spread during the AI ​​model training process.

[0130] For information on fault notifications, please refer to the relevant description of fault notifications in Measure 1.

[0131] Compared to Measure 2, Measure 1 addresses the complexity of AI models. If the AI ​​model is in its Nth iteration, Measure 2, which pauses training, can cause various problems, such as difficulty in preserving the model's state at the pause point and difficulty in resuming training. Measure 1 replaces the activation gradient of the first network layer with 0. This 0-value gradient is then transmitted to the next network layer in the backpropagation process. The next network layer calculates its own activation gradient based on the 0-value gradient and propagates it to the next layer, and so on, until the backpropagation process ends and the Nth iteration is complete. Maintenance personnel can then pause training to maintain the first computing node after the Nth iteration, as the training pauses at the end of the Nth iteration, facilitating the resumption of training.

[0132] In a specific embodiment of this application, the detection system may also execute steps S601-S605 shown in FIG6.

[0133] S601: The detection system obtains the activation gradient of the second network layer during the Mth iteration of AI model training. The second network layer is one of the multiple network layers in the AI ​​model or a part of a network layer. The second network layer is deployed on the second computing node.

[0134] M can be the same as N, or it can be different.

[0135] The second computing node can be the same as or different from the first computing node.

[0136] The second network layer can be the same as or different from the first network layer. For details on the second network layer, please refer to the introduction of the first network layer in S501. For details on the activation gradient of the second network layer in the Mth iteration, please refer to the introduction of the activation gradient of the first network layer in the Nth iteration in S501. These details will not be elaborated here.

[0137] The specific implementation process of S601 is similar to that of S501, and you can refer to the relevant description of S501.

[0138] S602: The detection system determines the norm of the activation gradient of the second network layer.

[0139] The norm of the activation gradient of the second network layer can be found in the introduction of the norm of the activation gradient of the first network layer in S502.

[0140] S603: The detection system compares the norm of the activation gradient of the second network layer with the first threshold and the second threshold.

[0141] The first threshold and the second threshold are used to determine whether the norm of the activation value gradient of the second network layer is abnormal. Specifically, if the norm of the activation value gradient of the second network layer is greater than the first threshold, the norm of the activation value gradient of the second network layer is determined to be abnormal. If the norm of the activation value gradient of the second network layer is less than or equal to the first threshold and greater than the second threshold, the norm of the activation value gradient of the second network layer is suspected to be abnormal. If the norm of the activation value gradient of the second network layer is less than or equal to the second threshold, the norm of the activation value gradient of the second network layer is determined to be normal. For details on the first threshold, please refer to the description of the first threshold in S503.

[0142] In practice, the second threshold can be set based on the actual scenario through expert experience or experimental statistics, and this application does not impose specific limitations. The second threshold is usually set relatively large, but smaller than the first threshold, such as setting the first threshold to 1,000,000 and the second threshold to 10,000, or setting the first threshold to 900,000 and the second threshold to 10,000, etc.

[0143] S604: If the detection system determines that the norm of the activation value gradient of the second network layer is less than or equal to the first threshold and greater than the second threshold, then it determines that the second computing node is suspected of having a silent data error.

[0144] If the detection system determines that the norm of the activation value gradient of the second network layer is less than or equal to the first threshold and greater than the second threshold, then the norm of the activation value gradient of the second network layer is suspected to be abnormal. Based on the suspected abnormality of the norm of the activation value gradient of the second network layer, it can be deduced that the activation value gradient of the second network layer is suspected to be abnormal. Based on the suspected abnormality of the activation value gradient of the second network layer, it can be deduced that the training process of the second network layer is suspected to be abnormal. Finally, based on the suspected abnormality of the training process of the second network layer, it can be deduced that the second computing node is suspected of having a silent data error.

[0145] S605: If the detection system records a certain number of times the second computing node is suspected of having a silent data error during multiple iterations of AI model training, then the system determines that the second computing node has a silent data error.

[0146] The number of times threshold can be 2 or 3 times, etc., and can be customized according to the actual scenario. This application does not impose specific limitations.

[0147] After the detection system determines that a silent data error has occurred in the second computing node, it takes corresponding measures, such as replacing the activation value gradient of the second network layer with 0 and issuing a fault notification to the maintenance personnel, or pausing the training of the AI ​​model and issuing a fault notification to the maintenance personnel, so that the maintenance personnel can perform timely maintenance on the second computing node with the silent data error and prevent the impact of the silent data error from continuing to spread in the AI ​​model.

[0148] In one possible embodiment, if the detection system determines that the norm of the activation value gradient of the second network layer is less than or equal to a second threshold, it determines that no silent data error has occurred in the second computing node.

[0149] It should be noted that the embodiments in Figures 5 and 6 are described using the example of the detection system detecting whether a silent data error has occurred on the first computing node, where the first network layer is any network layer or a part of any network layer deployed on the first computing node. In a preferred embodiment, the first network layer is the last network layer or a part of the last network layer running on the first computing node during the backpropagation process of the Nth iteration. Taking computing node 120B as shown in Figure 3A as an example, computing node 120B has a complete network layer 2 and network layer 3 deployed on it, and network layer 3 is the last network layer on computing node 120B. Therefore, the first network layer is network layer 3. Taking computing node 120B as shown in Figure 3B as another example, computing node 120B has a part of network layer 1 (i.e., sub-network 12) and a part of network layer 2 (i.e., sub-network 22) deployed on it. Sub-network 22 is a part of the last network layer (i.e., network layer 2) on computing node 120B. Therefore, the first network layer is sub-network 22.

[0150] In a preferred embodiment, the first network layer is the last network layer or a portion of the last network layer running on the first computing node during the backpropagation process of the Nth iteration because the first computing node may not have experienced a silent data error after the start of the Nth iteration of AI model training but before backpropagating to the first network layer. However, a silent data error may occur during backpropagation to the first network layer. In this case, if the detection system judges whether the first computing node has experienced a silent data error based on the activation value gradient of a non-last network layer or a portion of a non-last network layer running on the first computing node, it may mistakenly judge that the first computing node has not experienced a silent data error. Therefore, in the preferred embodiment, the detection system judges whether the first computing node has experienced a silent data error based on the activation value gradient of the last network layer or a portion of the last network layer running on the first computing node, which can improve the accuracy of the judgment result.

[0151] As can be seen, in the silent data error detection method during AI model training provided in this application, the detection system only needs to collect the activation gradient of the first network layer (one of the multiple network layers in the AI ​​model or a part of a network layer) running on the computing node where the AI ​​model is deployed. Then, the norm of the collected activation gradient of the first network layer is compared with a preset threshold to determine whether the norm of the activation gradient of the first network layer is abnormal. When it is determined that the norm of the activation gradient of the first network layer is abnormal, it is determined that the computing node where the first network layer is deployed has a silent data error, thereby determining that a silent data error has occurred during the AI ​​model training process. This achieves silent data error detection during the AI ​​model training process. Moreover, the parameters collected in the entire silent data detection process are relatively simple, and the detection steps are few, so the detection efficiency is high.

[0152] To enable a better understanding of the technical effects of this application, the following examples, in conjunction with Figures 7A and 7B, illustrate the interface diagrams that may appear in the silent data error detection method during the AI ​​model training process provided in this application.

[0153] Figure 7A is an example diagram of the configuration interface in a silent data error detection method during AI model training provided in this application. This interface is an exemplary demonstration and is not specifically limited in this application. As shown in Figure 7A, the configuration interface includes a model information configuration area 701 and a silent data error detection control 702.

[0154] The model information configuration area 701 includes a test model identifier input box and a test network layer identifier input box. The test model identifier input box is used for the user to input the identifier of the AI ​​model to be tested for silent data error detection, and the test network layer identifier input box is used for the user to input the identifier of the network layer in the AI ​​model to be tested for silent data error detection.

[0155] The silent data error detection control 702 is for users to click. After clicking, the detection system starts to locate the network layer to be inspected in the AI ​​model based on the information entered by the user in the model information configuration area 701, and performs silent data error detection.

[0156] Specifically, after the user clicks the silent data error detection control 702, the detection system executes the steps shown in Figure 5 or Figure 6.

[0157] For example, Figure 7B is a display interface diagram of a fault notification provided in this application. This interface is an exemplary demonstration and is not specifically limited by this application. As shown in Figure 7B, the display interface includes a fault notification display area 710.

[0158] The fault notification display area 710 is used to display fault notifications to the user. In Figure 7B, the example shown is a silent data error occurring on a compute node with fault notification identifier 0001, and should not be considered a specific limitation.

[0159] It should be noted that Figures 7A and 7B above are examples for illustration and are not intended to limit the scope of this application.

[0160] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0161] The architecture of the detection system and the silent data error detection method during the AI ​​model training process provided in this application have been described in detail above. The unit modules in the detection system provided in this application will be explained below with reference to Figure 8.

[0162] Figure 8 is a schematic diagram of the structure of a detection system 200 provided in this application. The detection system 200 can be the detection system described in the embodiments of Figures 1-7B. As shown in Figure 8, the detection system 200 may include an acquisition unit 211 and a detection unit 212. The functions of each unit module of the detection system 200 will be described by way of example below.

[0163] The acquisition unit 211 is used to acquire the activation gradient of the first network layer during the Nth iteration of AI model training. The AI ​​model includes multiple network layers, and the first network layer is one of the multiple network layers or a part of a network layer. The first network layer is deployed on the first computing node.

[0164] The detection unit 212 is used to determine the norm of the activation value gradient of the first network layer.

[0165] The detection unit 212 is also used to determine that a silent data error has occurred in the first computing node when the norm of the activation value gradient of the first network layer is greater than a first threshold.

[0166] In some possible embodiments, the first network layer is the last network layer running on the first computing node during the backpropagation process of the Nth iteration.

[0167] In some possible embodiments, as shown in FIG8, the detection system 200 further includes a processing unit 213, which is used to execute measure 1 or measure 2. Measure 1 is to replace the activation value gradient of the first network layer with 0 and issue a fault notification to the maintenance personnel, wherein the replaced activation value gradient is used to be transmitted to the next network layer of the first network layer in the backpropagation process. Measure 2 is to suspend the training of the AI ​​model and issue a fault notification to the maintenance personnel.

[0168] In some possible embodiments, the second network layer is one of multiple network layers or a portion of a network layer, and the second network layer is deployed on the second computing node; the acquisition unit 211 is further configured to acquire the activation value gradient of the second network layer during the Mth iteration of AI model training; the detection unit 212 is further configured to determine the norm of the activation value gradient of the second network layer; the detection unit 212 is further configured to determine that the second computing node is suspected of having a silent data error when the norm of the activation value gradient of the second network layer is less than or equal to a first threshold and greater than a second threshold, and to record the suspected silent data error of the second computing node; the detection unit 212 is further configured to determine that the second computing node has a silent data error when the number of times the suspected silent data error of the second computing node is recorded reaches a number threshold.

[0169] In some possible embodiments, the norm of the activation gradient of the first network layer is any one or more of the following: the maximum value of the absolute values ​​of the activation gradient of the first network layer; the standard deviation of the absolute values ​​of the activation gradient of the first network layer; or the average value of the absolute values ​​of the activation gradient of the first network layer.

[0170] In some possible implementations, the AI ​​model is a large language model.

[0171] In some possible embodiments, the silent data error is any of the following: the storage cell storing the first network layer in the first computing node is damaged; the computing cell running the first network layer in the first computing node is subjected to voltage disturbance or high-energy particle impact; the circuit of the computing cell running the first network layer in the first computing node is aged or the transistor is etched.

[0172] In specific implementations, the acquisition unit 211, the detection unit 212, and the processing unit 213 can all be implemented in software or in hardware. For example, the implementation of the detection unit 212 will be described below. Similarly, the implementation of other units such as the acquisition unit 211 can refer to the implementation of the detection unit 212.

[0173] As an example of a unit implemented in software, detection unit 212 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, detection unit 212 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0174] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0175] As an example of hardware implementation, detection unit 212 may include at least one computing device, such as a server. Alternatively, detection unit 212 may also be implemented using a CPU, application-specific integrated circuit (ASIC), programmable logic device (PLD), complex programmable logical device (CPLD), field-programmable gate array (FPGA), generic array logic (GAL), data processing unit (DPU), NPU, system on chip (SoC), offload card, accelerator card, or any combination thereof.

[0176] When the detection unit 212 includes multiple computing devices, these devices can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the detection unit 212 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the detection unit 212 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offloading cards, and accelerator cards.

[0177] It should be noted that, in other embodiments, the detection unit 212 can be used to execute any step in the silent data error detection method in the AI ​​model training process provided in this application, the acquisition unit 211 can be used to execute any step in the silent data error detection method in the AI ​​model training process provided in this application, and the processing unit 213 can be used to execute any step in the silent data error detection method in the AI ​​model training process provided in this application. The steps implemented by each unit in FIG8 can be specified as needed, and they can work together to implement different steps in the silent data error detection method in the AI ​​model training process provided in this application, so as to realize all the functions of the detection system 200.

[0178] It should be understood that the functions of each unit module described above are only the functions that the detection system 200 may have in some embodiments of this application, and this application does not limit the functions of each unit module.

[0179] It should also be understood that Figure 8 is an exemplary division method. The detection system 200 may also include more or fewer unit modules. The division method of the unit modules in the detection system 200 can be flexibly adjusted based on the actual business scenario. This application does not make specific limitations.

[0180] This application also provides a computing device 900, which can deploy the aforementioned detection system. The operation and / or function of each unit module in the computing device 900 are respectively to implement the corresponding steps in the silent data error detection method in the AI ​​model training process provided in this application.

[0181] As shown in Figure 9, the computing device 900 includes a processor 910, a memory 920, and a communication interface 930, wherein the processor 910, the memory 920, and the communication interface 930 can be interconnected via a bus 940.

[0182] The processor 910 can read program code (including instructions) stored in the memory 920, execute the program code stored in the memory 920, so that the computing device 900 executes the silent data error detection method in the AI ​​model training process provided in this application, or so that the computing device 900 deploys the detection system 200.

[0183] The processor 910 may include any one or more computing devices such as CPU, GPU, microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0184] The processor 910 executes various types of digital storage instructions, such as software or firmware programs stored in the memory 920, which enables the computing device 900 to provide a wide range of services.

[0185] In a specific implementation, as one example, the processor 910 includes one or more CPUs.

[0186] In a specific implementation, as one example, the computing device 900 also includes multiple processors, each of which can be a single-core processor (single-CPU) or a multi-core processor (multi-CPU). Here, a processor refers to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0187] The memory 920 is used to store program code, which is executed by the processor 910 to perform the silent data error detection method in the AI ​​model training process provided in this application. The program code may include one or more software modules, such as the software modules provided in the embodiment of FIG8, such as the acquisition unit 211, the detection unit 212, and the processing unit 213.

[0188] The memory 920 may include volatile memory, such as random access memory (RAM); the memory 920 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid-state drive (SSD); the memory 920 may also include combinations of the above types.

[0189] The communication interface 930 can be a wired interface (e.g., Ethernet interface, fiber optic interface, other types of interface (e.g., InfiniBand interface)) or a wireless interface (e.g., cellular network interface or wireless LAN interface) for communicating with other computing devices or apparatuses. The communication interface 930 can employ a protocol suite based on Transmission Control Protocol / Internet Protocol (TCP / IP), such as Remote Function Call (RFC), Simple Object Access Protocol (SOAP), Simple Network Management Protocol (SNMP), Common Object Request Broker Architecture (CORBA), and distributed protocols, etc.

[0190] The 940 bus can be a peripheral component interconnect express (PCIe) bus, an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL) bus, a cache coherent interconnect for accelerators (CCIX) bus, etc. The 940 bus can be divided into address bus, data bus, control bus, etc.

[0191] In addition to the data bus, bus 940 may also include a power bus, a control bus, and a status signal bus. However, for clarity, all buses are labeled as bus 940 in the figure. For ease of illustration, only one thick line is used in Figure 9, but this does not mean that there is only one bus or one type of bus.

[0192] As one possible implementation, the computing device 900 may also include a chip system, which includes a processor 910 and a power supply circuit. The power supply circuit supplies power to the processor 910, which executes the operation steps corresponding to the silent data error detection method during AI model training. For simplicity, further details are omitted here. The processor 910 can be implemented using a CPU, or it can be implemented using computing devices or AI chips such as GPUs, DPUs, NPUs, XPUs, SoCs, offloading cards, or accelerator cards.

[0193] As one possible implementation, the computing device 900 may include multiple types of processors 910, meaning the computing device 900 is a heterogeneous device. For example, the computing device 900 may include a CPU and a GPU, and at least one of these processors can execute the operation steps corresponding to the silent data error detection method during the AI ​​model training process. For the sake of brevity, further details will not be elaborated here.

[0194] The aforementioned computing device 900 is used to execute the silent data error detection method in the AI ​​model training process provided in this application. The specific implementation process is detailed in the above method embodiments and will not be repeated here.

[0195] It should be understood that the computing device 900 is merely one example provided in the embodiments of this application, and the computing device 900 may have more or fewer components than those shown in FIG. 9, may combine two or more components, or may be implemented with different configurations of components. For content not shown or described in the embodiments of this application, please refer to the relevant descriptions in the embodiments of FIG. 1-8 above, which will not be repeated here.

[0196] This application also provides a computing device cluster 1000, which can deploy the aforementioned detection system. The operation and / or function of each unit module in the computing device cluster 1000 are respectively to implement the corresponding steps in the silent data error detection method in the AI ​​model training process provided in this application.

[0197] As shown in Figure 10, the computing device cluster 1000 includes at least one computing device 900. The memory 920 of one or more computing devices 900 in the computing device cluster may store the same instructions for executing the silent data error detection method during the AI ​​model training process provided in this application. The computing device 900 may be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device 900 may also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0198] In some possible implementations, the memory 920 of one or more computing devices 900 in the computing device cluster 1000 may also store instructions for executing the silent data error detection method during the AI ​​model training process provided in this application. In other words, a combination of one or more computing devices 900 can jointly execute instructions for executing the silent data error detection method during the AI ​​model training process provided in this application.

[0199] It should be noted that the memory 920 in different computing devices 900 within the computing device cluster 1000 can store different instructions, which are used to execute certain functions of the detection system 200. That is, the instructions stored in the memory 920 of different computing devices 900 can implement the functions of one or more units among the acquisition unit 211, the detection unit 212, and the processing unit 213.

[0200] In some possible implementations, one or more computing devices 900 in the computing device cluster 1000 can be connected via a network. This network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 11 illustrates one possible implementation, where two computing devices 900A and 900B are connected via a network. Specifically, they are connected to the network through communication interfaces in each computing device. In this type of possible implementation, the memory 920 in computing device 900A stores instructions for executing the functions of the acquisition unit 211 and the processing unit 213. Simultaneously, the memory 920 in computing device 900B stores instructions for executing the functions of the detection unit 212.

[0201] The connection method between the computing device cluster 1000 shown in Figure 11 can be based on the silent data error detection method provided in this application for the AI ​​model training process, which requires silent data error detection for the training process of a large number of AI models for a large number of users. Therefore, it is considered to delegate the function implemented by the detection unit 212 to the computing device 900B.

[0202] It should be understood that the functions of computing device 900A shown in Figure 11 can also be performed by multiple computing devices 900. Similarly, the functions of computing device 900B can also be performed by multiple computing devices 900.

[0203] This application also provides a computer program product containing instructions, which may be a software or program product containing instructions capable of running on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute the silent data error detection method in the AI ​​model training process provided in this application.

[0204] This application also provides a computer-readable storage medium, which can be any available medium capable of being stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., high-density digital video disc (DVD), or a semiconductor medium (e.g., solid-state drive), etc. The computer-readable storage medium includes instructions that instruct a computing device to execute the silent data error detection method provided in this application during the AI ​​model training process.

[0205] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0206] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium, or a semiconductor medium, etc.

[0207] The above description is merely a specific embodiment of this application. Any variations or substitutions conceived by those skilled in the art based on the specific embodiments provided in this application should be covered within the protection scope of this application.

Claims

1. A method for detecting silent data errors during the training process of an artificial intelligence (AI) model, characterized in that, The AI model includes a plurality of network layers, a first network layer is one network layer or a part of one network layer in the plurality of network layers, the first network layer is deployed on a first computing node, and the method includes: obtaining the activation value gradient of the first network layer in the Nth iteration process of training the AI model; determining the norm of the activation value gradient of the first network layer; in a case where it is determined that the norm of the activation value gradient of the first network layer is greater than a first threshold, determining that the first computing node has a silent data error.

2. The method of claim 1, wherein, The first network layer is the last network layer running on the first computing node in the back propagation process of the Nth iteration.

3. The method according to claim 1 or 2, characterized in that, After determining that the first computing node has a silent data error, the method further includes: replacing the activation value gradient of the first network layer with 0, and sending a fault notification to maintenance personnel, wherein the replaced activation value gradient is used to be transmitted to the next network layer of the first network layer in the back propagation process; or, pausing training the AI model, and sending a fault notification to the maintenance personnel.

4. The method according to any one of claims 1 to 3, characterized in that, A second network layer is one network layer or a part of one network layer in the plurality of network layers, and the second network layer is deployed on a second computing node. The method further includes: obtaining the activation value gradient of the second network layer in the Mth iteration process of training the AI model; determining the norm of the activation value gradient of the second network layer; in a case where it is determined that the norm of the activation value gradient of the second network layer is less than or equal to the first threshold and greater than a second threshold, determining that the second computing node has a suspected silent data error, and recording that the second computing node has a suspected silent data error; in a case where the number of times that the second computing node has a suspected silent data error reaches a number threshold, determining that the second computing node has a silent data error.

5. The method according to any one of claims 1 to 4, characterized in that, The norm of the activation value gradient of the first network layer includes any one or more of: a maximum value in the absolute value of the activation value gradient of the first network layer; a standard deviation of the absolute value of the activation value gradient of the first network layer; an average value of the absolute value of the activation value gradient of the first network layer.

6. The method according to any one of claims 1 to 5, characterized in that, The AI model is a large language model.

7. The method according to any one of claims 1 to 6, characterized in that, The silent data error is any one of: a storage unit storing the first network layer in the first computing node is damaged; a computing unit running the first network layer in the first computing node is subjected to voltage disturbance or high-energy particle impact; a circuit of the computing unit running the first network layer in the first computing node is aged or transistor etching.

8. A detection system characterized by, The system includes: an obtaining unit configured to obtain an activation value gradient of a first network layer in an Nth iteration process of training an AI model, wherein the AI model includes a plurality of network layers, the first network layer is one network layer or a part of one network layer in the plurality of network layers, and the first network layer is deployed on a first computing node; a detection unit configured to determine a norm of the activation value gradient of the first network layer; The detection unit is further configured to determine that the first computing node has a silent data corruption in a case where it is determined that the norm of the activation gradient of the first network layer is greater than a first threshold.

9. The system of claim 8, wherein, The first network layer is a last network layer running on the first computing node in a backpropagation process of an Nth iteration.

10. The system of claim 8 or 9, characterized in that, The system further comprises a processing unit. The processing unit is configured to: replace the activation gradient of the first network layer with 0, and send a failure notification to a maintenance personnel, wherein the replaced activation gradient is used to be transmitted to a next network layer of the first network layer in the backpropagation process; or suspend training of the AI model, and send a failure notification to the maintenance personnel.

11. The system according to any one of claims 8 to 10, characterized in that, The second network layer is one network layer or a part of one network layer in the plurality of network layers, and the second network layer is deployed on a second computing node. The obtaining unit is further configured to obtain an activation gradient of the second network layer in an Mth iteration process of training of the AI model. The detection unit is further configured to determine the norm of the activation gradient of the second network layer. The detection unit is further configured to determine that the second computing node has a suspected silent data corruption in a case where it is determined that the norm of the activation gradient of the second network layer is less than or equal to the first threshold and greater than a second threshold, and record that the second computing node has the suspected silent data corruption. The detection unit is further configured to determine that the second computing node has a silent data corruption in a case where a number of times that the second computing node has the suspected silent data corruption reaches a number threshold.

12. The system according to any one of claims 8 to 11, characterized in that, The norm of the activation gradient of the first network layer includes any one or more of: a maximum value in the absolute value of the activation gradient of the first network layer; a standard deviation of the absolute value of the activation gradient of the first network layer; an average value of the absolute value of the activation gradient of the first network layer.

13. The system of any one of claims 8 to 12, wherein, The AI model is a large language model.

14. The system of any one of claims 8 to 13, wherein, The silent data corruption is any one of: a storage unit storing the first network layer in the first computing node is damaged; a computing unit running the first network layer in the first computing node is subjected to voltage disturbance or high-energy particle impact; a circuit of the computing unit running the first network layer in the first computing node is aged or a transistor of the computing unit is etched.

15. A cluster of computing devices, characterized in that, The at least one computing device includes a processor and a memory. The processor of the at least one computing device is configured to execute instructions stored in the memory of the at least one computing device, so that the computing device cluster executes the method according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, and the instructions are executed by a computing device or a computing device cluster to implement the method according to any one of claims 1 to 7.

17. A computer program product comprising instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the operation steps of the method according to any one of claims 1 to 7.

Images