Model training method, model storage method and related devices

Abstract

The application provides a model training method, a model storage method and related devices, which can be applied to the technical field of computers. The model training method comprises the following steps: when a fault occurs during training, using a first calculation result and intermediate results generated in multiple iterations after the generation of the first calculation result to restore the calculation results in the multiple iterations after the generation of the first calculation result, and then continuing to train the model according to the restored calculation results until the calculation result before the fault is restored, and then continuing to train backward. Since the first calculation result and the multiple intermediate results can be used to quickly calculate the multiple calculation results corresponding to the multiple intermediate results one by one, the method can reduce the calculation cost and time cost of restoring the model, and reduce the time cost of the training task.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a model training method, a model storage method, and related apparatus. Background Technology

[0002] The training of large models is trending towards scaling, and the resources required to execute training tasks are increasing dramatically. Therefore, large models are often trained in a distributed manner using computing device clusters to improve training speed. However, during distributed training, situations frequently arise where training tasks become abnormal due to the failure of a single device.

[0003] To address this, the computational results (including model parameters, optimizer state, etc.) of the model at different training stages can be periodically saved during the training of a large model. For example, the computational results of the model in each iteration can be saved, or the current computational results of the model can be saved every 50 iterations. In this way, when a device fails, the model training process can be restarted based on the periodically saved computational results.

[0004] However, the computational results are large in volume and limited by factors such as storage bandwidth and space of the computing device cluster. Saving the computation results of a single model typically takes hours, while iterating over a single model typically takes minutes. For example, if saving the computation results of a single model takes three hours and iterating over a single model takes five minutes, then starting from the first iteration, it takes three hours to save the computation results of the first iteration, but 36 iterations of the model have already been completed. Therefore, the more iterations there are, the longer the training time, and the greater the difference in iteration count between the current iteration and the iteration containing the currently stored computation results. If the equipment fails, the model can only be retrained based on the latest saved computation results until the computation results before the failure are restored, and then training can continue. Therefore, the computational and time costs of restoring training are significant, greatly increasing the time cost of the training task. Summary of the Invention

[0005] To address the aforementioned technical problems, this application provides a model training method, a model storage method, and related apparatus, which can reduce the computational and time overhead of resuming training when a training model malfunctions, thereby reducing the time cost of the training task.

[0006] Firstly, a model training method is provided, comprising: in the event of an anomaly during model training, obtaining a first calculation result and multiple intermediate results; calculating multiple calculation results based on the first calculation result and the multiple intermediate results; and continuing to iterate the model based on any one of the multiple calculation results. The first calculation result is the final output of the model at the end of one iteration. The multiple intermediate results are generated in multiple iterations after the generation of the first calculation result, and there is a one-to-one correspondence between the multiple intermediate results and the multiple iterations. The calculation results are used to indicate the final output of the model at the end of the corresponding iteration. The amount of data in the intermediate results is less than the amount of data in the corresponding calculation results.

[0007] This technical solution can recover the calculation results of subsequent iterations after the generation of the first calculation result by using the first calculation result and intermediate results generated in multiple iterations after the generation of the first calculation result when the training model encounters an anomaly. The model can then be trained again based on the recovered calculation results until the calculation results before the failure are restored, and training can continue. Since the process of calculating multiple calculation results corresponding one-to-one with the intermediate results using the first calculation result is much shorter than the existing method of periodically saving the calculation results of the model at different training stages and retraining the model using the latest saved calculation results to obtain the calculation results before the failure, this technical solution can reduce the computational and time costs of restoring training, thus reducing the time cost of the training task. Furthermore, while existing model saving schemes save the calculation results of the model in each iteration, providing the latest saved iteration's calculation results to recover the calculation results before the failure, this requires storing a large amount of data. This technical solution, however, uses the first calculation result and intermediate results generated in multiple iterations after the generation of the first calculation result to calculate the calculation results of multiple iterations after the generation of the first calculation result. This computation-based approach achieves the same effect of recovering the calculation results before the failure.

[0008] In some possible implementations, the above calculation results include the optimizer state, model weights, and biases. The optimizer state is used to update the model weights and biases. The intermediate results include the model's weight gradients and bias gradients in the corresponding iteration, or the intermediate results include the model's loss in the corresponding iteration, or the intermediate results include the input data of the model's loss function in the corresponding iteration.

[0009] In the above scheme, when the intermediate result includes the model's weight gradient and bias gradient in the corresponding iteration, the calculation result of the current iteration can be obtained simply by updating the calculation result of the previous iteration using the weight gradient and bias gradient of the model in a certain iteration. Since this calculation process is only a part of the backpropagation process in a complete iteration, this method omits the forward propagation process and the remaining calculation processes in the backpropagation process (specifically, the process of calculating the weight gradient and bias gradient based on the model's loss in that iteration). Therefore, this method has a short computation time and low computational cost for calculating the calculation result of one iteration. When the intermediate result is the model's loss in the corresponding iteration, the calculation result of the current iteration can be obtained simply by using the model's loss in a certain iteration to calculate the model's weight gradient and bias gradient in that iteration, and then using these weight gradients and bias gradients to update the calculation result of the previous iteration. Since this calculation process only involves the backpropagation of a complete iteration of the model, it omits the forward propagation process, resulting in shorter computation time and lower computational cost for calculating the result of a single iteration. When the intermediate result is the input data of the model's loss function in the corresponding iteration, the calculation can be completed by using the input data of the loss function from a particular iteration to calculate the model's loss in that iteration, then using that loss to calculate the model's weight and bias gradients, and finally using those gradients to update the result of the previous iteration. Because this calculation process only involves a portion of the forward propagation and backpropagation in a complete iteration, it omits the remaining computational steps in the forward propagation process (specifically, the process of inputting samples into the model obtained from the previous iteration to obtain the input data of the loss function in the current iteration). Therefore, this method results in shorter computation time and lower computational cost for calculating the result of a single iteration. In summary, whether the intermediate results include the model's weight gradients and bias gradients in the corresponding iteration, the intermediate results are the model's loss in the corresponding iteration, or the intermediate results are the input data of the model's loss function in the corresponding iteration, this technical solution recovers multiple computational results much faster than the existing method of periodically saving the model's computational results at different training stages and using the latest saved computational results to perform multiple complete iterations of the model to recover multiple computational results. Therefore, this technical solution can reduce the computational and time costs of recovering training results and reduce the time cost of training tasks.

[0010] In some possible implementations, the first calculation result mentioned above is N. X The calculation results of the first iteration in the next iteration, including multiple intermediate results, include the model in N. X The intermediate results from the second to the Qth iteration in the current iteration.

[0011] After performing N on the model X In the event of an anomaly during the Rth iteration, multiple calculation results are obtained based on the first calculation result and multiple intermediate results. These include: calculating the (q+1)th iteration result by sequentially using the calculation result of the qth iteration and the intermediate results of the (q+1)th iteration, in ascending order of q value, thereby obtaining the model's performance in N... X The calculation results from the 2nd to the Qth iteration in this iteration. Where N... X R, Q, and q are all positive integers, where R > 2 and Q < R.

[0012] The above-mentioned iteration of the model based on any one of the multiple calculation results includes:

[0013] Based on the model in N X In this iteration, any calculation result from the second to the Qth iteration is used to iterate the model again, thereby obtaining the model in N. X The calculation result of the Rth iteration in the nth iteration.

[0014] In the above scheme, when the model is performing N... X When an anomaly occurs during the Rth iteration in the Nth iteration, this technical solution retrieves the already stored model in N. X The model is recovered using the calculation results of the first iteration and the intermediate results from the second to the Qth iteration in N. X The calculation results from the second to the Qth iteration in the second iteration can then be used to apply the model to N. X In this iteration, any calculation result from the second to the Qth iteration continues to iterate the model until the calculation result before the anomaly is recovered (i.e., the model is in N). X The calculation result of the Rth iteration in the nth iteration). Since the process of calculating the result of the (q+1)th iteration using the calculation result of the qth iteration and the intermediate result of the (q+1)th iteration is much shorter than the process of directly using the calculation result of the qth iteration to perform a complete iteration of the model to obtain the calculation result of the (q+1)th iteration, this technical solution restores the model in N X The computation speed of the second to Qth iterations in this iteration is much faster than that of the existing scheme. The latest saved computation results for the model in N... X Given the calculation results of the first iteration in the next iteration, the model is used in N.X In this iteration, the calculation result of the first iteration is used to perform Q-1 consecutive complete iterations on the model to recover the model in N. X The process of calculating the results from the second to the Qth iteration in the next iteration. Therefore, this technical solution can reduce the computational and time overhead of resuming training and reduce the time cost of the training task.

[0015] In some possible implementations, the above model-based approach is used in N. X In the next iteration, any calculation result from the second to the Qth iteration can be used to continue iterating the model, including using the calculation result of the Qth iteration to continue iterating the model.

[0016] Since the calculation result of the Qth iteration is the closest to the calculation result before the anomaly among the calculation results from the 2nd to the Qth iteration, using the calculation result of the Qth iteration to continue iterating the model until the calculation result before the anomaly is restored can save the time and computational cost of the restoration process compared to using other calculation results.

[0017] In some possible implementations, the above model-based approach is used in N. X In this iteration, any calculation result from the second to the Qth iteration will continue to iterate the model, including: completing N on multiple computing devices. X In the case of the S-th iteration in the 1st iteration, if S < Q, the model is iterated again using the calculation result of the S-th iteration; if S ≥ Q, the model is iterated again using the calculation result of the Q-th iteration. Multiple computing devices are used to train the model, and S is a positive integer.

[0018] This technical solution provides a method for recovering models in distributed training scenarios. When multiple computing devices are used to train a model together, the different computing performance of each device leads to different iteration progress. This technical solution obtains the iteration progress of different computing devices and retrains the model based on the iteration progress of the lagging computing device, thereby ensuring data consistency among all computing devices.

[0019] In some possible implementations, when performing N on the model X In the case of an anomaly occurring during the Rth iteration in the nth iteration, the first calculation result is the model in N. M The calculation results of the first iteration in the next iteration, including multiple intermediate results: the model in N M In the second iteration to the Nth iteration M Intermediate results of the Nth iteration, and the model at N X The intermediate results from the 1st to the Qth iteration in this iteration. Where N... MIn the Nth iteration M The next iteration is N. X The iteration preceding the first iteration in the next iteration, N X R, Q, N M All are positive integers, R > 2, Q < R.

[0020] In the above scheme, after performing N on the model X In the case of an anomaly occurring during the Rth iteration in the nth iteration, even if the model is not stored in N... X In the next iteration, the calculation results of the first iteration can still be used to recover the model by using the stored calculation results from previous iterations. Specifically, when using the model in N... M In the Nth iteration M The intermediate results of the next iteration are used to calculate the model in N. M In the Nth iteration M After the calculation results of the nth iteration, since N M In the Nth iteration M The next iteration is N. X The iteration preceding the first iteration in the next iteration, therefore the model can be used again in N. M In the Nth iteration M The calculation results of the next iteration and the model in N X The intermediate results of the first iteration in the next iteration are used to calculate the model in N. X The calculation result of the first iteration in the next iteration can then be used sequentially with N. X The intermediate results from the second to the Qth iteration in the next iteration are used to recover the model in N. X The calculation results from the 2nd to the Qth iteration in this iteration. If the model is not stored in N... X The intermediate result of the first iteration in the nth iteration is then used in the model in N. M In the Nth iteration M The intermediate results of the next iteration are used to calculate the model in N. M In the Nth iteration M After calculating the results of the next iteration, the model needs to be used in N... M In the Nth iteration M The calculation results of the second iteration are used to perform a complete iteration of the model to obtain the model's performance in N. X The calculation result of the first iteration in the next iteration. Therefore, compared to not storing the model in N... X The intermediate results of the first iteration in the next iteration, when the model is stored in N X When using the intermediate results of the first iteration in the next iteration, the time and computational costs of restoring the model are both smaller.

[0021] Secondly, a model storage method is provided, which includes: storing the model in NM The calculation result of the first iteration in the next iteration. The storage model is in N. M The intermediate result of the Kth iteration in the Nth iteration. Based on storing the intermediate result of the Kth iteration, the storage model in N... M The intermediate results of the (K+1)th iteration in the Nth iteration. Based on storing the intermediate results of the Kth and (K+1)th iterations, the storage model in N... M The intermediate result of the (K+2)th iteration in the k-th iteration. The total number of iterations of the model is N, where N... M ≤N, N M Both N and K are positive integers. The intermediate results of the Kth iteration are used to calculate the result of the Kth iteration. The result of the Kth iteration is the final output of the model at the end of the Kth iteration. The amount of data in the intermediate results of the Kth iteration is less than the amount of data in the result of the Kth iteration, K ≥ 2, and K is an integer.

[0022] This technical solution can store the model in N. M This solution uses the calculation results of the first iteration and the intermediate results of each iteration starting from the second iteration. In the event of a training failure, the stored calculation results and intermediate results generated in subsequent iterations can be used to recover the calculation results of the corresponding iteration. This allows the recovered calculation results to be used for subsequent model training until the calculation results before the failure are restored, and then training can continue. Compared to existing solutions that periodically save the calculation results of the model at different training stages and can only provide the most recently saved calculation results to retrain the model to obtain the calculation results before the failure, this solution helps to shorten the model recovery process, reduce the computational and time costs of recovery training, and reduce the time cost of the training task. Furthermore, related model saving solutions save the calculation results of the model in each iteration, requiring a large amount of data to be stored. This solution only needs to store the calculation results of one iteration and intermediate results from multiple iterations. Since the amount of data in the intermediate results is less than that in the corresponding calculation results, this solution can reduce the amount of data stored. This method of breaking down the data into smaller parts reduces the model's storage cost.

[0023] In some possible implementations, the computation result of the first iteration includes the optimizer state, model weights, and biases. The optimizer state is used to update the model weights and biases. The intermediate result of the Kth iteration includes the model's weight gradient and bias gradient in the Kth iteration, or the intermediate result of the Kth iteration includes the model's loss in the Kth iteration, or the intermediate result of the Kth iteration includes the input data of the model's loss function in the Kth iteration.

[0024] In the above scheme, whether the intermediate results include the model's weight gradients and bias gradients in the corresponding iteration, the intermediate results are the model's loss in the corresponding iteration, or the intermediate results are the input data of the model's loss function in the corresponding iteration, the amount of data in the intermediate results is significantly less than the amount of data in the computational results. Therefore, compared with storing the computational results of the model in each iteration, this technical solution can reduce the amount of data stored and reduce the model's storage cost. In addition, this technical solution helps to shorten the calculation process of the corresponding computational results by storing intermediate results. Therefore, this technical solution helps to shorten the model recovery process, reduce the computational and time costs of recovery training, and reduce the time cost of the training task.

[0025] In some possible implementations, the above method also includes: storing the model in N M In the Nth iteration M Based on the intermediate results of the Nth iteration, the storage model is in N X The intermediate results of the Pth iteration in the nth iteration. The model is stored in N. M In the Nth iteration M The intermediate results of the next iteration and the model in N X Based on the intermediate results of the Pth iteration in the Nth iteration, the storage model is in N X The intermediate result of the (P+1)th iteration in the nth iteration. Where N M In the Nth iteration M The next iteration is N. X The iteration preceding the first iteration in the next iteration. The model in N... M In the Nth iteration M The intermediate results of the second iteration are used to calculate the model in N. M In the Nth iteration M The calculation results of the Nth iteration. The model in N M In the Nth iteration M The calculation results of the next iteration and the model in N X The intermediate results of the first iteration in the next iteration are used to calculate the model in N. X The result of the first iteration in the next iteration. N M <N, N X <N, N X P is a positive integer, P≥1, and P is an integer.

[0026] In the above scheme, by storing the model in N M In the Nth iteration M Based on the intermediate results of the Nth iteration, the storage model is in N X The intermediate results of the Pth iteration in the nth iteration can be stored in the model in N. XThe intermediate result of the first iteration in the next iteration. Thus, in the event of a failure during subsequent training, due to N... M In the Nth iteration M The next iteration is N. X The iteration preceding the first iteration in the next iteration, thus the model can be used in N. M In the Nth iteration M The calculation results of the next iteration and the model in N X The intermediate results of the first iteration in the next iteration are used to calculate the model in N. X The calculation result of the first iteration in the next iteration can then be used sequentially with N. X In each iteration, the intermediate results of each iteration starting from the second iteration are used to reconstruct the calculation results of the corresponding iteration. If the model is not stored in N... X The intermediate results of the first iteration in the next iteration need to be used with the model in N. M In the Nth iteration M The calculation results of the second iteration are used to perform a complete iteration of the model to obtain the model's performance in N. X The calculation result of the first iteration in the next iteration. Therefore, compared to not storing the model in N... X The intermediate results of the first iteration in the next iteration, when the model is stored in N X The intermediate results of the first iteration in the next iteration help to further reduce the time and computational costs of the recovery model.

[0027] In some possible implementations, the above method also includes: a storage model in N X The calculation result of the first iteration in the next iteration. Where N... M In the Nth iteration M The execution time of this iteration is earlier than N. X The execution time of the first iteration in the next iteration. The successful storage of the model in N... X Given the calculation results of the first iteration in the next iteration, delete the model in N. M The calculation results of the first iteration in the second iteration and the intermediate results of each iteration starting from the second iteration.

[0028] When new computation results are stored, if a training failure occurs, the model can be recovered directly from the new computation results without having to start from the old ones. Therefore, this technical solution supports deleting stored old computation results and intermediate results from multiple iterations between the old and new results, based on the storage of new computation results. This saves storage space and reduces storage costs.

[0029] In some possible implementations, the above N XThe value of is negatively correlated with the anomaly frequency of the computing device. Here, the anomaly frequency is the N-order frequency of the model implemented on the computing device. M The frequency of anomalies is determined during the iteration process, and / or the frequency of anomalies is determined during the operation of the computing device before the model is trained.

[0030] This technical solution provides an N X The method for determining the value. This method supports starting with N X Before the next iteration, N is determined based on the fault status of the computing device. X The numerical values ​​are adaptively adjusted, which essentially changes the frequency at which the calculation results are saved. In scenarios where the resources used to store the model are the same as those used to train the model, adaptively adjusting the frequency of saving the calculation results helps balance the efficiency of the training model and the speed of model recovery.

[0031] Thirdly, a computing device is provided, comprising a result acquisition unit and a recovery unit. The result acquisition unit is used to acquire a first calculation result and multiple intermediate results when an anomaly occurs in the training model. The recovery unit is used to calculate multiple calculation results based on the first calculation result and the multiple intermediate results, and then continue iterating the model based on any one of the multiple calculation results. The first calculation result is the final output of the model at the end of one iteration. The multiple intermediate results are generated in multiple iterations after the generation of the first calculation result, and there is a one-to-one correspondence between the multiple intermediate results and the multiple iterations. The calculation result is used to indicate the final output of the model at the end of the corresponding iteration. The amount of data in the intermediate results is less than the amount of data in the corresponding calculation result.

[0032] In some possible implementations, the above calculation results include the optimizer state, model weights, and biases. The optimizer state is used to update the model weights and biases. The above intermediate results include the model's weight gradients and bias gradients in the corresponding iteration, or the intermediate results include the model's loss in the corresponding iteration, or the intermediate results include the input data of the model's loss function in the corresponding iteration.

[0033] In some possible implementations, the first calculation result mentioned above is N. X The calculation results of the first iteration in the next iteration, including multiple intermediate results, include the model in N. X The intermediate results from the 2nd to the Qth iteration in this iteration. The aforementioned recovery unit is specifically used for: performing N iterations on the model... X In the case of an anomaly occurring during the Rth iteration, the result of the (q+1)th iteration is calculated by sequentially using the calculation result of the qth iteration and the intermediate result of the (q+1)th iteration, following the order of q values ​​from smallest to largest. This yields the model's result in N... XThe calculation results from the second to the Qth iteration in the next iteration are then used to calculate the model in N. X In this iteration, any calculation result from the second to the Qth iteration is used to iterate the model again, thereby obtaining the model in N. X The calculation result of the Rth iteration in the nth iteration. Where N X R, Q, and q are all positive integers, where R > 2 and Q < R.

[0034] In some possible implementations, the aforementioned recovery unit is specifically used to continue iterating the model using the computational results of the Qth iteration.

[0035] In some possible implementations, the aforementioned recovery unit is specifically used to: complete N on multiple computing devices X In the case of the S-th iteration in the 1st iteration, if S < Q, the model is iterated again using the calculation result of the S-th iteration; if S ≥ Q, the model is iterated again using the calculation result of the Q-th iteration. Multiple computing devices are used to train the model, and S is a positive integer.

[0036] In some possible implementations, when performing N on the model X In the event of an anomaly during the Rth iteration of the nth iteration, the first calculation result mentioned above represents the model in N. M The calculation results of the first iteration in the next iteration, including multiple intermediate results: the model in N M In the second iteration to the Nth iteration M Intermediate results of the Nth iteration, and the model at N X The intermediate results from the 1st to the Qth iteration in this iteration. Where N... M In the Nth iteration M The next iteration is N. X The iteration preceding the first iteration in the next iteration. N X R, Q, N M All are positive integers, R > 2, Q < R.

[0037] Fourthly, a storage device is provided, comprising a first storage unit and a second storage unit. The first storage unit is used to store a model in N... M The calculation result of the first iteration in the next iteration. The second storage unit is used to store the model in N. M The intermediate result of the Kth iteration in the Nth iteration. The second storage unit is also used to store the model in N, based on storing the intermediate result of the Kth iteration. M The intermediate result of the (K+1)th iteration in the next iteration. The second storage unit is also used to store the intermediate results of the Kth and (K+1)th iterations, and to store the model in N... MThe intermediate result of the (K+2)th iteration in the k-th iteration. The total number of iterations of the model is N, where N... M ≤N, N M Both N and K are positive integers. The intermediate results of the Kth iteration are used to calculate the result of the Kth iteration. The result of the Kth iteration is the final output of the model at the end of the Kth iteration. The amount of data in the intermediate results of the Kth iteration is less than the amount of data in the result of the Kth iteration. K ≥ 2, where K is an integer.

[0038] In some possible implementations, the computation result of the first iteration includes the optimizer state, model weights, and biases. The optimizer state is used to update the model weights and biases. The intermediate result of the Kth iteration includes the model's weight gradient and bias gradient in the Kth iteration, or the intermediate result of the Kth iteration includes the model's loss in the Kth iteration, or the intermediate result of the Kth iteration includes the input data of the model's loss function in the Kth iteration.

[0039] In some possible implementations, the aforementioned second storage unit is further used for: storing models in N M In the Nth iteration M Based on the intermediate results of the Nth iteration, the storage model is in N X The intermediate results of the Pth iteration in the nth iteration; in the model stored in N M In the Nth iteration M The intermediate results of the next iteration and the model in N X Based on the intermediate results of the Pth iteration in the Nth iteration, the storage model is in N X The intermediate result of the (P+1)th iteration in the nth iteration. Where N M In the Nth iteration M The next iteration is N. X The iteration preceding the first iteration in the next iteration. The model in N... M In the Nth iteration M The intermediate results of the second iteration are used to calculate the model in N. M In the Nth iteration M The calculation results of the Nth iteration. The model in N M In the Nth iteration M The calculation results of the next iteration and the model in N X The intermediate results of the first iteration in the next iteration are used to calculate the model in N. X The result of the first iteration in the next iteration. N M <N, N X <N, N X P is a positive integer, P≥1, and P is an integer.

[0040] In some possible implementations, the first storage unit is also used to store the model in N.X The calculation result of the first iteration in the next iteration. Where N... M In the Nth iteration M The execution time of this iteration is earlier than N. X The execution time of the first iteration in the next iteration. The first storage unit is also used to store the successful model in N. X Given the calculation results of the first iteration in the next iteration, delete the model in N. M The calculation result of the first iteration in the next iteration. The second storage unit is also used to store the successful model in N. X Given the calculation results of the first iteration in the next iteration, delete the model in N. M Intermediate results of each iteration starting from the second iteration in the next iteration.

[0041] In some possible implementations, the above N X The value of is negatively correlated with the anomaly frequency of the computing device. This anomaly frequency is defined as the N-order frequency of the model implemented on the computing device. M The frequency of anomalies is determined during the iteration process, and / or the frequency of anomalies is determined during the operation of the computing device before the model is trained.

[0042] Fifthly, a chip is provided, including a processor and a power supply circuit for supplying power to the processor, which is used to perform the method of any of the first aspects or the method of any of the second aspects.

[0043] In a sixth aspect, a computing device is provided, the computing device including a processor and a memory, the processor being configured to execute instructions stored in the memory such that the computing device implements the method of any of the first aspects, or implements the method of any of the second aspects.

[0044] In a seventh aspect, a computing device cluster is provided, comprising multiple computing devices, each computing device including a processor and memory.

[0045] The memory of the plurality of computing devices stores instructions for implementing the method as described in any of the first aspects, and the processor of the plurality of computing devices implements the instructions stored in the memory of the plurality of computing devices, so that the cluster of computing devices implements the method as described in any of the first aspects. Alternatively, the memory of the plurality of computing devices stores instructions for implementing the method as described in any of the second aspects, and the processor of the plurality of computing devices implements the instructions stored in the memory of the plurality of computing devices, so that the cluster of computing devices implements the method as described in any of the second aspects.

[0046] Eighthly, a computer program product comprising instructions is provided, which, when executed by a computing device, causes the computing device to implement the method of any of the first aspects, or the method of any of the second aspects.

[0047] A ninth aspect provides a computer-readable storage medium, characterized in that it includes computer program instructions, which, when executed by a computing device, implement the method of any of the first aspects or the method of any of the second aspects. Attached Figure Description

[0048] Figure 1 This is a system architecture diagram of a training model provided in an embodiment of this application;

[0049] Figure 2 This is a schematic diagram of a training model provided in an embodiment of this application;

[0050] Figure 3 This is a flowchart illustrating a model storage method provided in an embodiment of this application;

[0051] Figure 4 This is a schematic flowchart of a model training method provided in an embodiment of this application;

[0052] Figure 5 This is a schematic diagram of the structure of a storage device provided in an embodiment of this application;

[0053] Figure 6 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0054] Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0055] Figure 8 This is a schematic diagram of the structure of a computing device cluster provided in an embodiment of this application;

[0056] Figure 9 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. Detailed Implementation

[0057] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0058] See Figure 1 , Figure 1This is a system architecture diagram of a training model provided in an embodiment of this application. For example... Figure 1 As shown, the architecture includes a terminal device 100, a computing device 200, and a storage device 300. The terminal device 100 and the computing device 200, as well as the computing device 200 and the storage device 300, can communicate via wired or wireless means.

[0059] Terminal device 100 generates input data for the model—a dataset—and sends the dataset to computing device 200.

[0060] Among them, terminal device 100 is an electronic device used to access computing devices, including personal computers, smartphones, handheld computers (PDAs), tablet computers, mobile laptops, all-in-one handheld devices (such as game handhelds), smart conferencing devices, smart learning machines, etc.

[0061] The computing device 200 is used to train a model using a dataset, obtain a trained model, and then send the trained model to the storage device 300. Here, "model" refers to a system that can simulate human intelligent behavior, obtained through computer algorithms and data training; it is also called an artificial intelligence (AI) model, such as a deep learning model.

[0062] In some possible implementations, computing device 200 is an electronic device used for computing, processing and storing data, including servers, supercomputers, personal computers, workstations, mobile devices and so on.

[0063] In some other possible implementations, the computing device 200 is an artificial intelligence (AI) chip. This AI chip, also known as an AI accelerator or computing card, is a hardware device that integrates a dedicated AI processor. The AI ​​processor can be, for example, a graphics processing unit (GPU), a data processing unit (DPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or a machine learning unit (MLU).

[0064] Optionally, the number of computing devices 200 may be one or more.

[0065] Storage device 300 is used to store the trained model.

[0066] Storage device 300 is a device used for storing data. Storage devices can be implemented using volatile or non-volatile storage media. Volatile storage media refers to storage media in which data is lost after a power outage or power failure, including random access memory (RAM), dynamic random access memory (DRAM), and the cache / memory of AI chips. Non-volatile storage media refers to storage media that can retain data for a long time, even without power, including hard drives, solid-state drives (SSDs), and flash memory.

[0067] In a specific application scenario, the storage device 300 is specifically used to receive model acquisition requests from the terminal device 100, and send the trained model to the terminal device 100 according to the model acquisition request. Then, the user can use the trained model to perform inference tasks through the terminal device 100.

[0068] See Figure 2 , Figure 2 This is a schematic diagram of a training model provided in an embodiment of this application. For example... Figure 2 As shown, in a scenario where the model undergoes a total of N iterations, the training task includes the first training operation, the second training operation, the third training operation, ..., the Xth training operation, ... The first training operation is used to implement the model's N1th iterations, specifically from the first to the N1th iteration; the second training operation is used to implement the model's N2th iterations, specifically from the N1+1th to the N1+N2th iteration; the third training operation is used to implement the model's N3th iterations, specifically from the N1+N2+1th to the N1+N2+N3th iteration; ...; the Xth training operation is used to implement the model's N... X The N1+N2+N3+……+N iterations are specifically used to implement the model's N1+N2+N3+……+Nth iterations. X-1 +1 iterations to the N1+N2+N3+……+Nth generation X-1 +N X The next iteration; ... N1 + N2 + N3 + ... + N X +……=N, where X is a positive integer and N0=0.

[0069] Then, the computing device 200 uses the dataset to perform training tasks on the model, specifically by performing the first training operation, the second training operation, the third training operation, ..., and the Xth training operation on the model using the dataset. More specifically, the computing device 200 performs the Xth training operation by inputting the dataset into the N1+N2+N3+...+Nth training operation. X-1 The model obtained in the N1+N2+N3+……+Nth iteration is then used for the N1+N2+N3+……+Nth iteration. X-1 +1 iterations, to obtain the model at the N1+N2+N3+……+Nth step.X-1 The calculation result of the +1 iteration is used to determine the N1+N2+N3+……+Nth iteration. X-1 The model obtained from +1 iterations; then the dataset is input into the N1+N2+N3+……+Nth iteration. X-1 The model obtained from the +1st iteration is used for the N1+N2+N3+……+Nth iteration. X-1 After +2 iterations, the calculation result of the model in the N1+N2+2th iteration is obtained, and then the N1+N2+N3+……+Nth iteration is determined based on this calculation result. X-1 The model obtained after +2 iterations; ...; until the dataset is input into the model at the N1+N2+N3+...+Nth iteration. X-1 +N X The model obtained from the -1st iteration is used for the N1+N2+N3+……+Nth iteration. X-1 +N X In the N1+N2+N3+……+N iteration, the model is obtained. X-1 +N X The calculation result of the N1+N2+N3+……+N iterations is then used to determine the N1+N2+N3+……+Nth iteration. X-1 +N X The model obtained in the first iteration. Thus, the computing device 200 completes the Xth training operation. Optionally, the computing device 200 performs the first iteration in the first training operation by inputting the dataset into the initialized model, performing the first iteration, obtaining the calculation result of the model in the first iteration, and then determining the model obtained in the first iteration based on the calculation result.

[0070] It's understandable that in a scenario where the model undergoes a total of N iterations, when the training task only includes the first training operation, N1 = N. However, when N1 ≠ N, then N1 < N, N2 < N, N3 < N, ..., N. X <N.

[0071] When there are multiple computing devices (200), these computing devices jointly train the model, including at least the following three implementation methods:

[0072] Implementation Method 1: Different training operations are performed on different computing devices.

[0073] Implementation Method 2: All computing devices work together to perform the same training operation.

[0074] Implementation Method 3: Divide all computing devices into multiple groups. Computing devices in the same group perform the same training operation, while computing devices in different groups perform different training operations. Any two groups of computing devices can be completely identical, completely different, or not completely identical.

[0075] It should be understood that the above-mentioned methods of jointly training the model by the computing devices 200 (including implementation methods 1 to 3) are merely examples. In actual applications, the method of training the model by the computing devices 200 can be set as needed according to the actual application scenario, and this application does not impose any specific limitations.

[0076] The above text combined Figure 1 The system architecture of the training model provided in the embodiments of this application has been introduced. Next, a model storage method provided in the embodiments of this application will be described. It should be noted that the application scenarios of the model storage method provided in the embodiments of this application are not limited to those described above. Figure 1 The system architecture for training models and the scenarios in which the model storage method provided in the embodiments of this application can be applied are all within the protection scope of this application.

[0077] See Figure 3 , Figure 3 This is a flowchart illustrating a model storage method provided in an embodiment of this application. Figure 3 As shown, the model storage method provided in this application includes:

[0078] S101: The terminal device sends a dataset to the computing device.

[0079] Accordingly, the computing device receives the dataset from the terminal device.

[0080] Among them, the terminal device can be Figure 1 The terminal device 100 in the middle, the computing device can be Figure 1 The computing device 200 in the middle.

[0081] The dataset consists of multiple samples. Each sample serves as input data for the model. The model can compute the inference result for each sample. The specific content of the samples and the type of model vary across different application domains. For example, in natural language processing, the sample can be a question, and the model can be a large language model. By inputting each question into the large language model, the model's answer to each question can be obtained as the sample's inference result. Alternatively, in image recognition, the sample can be an image, and the model can be a convolutional neural network. By inputting each image into the convolutional neural network, the network's classification result for each image can be obtained as the inference result.

[0082] S102: The computing device uses the dataset to perform a training task on the model, which includes one or more training operations.

[0083] In some potential application scenarios, the model is a deep learning model, specifically a machine learning model based on deep neural network models and methods. The model includes two basic layer structures: linear layers and non-linear layers. The main function of a linear layer is to perform a linear transformation on the input data and output the transformed data. The linear transformation is achieved through a linear function, and its mathematical expression is: Y = AW + B, where Y is the output data, A is the activation value, W is the weight matrix, and B is the bias vector. As can be seen, the linear layer requires matrix inner product operations on the input data. The main function of a non-linear layer is to perform a non-linear transformation on the input data and output the transformed data. Non-linear transformations are achieved through non-linear functions, such as activation functions. Common activation functions include ReLU, Sigmoid, and Tanh.

[0084] Typically, a model consists of an input layer, one or more hidden layers, and an output layer. Therefore, a layer in a model can refer to an input layer, a hidden layer, or an output layer. The input layer is the first layer of the model, primarily responsible for receiving input data (i.e., samples from the dataset), converting the raw format of the input data (such as images, text, audio, etc.) into a format that the model can process (such as numerical vectors), and then passing it to subsequent hidden layers for processing. The number of nodes in the input layer is usually the same as the feature dimension of the input data, with each node corresponding to a feature of the input data. The hidden layers mainly extract and transform features from the input data, providing higher-level features for the output layer. Hidden layers include linear layers and non-linear layers. The number of linear and non-linear layers in the hidden layers can be adjusted according to the specific design of the model and the inference task. Of course, a hidden layer can also contain only linear layers without non-linear layers, or vice versa. The output layer is the last layer of the model, primarily responsible for generating the inference result. The structure and function of the output layer depend on the type of inference task. Different models can be constructed to meet the needs of different users by using different numbers of hidden layers, different linear and nonlinear layers within a single hidden layer, and different connection methods between multiple layers. Different linear layers are characterized by different linear functions, such as different weight matrices W and bias vectors B. Different nonlinear layers are characterized by different nonlinear functions, such as different activation functions.

[0085] Building upon this, the model can be represented using a combination of linear and nonlinear functions. Furthermore, linear functions can be represented using weight matrices and bias vectors. Since nonlinear functions do not change during model training, model training essentially involves continuously adjusting the weight matrices and bias vectors in the hidden layers to obtain weight matrices and bias vectors with definite values ​​to represent the trained model.

[0086] Before training the model, the computing device initializes the model. The parameters that need to be configured include, but are not limited to, the following: model architecture; model structure parameters (including the number of layers, layer structure, and connection methods between layers); training parameters (including the total number of iterations, batch size, optimizer type, and initial values ​​of optimizer states); initial weight values, initial bias values, and activation functions for each layer in the model; hyperparameters (such as regularization parameters, dropout ratio, etc.), etc., thus obtaining the initialized model. This process can be referenced from the initialization process of deep learning models such as large language models and convolutional neural networks; for the sake of brevity, it will not be elaborated further here.

[0087] After obtaining the initialized model, the computing device uses the dataset to perform a training task on the initialized model. In scenarios where the training task includes multiple training operations, the computing device uses the dataset to perform the first training operation, the second training operation, the third training operation, ..., and the Xth training operation on the model. The first training operation includes N1 iterations, specifically from the first iteration to the N1st iteration; the second training operation includes N2 iterations, specifically from the N1+1th iteration to the N1+N2th iteration; the third training operation includes N3 iterations, specifically from the N1+N2+1th iteration to the N1+N2+N3th iteration; ...; the Xth training operation includes N... X The iteration is specifically the N1+N2+N3+……+Nth iteration. X-1 +1 iterations to the N1+N2+N3+……+Nth generation X-1 +N X The next iteration. N1 + N2 + N3 + ... + N X +……=N, where X is a positive integer, N0=0, and N is the total number of iterations.

[0088] Specifically, the computing device performs the first training operation, which involves: inputting the dataset into the initialized model, performing the first iteration on the model, and obtaining the calculation result of the model in the first iteration; then inputting the dataset into the calculation result of the model in the first iteration, performing the second iteration on the model, and obtaining the calculation result of the model in the second iteration; then inputting the dataset into the calculation result of the model in the second iteration, performing the third iteration on the model, and obtaining the calculation result of the model in the third iteration; and so on; until the dataset is input into the calculation result of the model in the (N1-1)th iteration, performing the N1th iteration on the model, and obtaining the calculation result of the model in the N1th iteration. Thus, the computing device completes the first training operation.

[0089] The computing device performs the second training operation, specifically as follows: The dataset is input into the model's calculation result from the N1th iteration, and the model is iterated for the (N1+1)th iteration to obtain the model's calculation result for the (N1+1)th iteration; then the dataset is input into the model's calculation result from the (N1+1)th iteration, and the model is iterated for the (N1+2)th iteration to obtain the model's calculation result for the (N1+2)th iteration; ...; until the dataset is input into the model's calculation result from the (N1+N2-1)th iteration, and the model is iterated for the (N1+N2)th iteration to obtain the model's calculation result for the (N1+N2)th iteration. Thus, the computing device completes the second training operation.

[0090] The computing device performs the third training operation, specifically as follows: The dataset is input into the model's calculation result from the N1+N2th iteration, and the model is iterated for the N1+N2+1th iteration to obtain the model's calculation result for the N1+N2+1th iteration; then the dataset is input into the model's calculation result from the N1+N2+1th iteration, and the model is iterated for the N1+N2+2th iteration to obtain the model's calculation result for the N1+N2+2th iteration; ...; until the dataset is input into the model's calculation result from the N1+N2+N3-1th iteration, and the model is iterated for the N1+N2+N3th iteration to obtain the model's calculation result for the N1+N2+N3th iteration. Thus, the computing device completes the third training operation.

[0091] ...

[0092] And so on, the computing device performs the Xth training operation, specifically by inputting the dataset into the model at the N1+N2+N3+……+Nth training operation. X-1 The calculation results of the N1+N2+N3+……+N iterations are used to apply the model to the N1+N2+N3+……+N iterations. X-1 +1 iterations, to obtain the model at the N1+N2+N3+……+Nth step. X-1 The calculation results of +1 iteration; then input the dataset into the model at the N1+N2+N3+……+N iteration. X-1 The calculation results of +1 iteration are used to perform N1+N2+N3+……+N iterations on the model. X-1 +2 iterations, to obtain the calculation result of the model in the N1+N2+2th iteration; ...; until the dataset is input into the model in the N1+N2+N3+...+Nth iteration. X-1 +N X The calculation results of the -1st iteration are used to perform N1+N2+N3+……+Nth iterations on the model. X-1 +N X In the N1+N2+N3+……+N iteration, the model is obtained. X-1 +N X The calculation result of the Xth iteration. Thus, the computing device has completed the Xth training operation.

[0093] It's understandable that in a scenario where the model undergoes a total of N iterations, when the training task only includes the first training operation, N1 = N. However, when N1 ≠ N, then N1 < N, N2 < N, N3 < N, ..., N. X <N.

[0094] In summary, it can be seen that the process of each iteration in each training operation implemented by the computing device is similar. In each iteration, the dataset is input into the model obtained from the previous iteration, the model is processed for the current iteration, and the computational result of the model in the current iteration is obtained to represent the model obtained in the current iteration. For the sake of brevity, the following text focuses on the specific process of performing one iteration of the model (referred to as the t-th iteration).

[0095] In some possible implementations, the computing device performs the t-th iteration on the model, including implementing forward propagation and backward propagation of the model during the t-th iteration.

[0096] (1) Forward propagation

[0097] In some possible implementations, the computing device performs forward propagation of the model during the t-th iteration to obtain the model's loss in the t-th iteration. Forward propagation mainly includes the following computational steps:

[0098] Calculation step 11: The computing device obtains the model obtained in the (t-1)th iteration.

[0099] If the (t-1)th iteration is the first iteration out of N iterations, then the computing device acquires the initialized model, including the initial values ​​of the model parameters and the initial values ​​of the optimizer state. The model parameters represent the model. The model parameters include the weight matrix and the bias vector; therefore, the initial values ​​of the model parameters include the initial values ​​of the weight matrix, the initial values ​​of the bias vector, etc. The optimizer state is used to update the model parameters.

[0100] If the (t-1)th iteration is not the first iteration in N iterations, then the computing device obtains the model obtained in the (t-1)th iteration. The model obtained in the (t-1)th iteration includes the computational results of the model in the (t-1)th iteration. These computational results include model parameters and optimizer states. Model parameters include the weight matrix and bias vector. Since the process of determining the computational results of the model in the (t-1)th iteration is similar to the process of determining the computational results of the tth iteration, the process of the computing device performing the tth iteration to obtain the computational results of the tth iteration can be referred to in the following description. Furthermore, for the sake of brevity, the weight matrix will be simply referred to as "weight" and the bias vector as "bias" in the following text.

[0101] Specifically, let's assume the model has H layers, each of which can be represented by weights, biases, and activation functions. Since weights and biases change during model training, while activation functions do not, the aforementioned model parameters include the weights and biases of each layer in the model. Let's call the H layers of the model layer 1, layer 2, layer 3, ..., layer H, respectively. Then, the weights of each layer can also be called weight W1, weight W2, weight W3, ..., weight W... H The bias of each layer can also be referred to as bias B1, bias B2, bias B3, ..., bias B. H Among them, the weight W H With bias B H Correspondingly, i is a positive integer. The smaller i is, the closer the i-th layer is to the input layer of the model; the larger i is, the closer the i-th layer is to the output layer of the model.

[0102] Specifically, the optimizer states mentioned above include the learning rate, momentum term, and momentum coefficient. The learning rate controls the speed at which model parameters are updated. Optionally, the learning rate is dynamically adjusted during training, meaning different learning rates can be set for different iterations. The momentum term stores historical gradient information to accelerate model parameter updates. The momentum coefficient balances momentum information at different time steps, allowing for smoother model parameter updates.

[0103] Calculation step 12: The computing device inputs each sample in the dataset into the model obtained in the (t-1)th iteration to obtain the inference result of the sample in the tth iteration.

[0104] The computing device inputs the sample into the model obtained in the (t-1)th iteration. The sample passes through each layer of the model, and the output result (also called the activation value) of each layer is calculated through a weighted sum activation function, ultimately yielding the inference result of the sample. Specifically:

[0105] For the weighted input z of the i-th layer in the model in the t-th iteration i (t), the calculation formula is as follows:

[0106] z i (t)=A i-1 (t)·W i (t)+B i (t) Formula (1)

[0107] For the activation value A of the i-th layer in the model during the t-th iteration i (t), the calculation formula is as follows:

[0108] A i (t)=f(z i Formula (2)

[0109] In the above formulas (1) and (2), A i-1 (t) represents the activation value of the (i-1)th layer in the model during the t-th iteration. i The initial value of W(t) is A1(t), where A1(t) represents the activation value of the first layer (i.e., the input layer) in the model during the t-th iteration. i (t) represents the weights of the i-th layer in the model during the t-th iteration, B i (t) represents the bias of the i-th layer in the model in the t-th iteration, and f() represents the activation function. The specific structure and parameters of f() depend on the type and structure of the model.

[0110] After calculating the activation values ​​of each layer in the model, the computing device will use the activation value A of the H-th layer (i.e., the output layer) in the t-th iteration. H (t) is the reasoning result of the sample in the t-th iteration.

[0111] In different application domains, the specific content of the samples differs, the type of model differs, and the specific content of the inference results also differs. These mainly include the following two scenarios:

[0112] Scenario 1: The model is a supervised learning model

[0113] When the model is a supervised learning model, the inference result of a sample refers to the predicted label of the sample. For example, in the field of natural language processing, the sample can be a question, and the model can be a large language model. By inputting each question into the large language model, the answer of the large language model to each question can be obtained as the inference result. Or, in the field of image recognition, the sample can be an image, and the model can be a convolutional neural network. By inputting each image into the convolutional neural network, the classification result of the convolutional neural network for each image can be obtained as the inference result.

[0114] Scenario 2: The model is an unsupervised learning model

[0115] When the model is an unsupervised learning model, the inference results for samples differ across models. We will illustrate this with examples of autoencoders, generative adversarial networks (GANs), and clustering models. When the model is an autoencoder, the samples are the original data. Each original data point is input into the autoencoder, resulting in a low-dimensional compressed representation of the original data, which serves as the inference result. This inference result can be used to reconstruct the original data. When the model is a GAN, the samples are the original data. Each original data point is input into the GAN, resulting in target data generated by the GAN that is similar to the original data, which serves as the inference result. A GAN consists of a generator and a discriminator. The generator generates target data, while the discriminator distinguishes between the original and target data. The generator's goal is to generate target data similar to the original data, making it difficult for the discriminator to differentiate between them. When the model is a clustering model, the samples are the original data. A dataset containing multiple samples is input into the clustering model, resulting in the cluster label for each sample, which serves as the inference result.

[0116] It should be understood that the models in scenarios 1 and 2 of the above calculation step 12 are merely examples. In practical applications, the models can be of other types, and this application does not impose any specific limitations.

[0117] Calculation step 13: The computing device calculates the model loss in the t-th iteration based on the inference results and actual results of the samples in the t-th iteration.

[0118] The actual results of the samples depend on the type of model.

[0119] Corresponding to scenario 1 in calculation step 12 above, the actual result of a sample refers to the correct label of the sample. For example, in the field of natural language processing, when the sample is a question, the correct label of the sample can be the correct answer to the question. Or, in the field of image recognition, when the sample is an image, the correct label of the sample can be the correct category of the image. Therefore, when the model is a supervised learning model, the dataset includes not only multiple samples but also the correct label of each sample, so the computing device can directly obtain the correct label of each sample from the dataset as the actual result of the sample.

[0120] Corresponding to scenario 2 in calculation step 12 above, the actual results of the samples differ across different models. Let's continue using autoencoders, generative adversarial networks (GANs), and clustering models as examples of unsupervised learning models. When the model is an autoencoder, the actual result of the sample is a low-dimensional representation of the sample. When the model is a GAN, the actual result of the sample is the sample itself. When the model is a clustering model, the actual result of the sample is the center vector of the category to which the sample belongs. Therefore, when the model is an unsupervised learning model, the way the computing device obtains the actual result of the sample varies depending on what the actual result of the sample represents.

[0121] It should be understood that when the model is of other types, the actual results of the samples need to be determined according to the type of model, and this application does not make specific limitations.

[0122] After obtaining the inference results of the samples in the t-th iteration, the computing device calculates the model loss in the t-th iteration based on the inference results and actual results of the samples in the t-th iteration. The calculation formula is as follows:

[0123]

[0124] Where z represents the actual result of the sample; Let L(t) represent the inference result of the sample in the t-th iteration; let Loss(t) represent the loss of the model in the t-th iteration; L() is the loss function, and the specific structure of L() is determined by the user. For example, L() can be a 0-1 loss function, a squared loss function, an absolute loss function, a log loss function, a cross-entropy loss function, etc. As can be seen from formula (3), the input data of the loss function is the inference result of the sample. And the actual result z.

[0125] It should be noted that the above formula (3) only shows the way to calculate the loss of the model in the t-th iteration using a single sample. In practical applications, the loss of the model in the t-th iteration can also be calculated using a portion of the samples in the dataset, or the loss of the model in the t-th iteration can be calculated using all the samples in the dataset. This application does not make any specific limitations.

[0126] (2) Backpropagation

[0127] In some possible implementations, the computing device performs backpropagation of the model during the t-th iteration to obtain the computational result of the model in the t-th iteration. Backpropagation mainly includes the following computational steps:

[0128] Calculation step 21: The computing device calculates the weight gradient and bias gradient of the model in the t-th iteration based on the model loss in the t-th iteration.

[0129] The computing device uses the chain rule to calculate the weight gradient and bias gradient of each layer in the model, propagating backward from the output layer to the input layer, as follows:

[0130] For the activation value A of the i-th layer in the model during the t-th iteration i The gradient of (t) is calculated using the following formula:

[0131]

[0132] For the weight W of the i-th layer in the model in the t-th iteration i The gradient of (t) is calculated using the following formula:

[0133]

[0134] For the bias B of the i-th layer in the model in the t-th iteration i The gradient of (t) is calculated using the following formula:

[0135]

[0136] In formulas (4) to (6) above, δ i (t) represents the activation value A of the i-th layer in the model during the t-th iteration. i The gradient of (t), δ i+1 (t) represents the activation value A of the (i+1)th layer in the model during the t-th iteration. i+1 The gradient of (t), δ i The initial value of (t) is δ H (t), δ H (t) represents the activation value A of the H-th layer (i.e., the output layer) in the model during the t-th iteration. H The gradient of (t), and δ H (t) is also the error of the output layer. f′(z H Loss(t) represents the model loss in the t-th iteration, A H z(t) represents the activation value of the H-th layer in the model during the t-th iteration. H (t) represents the weighted input of the H-th layer in the model during the t-th iteration, f′() represents the derivative of the activation function f(), and W i+1 (t) represents the weights of the (i+1)th layer in the model during the t-th iteration. Indicates weight W i+1 The transpose of (t), z i (t) represents the weighted input of the i-th layer in the model during the t-th iteration, W i (t) represents the weights of the i-th layer in the model during the t-th iteration, A i-1 (t) represents the activation value of the (i-1)th layer in the model during the t-th iteration. Indicates activation value A i-1 The transpose of (t), B i (t) represents the bias of the i-th layer in the model during the t-th iteration.

[0137] It should be noted that the above calculation methods for weight gradients and bias gradients are merely examples. In practical applications, the calculation methods for weight gradients and bias gradients are related to the calculation method of the model's loss in the current iteration. When the calculation method of the model's loss in the current iteration changes, the calculation methods for weight gradients and bias gradients also change accordingly. This application does not impose any specific limitations.

[0138] Calculation step 22: The computing device uses the weight gradient and bias gradient of the model in the t-th iteration to update the model parameters, thereby determining the model obtained in the t-th iteration and obtaining the calculation result of the model in the t-th iteration.

[0139] The process by which a computing device updates model parameters using weight gradients and bias gradients depends on the optimizer within the device. The optimizer is a tool used to optimize model parameters. During model training, the optimizer specifically calculates optimizer states that influence model iterations and outputs based on weight gradients and bias gradients, and then uses these optimizer states to update model parameters. Different optimizers require different optimizer states to be calculated, and the process of using these states also differs. Once the optimizer is determined, the process by which the computing device updates model parameters based on weight gradients and bias gradients is also fixed.

[0140] The following uses Stochastic Gradient Descent (SGD) with Momentum, Adaptive Moment Estimation (Adam), and Root Mean Square Propagation (RMSprop) as examples of different optimizers to introduce the process by which the computational device updates model parameters based on weight gradients and bias gradients in different optimizers.

[0141] (a)SGD with Momentum

[0142] When the optimizer is SGD with Momentum, the process by which the computing device updates the model parameters using weight gradients and bias gradients is as follows:

[0143] The computing device uses weight W i Gradient update of weight W for (t) i (t-1), the calculation formula is as follows:

[0144]

[0145] W i (t)=W i (t-1)-η(t)·v 1i (t) Formula (8)

[0146] In formulas (7) and (8) above, v 1i (t) represents the cumulative weight gradient from the 1st iteration to the tth iteration, v 1i (t) is also called the momentum term; β represents the momentum coefficient, the value of which is determined by the user; v 1i (t-1) represents the cumulative weight gradient from the 1st iteration to the (t-1)th iteration; The weight W in the t-th iteration is represented by the weight W. i The gradient of (t); W i (t) represents the weights of the i-th layer in the model obtained in the t-th iteration; W i (t-1) represents the weight of the i-th layer in the model obtained in the (t-1)-th iteration; η(t) represents the learning rate in the t-th iteration.

[0147] Computing devices use bias B i Gradient update bias B of (t) i (t-1), the calculation formula is as follows:

[0148]

[0149] B i (t)=B i (t-1)-η(t)·v 2i (t) Formula (10)

[0150] In formulas (9) and (10) above, v 2i (t) represents the cumulative bias gradient from the 1st iteration to the (t-1)th iteration, v 2i (t) is also called the momentum term; β represents the momentum coefficient, the value of which is determined by the user; v 2i (t-1) represents the cumulative bias gradient from the first iteration to the (t-2)th iteration; In the t-th iteration, the bias B is represented. i The gradient of (t); B i (t) represents the bias of the i-th layer in the model obtained in the t-th iteration; B i (t-1) represents the bias of the i-th layer in the model obtained in the (t-1)-th iteration; η(t) represents the learning rate in the t-th iteration.

[0151] After the weights and biases of each layer in the model are updated, the model consisting of multiple updated weights and biases and activation functions can be obtained as the model obtained in the t-th iteration. This model can be applied to the (t+1)-th iteration.

[0152] After the weights and biases of each layer in the model are updated, the calculation result of the model in the t-th iteration can be obtained. The calculation result of the model in the t-th iteration refers to the final output of the model at the end of the t-th iteration. Specifically, the calculation result of the model in the t-th iteration includes the calculation results of each layer of the model. The calculation result of the i-th layer in the model includes the weights W calculated in formula (8). i The terms of (t) and the terms used in formula (10) to calculate the bias B i The terms of (t) specifically include: weight W i (t-1), momentum term v 1i (t), bias B i (t-1), momentum term v 2i (t) and the learning rate η(t). Typically, the weights W... i (t-1) and bias B i The elements in (t-1) are all half-precision floating-point numbers (FP16), and the momentum term v 1i (t) and momentum term v 2i The elements in (t) are all single-precision floating-point numbers (FP32), and the learning rate η(t) is a numerical value.

[0153] Optionally, if the learning rate η(t) remains constant throughout the N iterations (i.e., η(t) = η0), the learning rate η0 can be used as an initialization parameter of the model and stored in the initial configuration, rather than being part of the model's computational results. If the learning rate η(t) changes during the N iterations, then the learning rate η(t) will be included as part of the model's computational results.

[0154] (b)Adam

[0155] When the optimizer is Adam, the process by which the computing device updates the model parameters using weight gradients and bias gradients is as follows:

[0156] The computing device uses weight W i Gradient update of weight W for (t) i (t-1), the calculation formula is as follows:

[0157]

[0158] In the above formulas (11) to (15), The weight W in the t-th iteration is represented by the weight W. iThe gradient of (t); m 1i (t) represents the weight gradient from the 1st iteration to the tth iteration. The first moment estimate (mean), m 1i (t) is also called the first-order momentum (term); β1 represents the momentum coefficient, the value of which is determined by the user; m 1i (t-1) represents the weight gradient from the 1st iteration to the (t-1th iteration). The first moment estimate (mean); m 1i An unbiased estimate of (t), Also known as the first-order momentum (term); v 1i (t) represents the weight gradient from the 1st iteration to the tth iteration. The second moment estimate (variance), v 1i (t) is also called the second-order momentum (term); β2 represents the momentum coefficient, the value of which is determined by the user; v 1i (t-1) represents the weight gradient from the 1st iteration to the (t-1th iteration). The second moment estimate (variance); Indicates v 1i An unbiased estimate of (t), Also known as the second-order momentum (term); W i (t) represents the weights of the i-th layer in the model obtained in the t-th iteration; W i (t-1) represents the weight of the i-th layer in the model obtained in the (t-1)-th iteration; η(t) represents the learning rate in the t-th iteration; ∈ is a constant used to prevent division by zero errors.

[0159] Computing devices use bias B i Gradient update bias B of (t) i (t-1), the calculation formula is as follows:

[0160]

[0161] In the above formulas (16) to (20), In the t-th iteration, the bias B is represented. i The gradient of (t); m 2i (t) represents the bias gradient from the 1st iteration to the tth iteration. The first moment estimate (mean), m 2i (t) is also called the first-order momentum (term); β1 represents the momentum coefficient, the value of which is determined by the user; m 2i (t-1) represents the bias gradient from the 1st iteration to the (t-1)th iteration. The first moment estimate (mean); m 2i An unbiased estimate of (t), Also known as the first-order momentum (term); v 2i (t) represents the bias gradient from the 1st iteration to the tth iteration. The second moment estimate (variance), v 2i (t) is also called the second-order momentum (term); β2 represents the momentum coefficient, the value of which is determined by the user; v 2i (t-1) represents the bias gradient from the 1st iteration to the (t-1)th iteration. The second moment estimate (variance); Indicates v 2i An unbiased estimate of (t), Also known as second-order momentum (term); B i (t) represents the bias of the i-th layer in the model obtained in the t-th iteration; B i (t-1) represents the bias of the i-th layer in the model obtained in the (t-1)-th iteration; η(t) represents the learning rate in the t-th iteration; ∈ is a constant used to prevent division by zero errors.

[0162] After the weights and biases of each layer in the model are updated, the model consisting of multiple updated weights and biases and activation functions can be obtained as the model obtained in the t-th iteration. This model can be applied to the (t+1)-th iteration.

[0163] After the weights and biases of each layer in the model are updated, the calculation result of the model in the t-th iteration can be obtained. Specifically, the calculation result of the model in the t-th iteration includes the calculation result of each layer of the model. Among them, the calculation result of the i-th layer in the model includes the weight W used in formula (15) for calculating the weights. i The terms of (t) and the terms used in formula (20) to calculate the bias B i The terms of (t) specifically include: weight W i (t-1), first-order momentum Second momentum Bias B i (t-1), first-order momentum Second momentum The learning rate η(t) and the constant ∈ . Typically, the weights W i (t-1) and bias B i All elements in (t-1) are FP16, first-order momentum. Second momentum First-order momentum and second momentum All elements in the set are FP32, and the learning rate η(t) and constant ∈ are numerical values.

[0164] Optionally, if the learning rate η(t) remains constant throughout the N iterations (i.e., η(t) = η0), the learning rate η0 can be used as an initialization parameter of the model and stored in the initial configuration, rather than being part of the model's computational results. If the learning rate η(t) changes during the N iterations, then the learning rate η(t) will be included as part of the model's computational results.

[0165] Optionally, since the constant ∈ remains unchanged throughout the total number of iterations N, the constant ∈ can be used as an initialization parameter of the model and stored in the initial configuration of the model, rather than as part of the model's computational results.

[0166] (c)RMSprop

[0167] When the optimizer is RMSprop, the process by which the computing device updates the model parameters using weight gradients and bias gradients is as follows:

[0168] The computing device uses weight W i Gradient update of weight W for (t) i (t-1), the calculation formula is as follows:

[0169]

[0170] In the above formulas (21) and (22), v 1i (t) represents the moving average of the squared weight gradient from the 1st iteration to the tth iteration, v 1i (t) is also called the momentum term; β represents the momentum coefficient, the value of which is determined by the user; v 1i (t-1) represents the moving average of the squared weight gradient from the 1st iteration to the (t-1)th iteration; The weight W in the t-th iteration is represented by the weight W. i The gradient of (t); W i (t) represents the weights of the i-th layer in the model obtained in the t-th iteration; W i (t-1) represents the weight of the i-th layer in the model obtained in the (t-1)-th iteration; η(t) represents the learning rate in the t-th iteration; ∈ is a constant used to prevent division by zero errors.

[0171] Computing devices use bias B i Gradient update bias B of (t) i (t-1), the calculation formula is as follows:

[0172]

[0173] In the above formulas (23) and (24), v 2i (t) represents the moving average of the squared bias gradient from the 1st iteration to the tth iteration, v2i (t) is also called the momentum term; β represents the momentum coefficient, the value of which is determined by the user; v 2i (t-1) represents the moving average of the squared bias gradient from the first iteration to the (t-1)th iteration; In the t-th iteration, the bias B is represented. i The gradient of (t); B i (t) represents the bias of the i-th layer in the model obtained in the t-th iteration; B i (t-1) represents the bias of the i-th layer in the model obtained in the (t-1)-th iteration; η(t) represents the learning rate in the t-th iteration; ∈ is a constant used to prevent division by zero errors.

[0174] After the weights and biases of each layer in the model are updated, the model consisting of multiple updated weights and biases and activation functions can be obtained as the model obtained in the t-th iteration. This model can be applied to the (t+1)-th iteration.

[0175] After the weights and biases of each layer in the model have been updated, the calculation result of the model in the t-th iteration can be obtained. Specifically, the calculation result of the model in the t-th iteration includes the calculation result of each layer of the model. Among them, the calculation result of the i-th layer in the model includes the weight W used in formula (22) for calculating the weights. i The terms of (t) and the terms used in formula (24) to calculate the bias B i The terms of (t) specifically include: weight W i (t-1), momentum term v 1i (t), weight gradient Bias B i (t-1), momentum term v 2i (t), bias gradient The learning rate η(t) and the constant ∈ . Typically, the weights W i (t-1) and bias B i All elements in (t-1) are FP16, and the momentum term v 1i (t), weight gradient momentum term v 2i (t) and bias gradient All elements in the set are FP32, and the learning rate η(t) and constant ∈ are numerical values.

[0176] Optionally, if the learning rate η(t) remains constant throughout the N iterations (i.e., η(t) = η0), the learning rate η0 can be used as an initialization parameter of the model and stored in the initial configuration, rather than being part of the model's computational results. If the learning rate η(t) changes during the N iterations, then the learning rate η(t) will be included as part of the model's computational results.

[0177] Optionally, since the constant ∈ remains unchanged throughout the total number of iterations N, the constant ∈ can be used as an initialization parameter of the model and stored in the initial configuration of the model, rather than as part of the model's computational results.

[0178] It should be understood that the process of updating model parameters based on weight gradients and bias gradients in the above-mentioned SGD with Momentum, Adam and RMSprop is only an example. In actual applications, the process of updating model parameters based on weight gradients and bias gradients varies with different optimizers, and this application does not make any specific limitations.

[0179] Through the examples of the three optimizers above, it can be seen that the different computational results obtained using different optimizers all include model parameters and optimizer states, and the optimizer states are used to update the model parameters. Specifically, in the computational result of the t-th iteration obtained using SGDwithMomentum, the weight W... i (t-1) and bias B i (t-1) are model parameters, and the momentum term v 1i (t) and momentum term v 2i (t) is the optimizer state, and the momentum term v 1i (t) is used to update the weight W i (t-1) to obtain the weight W i (t), momentum term v 2i (t) is used to update bias B i (t-1) to obtain bias B i (t); In the calculation result of the t-th iteration obtained using Adam, the weight W i (t-1) and bias B i (t-1) is the model parameter, first-order momentum. Second momentum First-order momentum and second momentum It is the optimizer state, and the first-order momentum and second momentum Used to update weight W i (t-1) to obtain the weight W i (t), first-order momentum and second momentum Used to update bias B i (t-1) to obtain bias B i (t); In the calculation result of the t-th iteration obtained using RMSprop, the weight W i (t-1) and bias B i (t-1) are model parameters, and the momentum term v 1i(t) and momentum term v 2i (t) is the optimizer state, and the momentum term v 1i (t) is used to update the weight W i (t-1) to obtain the weight W i (t), momentum term v 2i (t) is used to update bias B i (t-1) to obtain bias B i (t). It is understandable that the computation results obtained using other optimizers also include model parameters and optimizer states, and the optimizer states are used to update model parameters.

[0180] Furthermore, the computation results require a large amount of storage space, and the storage space required for the optimizer states in the computation results is greater than the storage space required for the model parameters. The following will continue to illustrate this using SGD with Momentum, Adam, and RMSprop as examples of optimizers.

[0181] The computational results obtained through SGD with Momentum include model parameters and optimizer states. Among these, the model parameters—weights W—are... i (t-1), bias B i (t-1) is stored in FP16 data format, optimizer state—momentum term v 1i (t), momentum term v 2i (t) is stored in FP32 data format, and the momentum term v 1i The matrix dimension and weight W of (t) i The matrix dimensions of (t-1) are the same, and the momentum term v 1i (t) and weight W i For (t-1), store all momentum terms v. 1i (t) The required storage space is the total weight of the storage unit W. i (t-1) requires twice the storage space, while the momentum term v 2i (t) and bias B i The relationship between (t-1) is similar to that of the momentum term v. 1i (t) and weight W i The relationship between (t-1) means that the storage space occupied by the optimizer state is twice that occupied by the model parameters, and the storage space occupied by the calculation results, including the optimizer state and the model parameters, is at least three times that occupied by the model parameters.

[0182] The computational results obtained using Adam include model parameters and optimizer states. Among these, the model parameters are the weights W. i (t-1), bias B i(t-1) is stored in FP16 data format, optimizer state—first-order momentum. Second momentum First-order momentum and second momentum All data are stored using the FP32 data format, first-order momentum Second momentum Matrix dimensions and weights W i The matrix dimensions of (t-1) are the same, and the first-order momentum Second momentum With weight W i For (t-1), then store all first-order momentum. The required storage space is the total weight of the storage unit, W. i Twice the storage space required for (t-1), to store all second-order momentum. The required storage space is the total weight of the storage unit, W. i (t-1) requires twice the storage space, while the first momentum Second momentum With bias B i The relationship between (t-1) is similar to that between first-order momentum. Second momentum With weight W i The relationship between (t-1) means that the storage space occupied by the optimizer state is four times that occupied by the model parameters, and the storage space occupied by the calculation results, including the optimizer state and the model parameters, is at least five times that occupied by the model parameters.

[0183] The calculation results obtained through RMSprop include model parameters, optimizer state, weight gradients, and bias gradients. Among these, the model parameters—weights W—are... i (t-1), bias B i (t-1) is stored in FP16 data format, optimizer state—momentum term v 1i (t), momentum term v 2i (t) are all stored in FP32 data format, and the weight gradients are... Bias gradient All data are stored using the FP32 data format, and the momentum term v 1i (t), weight gradient Matrix dimensions and weights W i The matrix dimensions of (t-1) are the same, and the momentum term v 1i (t), weight gradient With weight W i For (t-1), store all momentum terms v. 1i (t) The required storage space is the total weight of the storage unit W.i Twice the storage space required for (t-1), storing all weight gradients. The required storage space is the total weight of the storage unit, W. i (t-1) requires twice the storage space, while the momentum term v 2i (t), bias gradient With bias B i The relationship between (t-1) is similar to that of the momentum term v. 1i (t), weight gradient With weight W i The relationship between (t-1) is such that the storage space occupied by the optimizer state is twice that occupied by the model parameters, the storage space occupied by the weight gradient and the bias gradient is twice that occupied by the model parameters, and the storage space occupied by the calculation results, including the optimizer state, weight gradient, bias gradient and model parameters, is at least five times that occupied by the model parameters.

[0184] In summary, the more layers a model has, the larger the amount of model parameters and optimizer states obtained in a single iteration, and the larger the storage space required for the computation results. For large models, storing the computation results obtained in a single iteration requires even more storage space.

[0185] S103: The computing device stores the computation results of the model in the first iteration of each training operation.

[0186] After the first iteration of each training operation, the computing device stores the computational results of the model obtained in that iteration. Specifically:

[0187] After completing the first iteration in the first training operation, the computing device acquires and stores the computation result of the model in the first iteration of the first training operation (i.e., the computation result of the model in the first iteration in a total of N iterations).

[0188] After completing the first iteration in the second training operation, the computing device acquires and stores the computation result of the model in the first iteration of the second training operation (i.e., the computation result of the model in the N1+1th iteration in a total of N iterations).

[0189] After completing the first iteration in the third training operation, the computing device acquires and stores the computation result of the model in the first iteration of the third training operation (i.e., the computation result of the model in the N1+N2+1th iteration in a total of N iterations).

[0190] ...

[0191] And so on. After completing the first iteration in the Xth training operation, the computing device acquires and stores the computation result of the model in the first iteration of the Xth training operation (i.e., the model in the N1+N2+N3+...+Nth iteration in a total of N iterations). X-1 (Calculation results of +1 iteration).

[0192] The specific content of the above calculation results depends on the type of optimizer. Continuing with SGD with Momentum, Adam, and RMSprop as examples of optimizers, the calculation results obtained through SGD with Momentum, Adam, or RMSprop all include model parameters and optimizer states. Furthermore, the calculation results obtained through RMSprop also include weight gradients and bias gradients. For details regarding the specific content of model parameters and optimizer states in different calculation results, please refer to the description in step S102 above regarding "different calculation results obtained using different optimizers all include model parameters and optimizer states," which will not be repeated here.

[0193] In some possible implementations, the storage format of the above calculation results can be snapshots, logs, data files, etc.

[0194] In some possible implementations, the computing device stores the calculation results on its hard drive, which ensures that the results are not lost when the computing device is powered off or interrupted.

[0195] In some possible implementations, the computing device stores the computation result of the model's first iteration in any training operation, in the order of execution of the training operations. The following example illustrates this, assuming the computing device has just completed the first iteration of the Xth training operation and needs to store the computation result of the model's first iteration in the Xth training operation. The main scenarios for the computing device storing the computation result of the model's first iteration in the Xth training operation include the following three:

[0196] Case 1: If the computing device is idle, it will start storing the computation results of the first iteration of the model in the Xth training operation.

[0197] Scenario 2: The computing device is currently busy, specifically storing the computation results of the model in the first iteration of the Mth training operation, where the Nth iteration of the Mth training operation... M If the next iteration is the iteration preceding the first iteration in the Xth training operation, then after storing the calculation results of the model in the first iteration in the Mth training operation, the computing device begins to store the calculation results of the model in the first iteration in the Xth training operation.

[0198] Scenario 3: The computing device is currently busy. Specifically, the computing device is storing the calculation results of the first iteration of the model in the Mth training operation. The execution time of the Mth training operation is earlier than that of the Xth training operation. Furthermore, there are training operations M+1, M+2, etc., between the Mth and Xth training operations. In this case, after storing the calculation results of the first iteration of the model in the Mth training operation, the computing device begins storing the calculation results of the first iteration of the model in the M+1 training operation; after storing the calculation results of the first iteration of the model in the M+1 training operation, it begins storing the calculation results of the first iteration of the model in the M+2 training operation; and so on; until it finishes storing the calculation results of the first iteration of the model in the X-1 training operation, it begins storing the calculation results of the first iteration of the model in the Xth training operation.

[0199] It should be understood that cases 1-3 in step S103 above are merely examples and are not specifically limited in this application.

[0200] S104: The computing device stores the intermediate results of multiple iterations of the model in each training operation, in the order of iterations in each training operation.

[0201] Based on the specific process of the computing device performing the t-th iteration on the model in step S102 above, the intermediate result of the model in the t-th iteration can be any one of the following three cases:

[0202] Case 1: The weight gradient and bias gradient of the model in the t-th iteration;

[0203] Case 2: The loss of the model in the t-th iteration;

[0204] Case 3: Input data for the loss function of the model in the t-th iteration.

[0205] Corresponding to case 1 in step S104, the intermediate result includes the weight gradient in formula (5) in step 21 of step S102 above. and the bias gradient in formula (6) Specifically, this includes weight gradients. and bias gradient

[0206] When intermediate results include weight gradients and bias gradient At this time, the amount of data in the intermediate results is less than the amount of data in the calculation results of the model in the t-th iteration in step S102 above. Specifically, the intermediate results—weight gradients Bias gradient The data is stored using the FP32 data format, and the weight gradients are... Matrix dimensions and weights Wi The matrix dimensions of (t-1) are the same, and the weight gradients With weight W i For (t-1), store all weighted gradients. The required storage space is the total weight of the storage unit, W. i (t-1) requires twice the storage space, while the bias gradient With bias B i The relationship between (t-1) is similar to that of the weight gradient. With weight W i The relationship between (t-1) indicates that the storage space occupied by intermediate results is twice that of the model parameters. Compared to the optimization method SGDwithMomentum, where the storage space occupied by the computational results is at least three times that of the model parameters, or the optimization method SGD withMomentum or RMSprop, where the storage space occupied by the computational results is at least five times that of the model parameters, the storage space occupied by intermediate results is significantly less than that of the computational results. This indicates that the amount of data in intermediate results is significantly less than the amount of data in the computational results.

[0207] For case 2 corresponding to step S104, the intermediate result is the loss value Loss(t) in formula (3) in step 13 of step S102 above.

[0208] When the intermediate result is the loss value Loss(t), the amount of data in the intermediate result is less than the amount of data in the model's calculation result at the t-th iteration in step S102 above. Specifically, the intermediate result—the loss value Loss(t)—is a numerical value stored in FP32 data format. Therefore, the storage space occupied by the intermediate result is less than the storage space occupied by the model parameters. Compared to the situation where the storage space occupied by the calculation result is at least three times that of the model parameters when the optimizer is SGD with Momentum, or at least five times that of the model parameters when the optimizer is SGD with Momentum or RMSprop, the storage space occupied by the intermediate result is significantly less than that of the calculation result. This indicates that the amount of data in the intermediate result is significantly less than the amount of data in the calculation result.

[0209] Corresponding to case 3 in step S104, the intermediate results include the actual result z of the sample in formula (3) in calculation step 13 of step S102 above, and the inference result of the sample. Specifically, when the model belongs to scenario 1 in calculation step 12 of step S102 above, since the actual result z of the sample remains unchanged throughout the total number of iterations N, the actual result z of the sample can be used as the initialization parameter of the model and stored in the initial configuration of the model, without needing to save the actual result z of the sample in each iteration. When the model belongs to scenario 2 in calculation step 12 of step S102 above, since the actual result z of the sample may change as the model is trained, it is necessary to save the actual result z of the sample in each iteration.

[0210] When the intermediate result includes both the actual result z of the sample and the inference result of the sample At that time, the amount of data in the intermediate results is less than the amount of data in the calculation results of the model in the t-th iteration in step S102 above. Specifically, the intermediate results are the actual results z of the sample and the inference results of the sample. Data is stored using the FP316 format. Typically, in classification tasks, the actual result z of a sample and the inferred result... The shared storage space does not exceed twice the storage space occupied by the model parameters. Compared to the situation where the storage space occupied by the calculation results is at least three times that of the model parameters when the optimizer is SGD with Momentum, or at least five times that of the model parameters when the optimizer is SGD with Momentum or RMSprop, the storage space occupied by the intermediate results is significantly less than that occupied by the calculation results. This indicates that the amount of data in the intermediate results is significantly less than the amount of data in the calculation results.

[0211] It should be understood that the specific content of the intermediate results in cases 1-3 above is merely an example and is not specifically limited in this application.

[0212] In some possible implementations, the computing device stores the intermediate results of multiple iterations of the model in each training operation, in the order of iterations in each training operation, including at least the following two implementations:

[0213] Implementation method 1: The computing device stores the intermediate results of each iteration of the model in each training operation according to the iteration order in each training operation.

[0214] Implementation Method 2: The computing device stores the intermediate results of each iteration of the model in each training operation, starting from the second iteration, in the order of iterations in each training operation.

[0215] In implementation method 1, the storage operations of the computing device are as follows:

[0216] When the computing device performs the Kth iteration in the first training operation and generates the intermediate result (1,K) of the Kth iteration, the computing device stores the intermediate result (1,K); based on successfully storing the intermediate result (1,K) of the Kth iteration and generating the intermediate result (1,K+1) of the (K+1)th iteration, the computing device stores the intermediate result (1,K+1) of the (K+1)th iteration; the computing device successfully stores the intermediate result (1,K) of the Kth iteration and the intermediate result (1,K+1) of the (K+1)th iteration. Based on generating the intermediate result (1,K+2) of the (K+2)th iteration, the computing device stores the intermediate result (1,K+2) of the (K+2)th iteration; based on successfully storing the intermediate result (1,K) of the Kth iteration, the intermediate result (1,K+1) of the (K+1)th iteration, and the intermediate result (1,K+2) of the (K+2)th iteration, and generating the intermediate result (1,K+3) of the (K+3)th iteration, the computing device stores the intermediate result (1,K+3) of the (K+3)th iteration; ... where K is a positive integer. As the value of K increases, the computing device can sequentially generate and store the intermediate result of the model in the total number of iterations N: the intermediate result of the 1st iteration (i.e., intermediate result (1,1)), the intermediate result of the 2nd iteration (i.e., intermediate result (1,2)), the intermediate result of the 3rd iteration (i.e., intermediate result (1,3)), ..., the intermediate result of the N1th iteration (i.e., intermediate result (1,N1)).

[0217] When the computing device performs the Kth iteration in the second training operation and generates the intermediate result (2,K) of the Kth iteration, the computing device stores the intermediate result (2,K); based on successfully storing the intermediate result (2,K) of the Kth iteration and generating the intermediate result (2,K+1) of the K+1th iteration, the computing device stores the intermediate result (2,K+1) of the K+1th iteration; the computing device successfully stores the intermediate result (2,K) of the Kth iteration and the intermediate result (2,K+1) of the K+1th iteration. Based on generating the intermediate result (2,K+2) of the (K+2)th iteration, the computing device stores the intermediate result (2,K+2) of the (K+2)th iteration; based on successfully storing the intermediate results (2,K), (2,K+1), and (2,K+2) of the (K+1)th iteration, and generating the intermediate result (2,K+3) of the (K+3)th iteration, the computing device stores the intermediate result (2,K+3) of the (K+3)th iteration; ... where K is a positive integer. As the value of K increases, the computing device can sequentially generate and store the intermediate results of the (N1+1)th iteration (i.e., intermediate result (2,1)), the (N1+2)th iteration (i.e., intermediate result (2,2)), ..., the (N1+N2)th iteration (i.e., intermediate result (2,N2)) of the model in a total of N iterations.

[0218] When the computing device performs the Kth iteration in the third training operation and generates the intermediate result (3,K) of the Kth iteration, the computing device stores the intermediate result (3,K); based on successfully storing the intermediate result (3,K) of the Kth iteration and generating the intermediate result (3,K+1) of the K+1th iteration, the computing device stores the intermediate result (3,K+1) of the K+1th iteration; the computing device successfully stores the intermediate result (3,K) of the Kth iteration and the intermediate result (3,K+1) of the K+1th iteration. Based on generating the intermediate result (3,K+2) of the (K+2)th iteration, the computing device stores the intermediate result (3,K+2) of the (K+2)th iteration; based on successfully storing the intermediate results (3,K), (3,K+1), and (3,K+2) of the (K+1)th iteration, and generating the intermediate result (3,K+3) of the (K+3)th iteration, the computing device stores the intermediate result (3,K+3) of the (K+3)th iteration; ... where K is a positive integer. As the value of K increases, the computing device can sequentially generate and store the intermediate results of the model in the N1+N2+1th iteration (i.e., intermediate result (3,1)), the N1+N2+2th iteration (i.e., intermediate result (3,2)), ..., the N1+N2+N3th iteration (i.e., intermediate result (3,N3)) in a total of N iterations.

[0219] ...

[0220] Similarly, when the computing device performs the Kth iteration of the Xth training operation and generates the intermediate result (X,K) of the Kth iteration, the computing device stores the intermediate result (X,K); after successfully storing the intermediate result (X,K) of the Kth iteration and generating the intermediate result (X,K+1) of the K+1th iteration, the computing device stores the intermediate result (X,K+1) of the K+1th iteration; after successfully storing the intermediate result (X,K) of the Kth iteration and the intermediate result (X,K+1) of the K+1th iteration... 1) Based on generating the intermediate result (X,K+2) of the (K+2)th iteration, the computing device stores the intermediate result (X,K+2) of the (K+2)th iteration; After successfully storing the intermediate results (X,K) of the Kth iteration, the (K+1)th iteration, and the (K+2)th iteration, and generating the intermediate result (X,K+3) of the (K+3)th iteration, the computing device stores the intermediate result (X,K+3) of the (K+3)th iteration; ... where K is a positive integer. As the value of K increases, the computing device can sequentially generate and store the N1+N2+N3+...+Nth iterations of the model in a total of N iterations. X-1 The intermediate result of the +1th iteration (i.e., intermediate result (X,1)), the N1+N2+N3+……+Nth iterationX-1 Intermediate results of +2 iterations (i.e., intermediate result (X,2)), ..., N1+N2+N3+...+N X-1 +N X The intermediate result of the second iteration (i.e., the intermediate result (X, N)) X N1+N2+N3+……+N X +……=N, where X is a positive integer and N0=0.

[0221] It is understandable that implementation method 1 stores the intermediate results of each iteration of the model in each training operation, starting from the second iteration.

[0222] In implementation method 2, the storage operation of the computing device is specifically as follows: when the computing device executes the Kth iteration in the Xth training operation and generates the intermediate result (X,K) of the Kth iteration, the computing device acquires and stores the intermediate result (X,K). Based on successfully storing the intermediate result (X,K) of the Kth iteration and generating the intermediate result (X,K+1) of the K+1th iteration, the computing device stores the intermediate result (X,K+1) of the K+1th iteration. The computing device generates intermediate results (X,K+1) for the K+2th iteration and stores the intermediate results (X,K+2) for the K+2th iteration. After successfully storing the intermediate results (X,K) for the Kth iteration, the K+1th iteration, and the K+2th iteration, and generating the intermediate results (X,K+3) for the K+3th iteration, the computing device stores the intermediate results (X,K+3) for the K+3th iteration. ... where K≥2, and K is an integer. As the value of K increases, the computing device can sequentially generate and store the N1+N2+N3+...+Nth iterations of the model in a total of N iterations. X-1 +2 intermediate results (i.e., intermediate result (X,2)), N1+N2+N3+……+N X-1 Intermediate results of +3 iterations (i.e., intermediate result (X,3)), ..., N1+N2+N3+...+N X-1 +N X The intermediate result of the second iteration (i.e., the intermediate result (X, N)) X N1+N2+N3+……+N X +……=N, where X is a positive integer and N0=0.

[0223] It should be understood that the above-described implementation methods 1 and 2 for storing intermediate results are merely examples, and this application does not impose any specific limitations. In practical applications, any implementation method that can store the intermediate results of multiple iterations of the model in each training operation according to the iteration order in each training operation is within the scope of protection of this application.

[0224] In some possible implementations, the storage format for the intermediate results can be snapshots, logs, data files, etc. When using snapshots to store intermediate results, the computing device can perform a snapshot operation when generating the intermediate result (X,K) of the Kth iteration in the Xth training operation, generating a snapshot (X,K) and storing it, thus achieving the storage of the intermediate results. When using logs to store intermediate results, after generating the intermediate result (X,K) of the Kth iteration in the Xth training operation, the computing device can obtain the intermediate result (X,K) and then write it to the log, thus achieving the storage of the intermediate results. When using data files to store intermediate results, after generating the intermediate result (X,K) of the Kth iteration in the Xth training operation, the computing device can obtain the intermediate result (X,K) and then write it to the data file, thus achieving the storage of the intermediate results.

[0225] It should be noted that when using snapshots, logs, or data files to store intermediate results, the storage speed for a single iteration may be slow due to factors such as the read / write performance of the computing device and the size of the snapshot (or log, data file). This can lead to the computing device generating intermediate results for subsequent iterations before the current iteration's intermediate results are fully stored. In such cases, after storing the current intermediate result, the computing device begins storing the intermediate results generated in the next iteration; after storing the intermediate results of the next iteration, it begins storing the intermediate results generated in subsequent iterations. This ensures that the intermediate results of the model across multiple iterations are stored in the order of iteration.

[0226] It should be understood that the above-mentioned use of snapshots, logs or data files to store intermediate results is merely an example. In practical applications, other formats can also be used to store intermediate results, and this application does not impose any specific limitations.

[0227] In summary, given the determined storage format of the intermediate results, the computing device uses implementation method 1 of step S104 to store the intermediate results. This essentially means storing the intermediate results from the Nth training operation of the model during the Mth training operation. M Based on the intermediate results of the previous iteration, the intermediate results of the model in the Pth iteration of the Xth training operation are stored. Furthermore, the computing device stores the intermediate results of the model in the Nth iteration of the Mth training operation. MBased on the intermediate results of the nth iteration and the intermediate results of the model in the Pth iteration of the Xth training operation, the intermediate results of the model in the (P+1)th iteration of the Xth training operation are then stored. Wherein, in the Mth training operation, the Nth... M The next iteration is the iteration preceding the first iteration in the Xth training operation, where P ≥ 1, and P is an integer. This allows the computing device to store the intermediate results of the model in the first iteration of the Xth training operation. If a failure occurs in subsequent training, since the Nth iteration in the Mth training operation... M The iteration preceding the first iteration in the Xth training operation can be used to model the Nth iteration in the Mth training operation. M The calculation result of the first iteration in training operation X is obtained by combining the calculation result of the second iteration with the intermediate result of the first iteration in training operation X. Then, the intermediate results of each iteration starting from the second iteration in training operation X can be used to reconstruct the calculation result of the corresponding iteration. If the intermediate result of the first iteration in training operation X is not stored, then the intermediate result of the Nth iteration in training operation M needs to be used. M The calculation results of the second iteration are used to perform a complete iteration of the model to obtain the model's performance in N. X The calculation result of the first iteration in the Xth iteration. Therefore, compared to not storing the intermediate result of the model in the first iteration of the Xth training operation, storing the intermediate result of the model in the first iteration of the Xth training operation helps to further reduce the time and computational cost of recovering the model.

[0228] In some possible implementations, the computing device stores intermediate results on its hard drive, which ensures that the results are not lost when the computing device is powered off or interrupted.

[0229] In some possible implementations, when a computing device needs to store the computation results and intermediate results of a model in different iterations, the computing device can utilize its parallel storage capability to simultaneously store the computation results and intermediate results belonging to different iterations.

[0230] In some possible implementations, when a computing device needs to store the computational results and intermediate results of a model in the same iteration, the computing device can utilize its parallel storage capability to store the computational results and intermediate results of the same iteration simultaneously.

[0231] In some possible implementations, if the computing device successfully stores the computational results of the model in the first iteration of the Xth training operation, the computing device deletes its stored computational results of the model in the first iteration of the Mth training operation, as well as the intermediate results of each iteration from the second iteration onwards (including the results from the second iteration to the Nth iteration). M(Intermediate results of the Nth iteration). Among them, the Nth iteration in the Mth training operation... M The execution time of this iteration is earlier than the execution time of the first iteration in the Xth training operation. If the intermediate results are stored using implementation method 1 in step S104 above, the computing device also stores the intermediate results of the model's first iteration in the Mth training operation. Therefore, the computing device also deletes the stored intermediate results of the model's first iteration in the Mth training operation. When the computing device stores new computational results, if a subsequent training failure occurs, the model can be recovered directly from the new computational results without having to recover from the old computational results. Therefore, this technical solution supports deleting the stored old computational results and the intermediate results of multiple iterations between the old and new computational results, based on the storage of new computational results. This can save the storage space of the computing device and reduce storage costs.

[0232] It should be noted that when the computing device used to train the model (corresponding to the aforementioned...) Figure 1 When there are multiple computing devices (200) in a model, these devices are typically used to train different linear layers of the model. In a single complete iteration, each computing device generates the weights and biases of one or more linear layers in the model, and naturally, also generates weight gradients and bias gradients. Because there are data dependencies between the multiple linear layers in the model—including the input data of the subsequent linear layer depending on the output data of the previous linear layer during forward propagation, and the calculation of the weight gradient of the previous linear layer depending on the weight gradient of the subsequent linear layer during backpropagation—it is necessary to transfer relevant model data between the multiple computing devices.

[0233] Taking the example of computing device i being responsible for training the i-th layer of the model, then the t-th iteration of the model by the computing device in step S102 above refers to multiple computing devices jointly implementing the t-th iteration. Among them, computing device i is responsible for generating the weighted input z of the i-th layer of the model in the t-th iteration during the forward propagation process. i (t) and activation value A i (t), and the weight gradient of the i-th layer in the model during the t-th iteration generated during backpropagation. and bias gradient Using weight gradient Bias gradient Update the weights W of the i-th layer in the model i (t-1), bias B i (t-1) to obtain the weight W i (t), bias B i (t).

[0234] In step S103 above, storing the computational results of the model in the first iteration of each training operation by multiple computing devices means that multiple computing devices jointly store the computational results of the model in the first iteration of each training operation. Specifically, computing device i is responsible for storing the weights W of the i-th layer in the model in the first iteration of each training operation. i and bias B i .

[0235] In step S104 above, storing intermediate results of multiple iterations of the model in each training operation by multiple computing devices refers to multiple computing devices jointly storing the intermediate results of multiple iterations of the model in each training operation. When the intermediate results include the model's weight gradients and bias gradients, computing device i is responsible for storing the weight gradients of the i-th layer of the model in different iterations. and bias gradient When the intermediate result is the model's loss, the computing device responsible for calculating the model's loss is responsible for storing the model's loss in different iterations. When the intermediate result is the input data for the model's loss function, the computing device responsible for calculating the model's loss is responsible for storing the input data for the model's loss function in different iterations.

[0236] In summary, the model storage method provided in this application can store the computation result of the first iteration in each training operation and the intermediate results of each iteration starting from the second iteration. Thus, when a model malfunctions during subsequent training, the first computation result from all stored computation results, along with the intermediate results generated in multiple iterations after generating the first computation result, can be used to recover the computation results from those iterations. This recovered computation result can then be used for subsequent model training until the computation result before the malfunction is restored, and training can continue. Compared to existing schemes that periodically save the computation results of the model at different training stages and can only provide the most recently saved computation result to retrain the model to obtain the computation result before the malfunction, this technical solution shortens the computation process for the computation result corresponding to the intermediate result by storing intermediate results. Therefore, this technical solution helps to effectively shorten the model recovery process, reduce the computational and time costs of recovery training, and reduce the time cost of the training task.

[0237] On the other hand, regardless of whether the intermediate results include the weight gradients and bias gradients of the model in the corresponding iteration, the intermediate results are the loss of the model in the corresponding iteration, or the intermediate results are the input data of the loss function of the model in the corresponding iteration, the amount of data in the intermediate results is significantly less than the amount of data in the calculation results. Therefore, compared with the existing schemes that store the calculation results of the model in each iteration, which require a large amount of data storage, this technical solution only needs to store the calculation results of one iteration and the intermediate results of multiple iterations. Since the amount of data in the intermediate results is less than the amount of data in the corresponding calculation results, this technical solution can reduce the amount of data stored, reduce the storage cost of the model, and achieve the goal of breaking down the whole into parts.

[0238] See Figure 4 , Figure 4 This is a schematic flowchart illustrating a model training method provided in an embodiment of this application. This method can resume model training operations in the event of an anomaly. For example, it can resume model training operations after a computing device malfunctions, or after previous training operations have been paused, canceled, or otherwise interrupted. The following will use a computing device malfunction during model training as an example to illustrate the model training method provided in this application. Figure 4 As shown, the model training method provided in this application includes:

[0239] S201: In the event of an anomaly during training, the computing device acquires the model's first calculation result and multiple intermediate results.

[0240] When the computing device used to train the model (corresponding to the aforementioned) Figure 1 When there is only one computing device (200) in the system, if the computing device malfunctions during model training, the training model will become abnormal. In this case, either the computing device can retrieve the first calculation result and multiple intermediate results of the model stored in it after it recovers and execute subsequent steps until the model before the malfunction is restored; or another new computing device can retrieve the first calculation result and multiple intermediate results of the model stored in the malfunctioning computing device and execute subsequent steps until the model before the malfunction is restored.

[0241] When the computing device used to train the model (corresponding to the aforementioned) Figure 1 When there are multiple computing devices (200) in the model, each computing device is responsible for training one or more linear layers in the model. Since multiple computing devices need to transfer the relevant data of the linear layers they are responsible for to complete the training of different linear layers in the same iteration, when any computing device fails and causes the training model to malfunction, the model cannot be recovered by the computing device that failed on its own. It is necessary to use the relevant data of the linear layers stored on other computing devices.

[0242] For example, computing device 1 is used to train linear layer 1, computing device 2 is used to train linear layer 2, and the next linear layer after linear layer 1 is linear layer 2. During backpropagation, when computing device 2 generates the weight gradient of linear layer 2... And use weighted gradients Only after updating the weights W2 of linear layer 2 can computing device 1 obtain and use the updated weights W2 to calculate the weight gradient of linear layer 1. Then use weight gradient Update the weights W1 of linear layer 1 to obtain the updated weights W1.

[0243] Therefore, when any computing device fails, all computing devices stop training the model. They can either wait for the failed computing device to recover and then work together to restore the model before the failure; or a new computing device can obtain the relevant data of the model stored in the failed computing device and work together with the other computing devices to restore the model before the failure.

[0244] For ease of explanation, the computing device that experiences the anomaly will be referred to as the first computing device, and the computing device performing the model recovery operation will be referred to as the computing device. Furthermore, an example will be given using the Xth training operation as the training operation currently being performed by the first computing device, and the Rth iteration within the Xth training operation as the training currently being performed by the first computing device. That is, the following explanation will use the Xth training operation (i.e., Nth iteration) as the training operation currently being performed by the first computing device. X An exception occurs during the Rth iteration of the model (in the nth iteration) as an example scenario. Here, X and R are both positive integers.

[0245] The first computing device in the Xth training operation (i.e. N) X After an anomaly occurs during the R-th iteration of the model (in the nth iteration), the computing device acquires the first calculation result of the model and multiple intermediate results. The first calculation result is the final output of the model at the end of one iteration. The multiple intermediate results are generated in multiple iterations after the generation of the first calculation result, and there is a one-to-one correspondence between the multiple intermediate results and the multiple iterations.

[0246] In some possible implementations, the first computation result can be the result of the first iteration of any training operation stored in the computing device. Three examples illustrate this below:

[0247] Example 11: When the computing device stores the computation result of the model in the first iteration of the Xth training operation, the first computation result can be the computation result of the model in the first iteration of the Xth training operation.

[0248] Example 12: When the computing device stores the computation result of the model in the first iteration of the Mth training operation, the first computation result can be the computation result of the model in the first iteration of the Mth training operation.

[0249] The execution time of the Mth training operation is earlier than that of the Xth training operation, because the Mth training operation is derived from the model's N... M The Xth training operation consists of N iterations of the model. X The iterations are composed of, therefore, N M In each iteration, the execution time of any training iteration is earlier than N. X The execution time of any training iteration in each iteration, where M is a positive integer.

[0250] Example 13: When a computing device stores the calculation results of the model in the first iteration of the Mth training operation and the calculation results of the model in the first iteration of the Xth training operation, the first calculation result can be either the calculation result of the model in the first iteration of the Mth training operation or the calculation result of the model in the first iteration of the Xth training operation.

[0251] Optionally, the first calculation result is the latest calculation result stored in the computing device. That is, when the computing device stores the calculation result of the model in the first iteration of the Mth training operation and the calculation result of the model in the first iteration of the Xth training operation, the calculation result of the model in the first iteration of the Xth training operation is selected as the first calculation result.

[0252] It should be understood that Examples 11-13 above are merely examples and are not intended to be specific limitations in this application.

[0253] For details of the first calculation results in Examples 11-13 above, please refer to the foregoing. Figure 3 The description of the calculation result of the model in step S102 of the model storage method regarding the t-th iteration will not be elaborated here.

[0254] In some possible implementations, multiple intermediate results can be intermediate results generated in multiple iterations after the generation of the first computation result, stored in the computing device. Four examples illustrate this below:

[0255] Example 21: When the first calculation result is the result of the model's first iteration in the Xth training operation, multiple intermediate results include intermediate results from the second to the Qth iteration in the Xth training operation. Here, Q is a positive integer, R > 2, and Q < R.

[0256] It should be noted that when R=1, if the first calculation result is the result of the model's first iteration in the Xth training operation, then the computing device has not yet executed the iterations after generating the first calculation result, and therefore the computing device cannot obtain multiple intermediate results.

[0257] Example 22: When the first calculation result is the result of the model's first iteration in the Mth training operation, multiple intermediate results include intermediate results from the second to the Qth iterations of the model in the Mth training operation. Where Q ≤ N M .

[0258] Example 23: When the first calculation result is the result of the model's calculation in the first iteration of the Mth training operation, and the Mth training operation is the (X-1)th training operation, if the aforementioned... Figure 3 In the model storage method, if step S104, implementation method 1 stores intermediate results, then multiple intermediate results include: the model from the second iteration to the Nth iteration in the Mth training operation. M The intermediate results of the iteration, and the intermediate results of the model from the 1st to the Qth iteration in the Xth training operation; if the aforementioned Figure 3 In the model storage method, step S104, implementation method 2 stores intermediate results. Therefore, multiple intermediate results include: the model's results from the second iteration to the Nth iteration in the Mth training operation. M The intermediate result of the nth iteration. Among them, the Nth iteration in the Mth training operation... M The next iteration is adjacent to the first iteration in the Xth training operation, and the Nth iteration in the Mth training operation... M The execution time of this iteration is earlier than the execution time of the first iteration in the Xth training operation.

[0259] Example 24: When the first calculation result is the result of the model's first iteration in the Mth training operation, and the Mth training operation is not adjacent to the Xth training operation, if the aforementioned method is used... Figure 3 In the model storage method, if step S104, implementation method 1 stores intermediate results, then multiple intermediate results include: the model from the second iteration to the Nth iteration in the Mth training operation. M The intermediate result of the iteration, the model in the (M+1)th training operation from the 1st to the Nth iteration. M+1 The intermediate result of the iteration, the model from the first iteration to the Nth iteration in the (M+2)th training operation. M+2 The intermediate results of the first iteration, ..., and the intermediate results of the model from the first to the Qth iteration in the Xth training operation; if the aforementioned Figure 3 In the model storage method, step S104, implementation method 2 stores intermediate results. Therefore, multiple intermediate results include: the model's results from the second iteration to the Nth iteration in the Mth training operation. M Intermediate results of the next iteration.

[0260] It should be understood that Examples 21-24 above are merely examples and are not intended to be specific limitations in this application.

[0261] S202: The computing device calculates multiple calculation results based on the first calculation result of the model and multiple intermediate results, determines the target calculation result from the multiple calculation results, and then continues to iterate the model based on the target calculation result.

[0262] In some possible implementations, the computing device calculates multiple results based on the first calculation result of the model and multiple intermediate results. Specifically, starting with the q-value corresponding to the iteration that generated the first calculation result, the calculation result of the q-th iteration and the intermediate results of the (q+1)-th iteration are used sequentially, from smallest to largest, to calculate the (q+1)-th iteration result, thus obtaining multiple calculation results that correspond one-to-one with the multiple intermediate results. Here, q is a positive integer.

[0263] The following examples 21-24 of step S202 above illustrate the specific process of calculating multiple calculation results based on the first calculation result and multiple intermediate results.

[0264] In Example 21 corresponding to step S202 above, if the first calculation result is the calculation result of the model in the first iteration of the Xth training operation, and multiple intermediate results include intermediate results from the second iteration to the Qth iteration of the model in the Xth training operation, the q value corresponding to the iteration that generates the first calculation result can be set to 1 to indicate that the qth iteration under different q values ​​is in N X The order of iterations; alternatively, the q value corresponding to the iteration that generates the first calculation result can be set to N1+N2+N3+……+N X-1 +1 is added to indicate the order of the q-th iteration within a total of N iterations for different q values. Next, the result of the q-th iteration and the intermediate result of the (q+1)-th iteration are used to calculate the result of the (q+1)-th iteration. Specifically, the result of the 2nd iteration is calculated using the result of the 1st iteration and the intermediate result of the 2nd iteration in training operation X; the result of the 3rd iteration is calculated using the result of the 2nd iteration and the intermediate result of the 3rd iteration; the result of the 4th iteration is calculated using the result of the 3rd iteration and the intermediate result of the 4th iteration; and so on, until the result of the Q-1-th iteration and the intermediate result of the Q-th iteration are used to calculate the result of the Q-th iteration. Thus, the results of the model from the 2nd to the Qth iterations in training operation X can be obtained. It can be seen that there is a one-to-one correspondence between the results of the 2nd to the Qth iterations and the intermediate results of the 2nd to the Qth iterations.

[0265] In Example 22 corresponding to step S202 above, if the first calculation result is the calculation result of the model in the first iteration of the Mth training operation, and multiple intermediate results include intermediate results from the second iteration to the Qth iteration of the model in the Mth training operation, the q value corresponding to the iteration that generates the first calculation result can be set to 1 to indicate that the qth iteration under different q values ​​is in N M The order of iterations; alternatively, the q value corresponding to the iteration that generates the first calculation result can be set to N1+N2+N3+……+N M-1 +1 is added to indicate the order of the q-th iteration in the total number of iterations N for different q values. Next, the result of the q-th iteration and the intermediate result of the (q+1)-th iteration are used to calculate the result of the (q+1)-th iteration. Specifically, the result of the 2nd iteration is calculated using the result of the 1st iteration and the intermediate result of the 2nd iteration in the M-th training operation; the result of the 3rd iteration is calculated using the result of the 2nd iteration and the intermediate result of the 3rd iteration; the result of the 4th iteration is calculated using the result of the 3rd iteration and the intermediate result of the 4th iteration; and so on, until the result of the Q-1-th iteration and the intermediate result of the Q-th iteration are used to calculate the result of the Q-th iteration. Thus, the results of the 2nd to Qth iterations in the M-th training operation can be obtained. It can be seen that there is a one-to-one correspondence between the results of the 2nd to Qth iterations and the intermediate results of the 2nd to Qth iterations.

[0266] Example 23, corresponding to step S202 above, shows that the first calculation result is the calculation result of the model's first iteration in the Mth training operation, where the Mth training operation is the (X-1)th training operation, and multiple intermediate results include the model's second iteration to the Nth iteration in the Mth training operation. M Given the intermediate results of the iteration and the intermediate results of the model from the 1st to the Qth iteration in the Xth training operation, the q value corresponding to the iteration that generates the first calculation result is set to N1+N2+N3+……+N M-1 +1 is added to indicate the order of the q-th iteration in the total number of iterations N for different q values. Next, the result of the (q+1)-th iteration is calculated using the result of the q-th iteration and the intermediate result of the (q+1)-th iteration. Specifically, the result of the (N1+N2+N3+…+N)-th iteration is calculated using the intermediate result of the (N1+N2+N3+…+N)-th iteration. M-1 The calculation results of +1 iterations (i.e., the calculation results of the model in the first iteration of the Mth training operation) and the calculation results of the N1+N2+N3+……+Nth iterations. M-1 The intermediate results of the +2 iterations are used to calculate the N1+N2+N3+……+Nth iteration. M-1 The calculation results of +2 iterations are then used for the N1+N2+N3+……+Nth iterations. M-1The calculation results of +2 iterations and the N1+N2+N3+……+Nth iterations M-1 The intermediate results of the +3 iterations are used to calculate the N1+N2+N3+……+Nth iteration. M-1 +3 iterations of calculation results, ..., until the N1+N2+N3+...+Nth iteration is used. M The calculation results of the -1st iteration and the N1+N2+N3+……+Nth iteration. M The intermediate results of the next iteration are used to calculate the N1+N2+N3+……+Nth iteration. M The calculation result of the Nth iteration (i.e., the model in the Mth training operation) M The calculation result of the N1+N2+N3+……+N iterations is then used. M The calculation results of the N1+N2+N3+……+N iterations and the Nth iteration. M The intermediate results of the +1 iteration are used to calculate the N1+N2+N3+……+Nth iteration. M The calculation result of the +1 iteration (i.e., the calculation result of the model in the first iteration of the Xth training operation) is then used in the N1+N2+N3+……+Nth iteration. M The calculation results of +1 iteration and the N1+N2+N3+……+Nth iterations M The intermediate results of the +2 iterations are used to calculate the N1+N2+N3+……+Nth iteration. M-1 +2 iterations of calculation results, ..., until the N1+N2+N3+...+Nth iteration is used. M The calculation results of +Q-1 iterations and the N1+N2+N3+……+N M The intermediate results of +Q iterations are used to calculate the N1+N2+N3+……+Nth iteration. M The calculation result of +Q iterations (i.e., the calculation result of the model in the Qth iteration of the Xth training operation). Therefore, the calculation results of the model from the 2nd iteration to the Nth iteration in the Mth training operation can be obtained. M The calculation results of the nth iteration, and the calculation results of the model from the 1st to the Qth iteration in the Xth training operation. It can be seen that the model from the 2nd to the Nth iteration in the Mth training operation... M The calculation results of the iteration are compared with the model in the second iteration to the Nth iteration in the Mth training operation. M There is a one-to-one correspondence between the iterations. The calculation results of the model from the 1st to the Qth iteration in the Xth training operation correspond one-to-one with the model from the 1st to the Qth iteration in the Xth training operation.

[0267] Example 23, corresponding to step S202 above, shows that the first calculation result is the calculation result of the model's first iteration in the Mth training operation, where the Mth training operation is the (X-1)th training operation, and multiple intermediate results include the model's second iteration to the Nth iteration in the Mth training operation.M In the case of intermediate results of the nth iteration, the model is obtained from the second iteration to the Nth iteration in the Mth training operation. M The process of calculating the result of the next iteration is similar to that in Example 22 corresponding to step S202 above, where the first calculation result is the calculation result of the model in the first iteration of the M training operation, and multiple intermediate results include the intermediate results of the model in the second iteration to the Q iteration of the M training operation. For the sake of brevity, it will not be elaborated here.

[0268] Example 24, corresponding to step S202 above, shows that the first calculation result is the calculation result of the model's first iteration in the Mth training operation, where the Mth training operation is not adjacent to the Xth training operation, and multiple intermediate results include the model's second iteration to the Nth iteration in the Mth training operation. M The intermediate results of the next iteration, the model from the first iteration to the Nth iteration in the (M+1)th training operation. M+1 The intermediate results of the next iteration, the model from the first iteration to the Nth iteration in the (M+2)th training operation. M+2 Given the intermediate results of the nth iteration, ..., and the intermediate results of the model from the 1st to the Qth iteration in the Xth training operation, we can obtain the model from the 2nd to the Nth iteration in the Mth training operation. M The calculation results of the iteration, the model from the first iteration to the Nth iteration in the (M+1)th training operation M+1 The calculation results of the next iteration, and the model from the first iteration to the Nth iteration in the (M+2)th training operation. M+2 The process of calculating the results of each iteration, ..., and the calculation results of the model from the first iteration to the Qth iteration in the Xth training operation, is similar to Example 23 corresponding to step S202 above. In Example 23, the first calculation result is the calculation result of the model in the first iteration in the Mth training operation, where the Mth training operation is the (X-1)th training operation. Multiple intermediate results include the calculation results of the model from the second iteration to the Nth iteration in the Mth training operation. M Given the intermediate results of the nth iteration and the intermediate results of the model from the 1st to the Qth iteration in the Xth training operation, we can obtain the model from the 2nd to the Nth iteration in the Mth training operation. M The calculation results of the iteration are similar to those of the model from the first iteration to the Qth iteration in the Xth training operation. For the sake of brevity, they will not be elaborated here.

[0269] The following describes in detail the process by which the computing device calculates the (q+1)th iteration using the result of the q-th iteration and the intermediate result of the (q+1)-th iteration. The calculation process for the result of the (q+1)-th iteration depends on the specific content of the intermediate result of the (q+1)-th iteration.

[0270] When the intermediate result of the (q+1)th iteration belongs to the aforementioned Figure 3 In step S104 of the model storage method, under case 1, the intermediate result of the (q+1)th iteration is the weight gradient and bias gradient of the model in the (q+1)th iteration. The calculation process of the calculation result of the (q+1)th iteration is as follows: the model parameters in the calculation result of the (q+1)th iteration are updated using the weight gradient and bias gradient of the model in the (q+1)th iteration, thereby obtaining the calculation result of the model in the (q+1)th iteration.

[0271] When there is only one computing device, the above calculation process is as follows: the computing device uses the weight gradient and bias gradient of the model in the (q+1)th iteration to update the model parameters, thereby determining the model obtained in the (q+1)th iteration and obtaining the calculation result of the model in the (q+1)th iteration.

[0272] When there are multiple computing devices, the above calculation process is as follows: computing device i uses the weight gradient of the model in the (q+1)th iteration. Update the weights W of the i-th layer in the model i (q) to obtain the weight W i (q+1), using the bias gradient of the model in the (q+1)th iteration. Update the bias B of the i-th layer in the model i (q) to obtain bias B i (q+1). After updating the weights and biases of all linear layers in the model, the computation result of the model in the (q+1)th iteration can be obtained.

[0273] More specifically, when the optimizer is SGDwith Momentum, the computing device implements the aforementioned Figure 3 In the model storage method, formulas (7)-(10) in step S102; when the optimizer is Adam, the computing device implements the aforementioned Figure 3 In the model storage method, formulas (11)-(20) in step S102; when the optimizer is RMSprop, the computing device implements the aforementioned Figure 3 Formulas (21)-(24) in step S102 of the model storage method.

[0274] As can be seen, the process of calculating the result of the (q+1)th iteration using the calculation result of the q-th iteration and the weight gradient and bias gradient of the model in the (q+1)-th iteration is only a part of the calculation process in the backpropagation process of a complete iteration of the model. This calculation method omits the forward propagation process in a complete iteration of the model, as well as the remaining calculation process in the backpropagation process in a complete iteration of the model (specifically, the process of calculating the weight gradient and bias gradient in the q-th iteration based on the loss of the model in the q-th iteration). Therefore, the process of obtaining the calculation result of one iteration using this method is much shorter than the process of directly using the calculation result of the q-th iteration to perform the model in the (q+1)-th iteration to obtain the calculation result of the (q+1)-th iteration (see the implementation process of the model in the t-th iteration for details). Therefore, this method has a shorter calculation time and lower computational cost.

[0275] When the intermediate result of the (q+1)th iteration belongs to the aforementioned Figure 3 In step S104 of the model storage method, in case 2, the intermediate result of the (q+1)th iteration is the loss of the model in the (q+1)th iteration. The calculation process of the calculation result of the (q+1)th iteration is as follows: based on the loss of the model in the (q+1)th iteration, calculate the weight gradient and bias gradient of the model in the (q+1)th iteration; then use the weight gradient and bias gradient of the model in the (q+1)th iteration to update the model parameters in the calculation result of the (q)th iteration, thereby obtaining the calculation result of the model in the (q+1)th iteration.

[0276] When there is only one computing device, the above calculation process is as follows: the computing device calculates the weight gradient and bias gradient of the model in the (q+1)th iteration based on the loss of the model in the (q+1)th iteration; then it uses the weight gradient and bias gradient of the model in the (q+1)th iteration to update the model parameters in the calculation result of the qth iteration, thereby obtaining the calculation result of the model in the (q+1)th iteration.

[0277] When there are multiple computing devices, the above calculation process is as follows: Computing device i first calculates the weight gradient of the model in the (q+1)th iteration based on the model's loss in the (q+1)th iteration. and bias gradient Then use the model's weight gradient from the (q+1)th iteration. Update the weights W of the i-th layer in the model i (q) to obtain the weight W i (q+1), using the bias gradient of the model in the (q+1)th iteration. Update the bias B of the i-th layer in the model i (q) to obtain bias B i(q+1). After updating the weights and biases of all linear layers in the model, the computation result of the model in the (q+1)th iteration can be obtained.

[0278] More specifically, the process by which each of the multiple computing devices calculates the weight gradient and bias gradient of the model in the (q+1)th iteration is to achieve the aforementioned Figure 3 The execution process of calculation step 21 in step S102 of the model storage method is specifically to implement formulas (4)-(6). The process by which each computing device in multiple computing devices updates the model parameters in the calculation result of the qth iteration using the weight gradient and bias gradient of the model in the (q+1)th iteration depends on the optimizer type. When the optimizer is SGDwithMomentum, each computing device implements the aforementioned Figure 3 In the model storage method, formulas (7)-(10) in step S102; when the optimizer is Adam, each computing device implements the aforementioned Figure 3 In the model storage method, formulas (11)-(20) in step S102; when the optimizer is RMSprop, each computing device implements the aforementioned Figure 3 Formulas (21)-(24) in step S102 of the model storage method.

[0279] As can be seen, the process of using the calculation result of the qth iteration and the loss calculation of the model in the (q+1)th iteration to obtain the calculation result of the (q+1)th iteration is just the backpropagation process of a complete iteration of the model. This calculation method omits the forward propagation process of a complete iteration of the model. Therefore, the process of obtaining the calculation result of one iteration using this method is much shorter than the process of directly using the calculation result of the qth iteration to perform the (q+1)th iteration of the model to obtain the calculation result of the (q+1)th iteration (see the implementation process of the tth iteration of the model for details). Therefore, this method has a shorter calculation time and lower computational cost.

[0280] When the intermediate result of the (q+1)th iteration belongs to the aforementioned Figure 3 In step S104 of the model storage method, in case 3, the intermediate result of the (q+1)th iteration is the input data of the loss function in the (q+1)th iteration. The calculation process of the calculation result of the (q+1)th iteration is as follows: calculate the loss of the model in the (q+1)th iteration based on the input data of the loss function in the (q+1)th iteration; then calculate the weight gradient and bias gradient of the model in the (q+1)th iteration based on the loss of the model in the (q+1)th iteration; then update the model parameters in the calculation result of the (q)th iteration using the weight gradient and bias gradient of the model in the (q+1)th iteration, thereby obtaining the calculation result of the model in the (q+1)th iteration.

[0281] When there is only one computing device, the above calculation process is as follows: the computing device calculates the model loss in the (q+1)th iteration based on the inference results and actual results of the samples in the (q+1)th iteration; then, based on the model loss in the (q+1)th iteration, it calculates the weight gradient and bias gradient of the model in the (q+1)th iteration; then, it uses the weight gradient and bias gradient of the model in the (q+1)th iteration to update the model parameters in the calculation results of the (q)th iteration, thereby obtaining the calculation result of the model in the (q+1)th iteration.

[0282] When there are multiple computing devices, the above calculation process is as follows: First, the computing device responsible for calculating the model's loss calculates the model's loss in the (q+1)th iteration based on the inference results and actual results of the samples in the (q+1)th iteration. Next, computing device i performs the following operation: first, it calculates the model's weight gradient in the (q+1)th iteration based on the model's loss in the (q+1)th iteration. and bias gradient Then use the model's weight gradient from the (q+1)th iteration. Update the weights W of the i-th layer in the model i (q) to obtain the weight W i (q+1), using the bias gradient of the model in the (q+1)th iteration. Update the bias B of the i-th layer in the model i (q) to obtain bias B i (q+1). After updating the weights and biases of all linear layers in the model, the computation result of the model in the (q+1)th iteration can be obtained.

[0283] More specifically, the process by which the computing device responsible for calculating the model's loss in the (q+1)th iteration calculates the model's loss is what is meant by achieving the aforementioned... Figure 3 The execution process of calculation step 13 in step S102 of the model storage method is specifically to implement formula (3). The process of each computing device in multiple computing devices calculating the weight gradient and bias gradient of the model in the (q+1)th iteration is to implement the aforementioned Figure 3 The execution process of calculation step 21 in step S102 of the model storage method is specifically to implement formulas (4)-(6). The process by which each computing device in multiple computing devices updates the model parameters in the calculation result of the qth iteration using the weight gradient and bias gradient of the model in the (q+1)th iteration depends on the optimizer type. When the optimizer is SGDwithMomentum, each computing device implements the aforementioned Figure 3 In the model storage method, formulas (7)-(10) in step S102; when the optimizer is Adam, each computing device implements the aforementioned Figure 3In the model storage method, formulas (11)-(20) in step S102; when the optimizer is RMSprop, each computing device implements the aforementioned Figure 3 Formulas (21)-(24) in step S102 of the model storage method.

[0284] As can be seen, the process of calculating the result of the (q+1)th iteration using the calculation result of the qth iteration and the input data of the loss function in the (q+1)th iteration is only a part of the calculation process of the forward propagation process and the back propagation process in a complete iteration of the model. This calculation method omits the remaining calculation process in the forward propagation process of a complete iteration of the model (specifically, the process of inputting the sample into the model obtained in the (q-1)th iteration to obtain the input data of the loss function in the qth iteration). Therefore, the process of obtaining the calculation result of one iteration using this method is much shorter than the process of directly using the calculation result of the qth iteration to perform the model in the (q+1)th iteration to obtain the calculation result of the (q+1)th iteration (see the implementation process of the model in the tth iteration for details). Therefore, this method has a shorter calculation time and lower computational cost.

[0285] In summary, whether the intermediate results include the model's weight gradients and bias gradients in the corresponding iteration, the intermediate results are the model's loss in the corresponding iteration, or the intermediate results are the input data of the model's loss function in the corresponding iteration, this technical solution recovers multiple computational results much faster than the existing method of periodically saving the model's computational results at different training stages and using the latest saved computational results to perform multiple complete iterations of the model to recover multiple computational results. Therefore, this technical solution can reduce the computational and time costs of recovering training results and reduce the time cost of training tasks.

[0286] It should be understood that the above three calculation processes for the results of the (q+1)th iteration are merely examples. In practical applications, the calculation process for the results of the (q+1)th iteration is related to the specific content of the intermediate results of the (q+1)th iteration. When the specific content of the intermediate results of the (q+1)th iteration changes, the calculation process for the results of the (q+1)th iteration also changes accordingly. This application does not impose any specific limitations.

[0287] In some possible implementations, after obtaining multiple computation results, the computing device takes any one of the multiple computation results as the target computation result, and then continues to iterate the model based on the target computation result until the computation result of the model in the Rth iteration of the Xth training operation is obtained, and then iterates onwards.

[0288] The following section continues to use the various calculation results obtained in the different examples (including Examples 21-23) as examples to illustrate the process by which the computing device determines the target calculation result.

[0289] In Example 21 corresponding to step S202 above, when multiple calculation results include the calculation results of the model from the second iteration to the Qth iteration in the Xth training operation, the computing device selects any one of the calculation results from the second iteration to the Qth iteration in the Xth training operation as the target calculation result.

[0290] For example, the computing device selects the calculation result of the second iteration as the target calculation result, and then performs a third iteration on the model based on the calculation result of the second iteration to obtain the calculation result of the third iteration. Then, based on the calculation result of the third iteration, the model is iterated a fourth time to obtain the calculation result of the fourth iteration, and so on, until the calculation result of the Rth iteration is obtained. The process of each iteration can be referred to the aforementioned... Figure 3 The implementation process of the t-th iteration of the model in step S102 of the model storage method.

[0291] Alternatively, the computing device selects the latest calculation result as the target calculation result, that is, selects the calculation result of the Qth iteration as the target calculation result. Then, based on the calculation result of the Qth iteration, the model is iterated for the (Q+1)th iteration to obtain the calculation result of the (Q+1)th iteration. Then, based on the calculation result of the (Q+1)th iteration, the model is iterated for the (Q+2)th iteration to obtain the calculation result of the (Q+2)th iteration, and so on, until the calculation result of the Rth iteration is obtained. Since the calculation result of the Qth iteration is the closest to the calculation result before the anomaly among the calculation results from the 2nd to the Qth iterations, using the calculation result of the Qth iteration to continue iterating the model until the calculation result before the anomaly is restored can save the time and computational costs of the recovery process compared to using other calculation results.

[0292] In Example 22 corresponding to step S202 above, when multiple calculation results include the calculation results of the model from the second iteration to the Qth iteration in the Mth training operation, the computing device selects any one of the calculation results from the second iteration to the Qth iteration in the Mth training operation as the target calculation result.

[0293] Example 23, corresponding to step S202 above, includes multiple calculation results, from the second iteration to the Nth iteration in the Mth training operation. M Given the calculation results of the nth iteration, and the calculation results of the model from the 1st to the Qth iteration in the Xth training operation, the computing device calculates the model from the 2nd to the Nth iteration in the Mth training operation. MThe computing device may select any calculation result from the calculation results of the iteration as the target calculation result, or select any calculation result from the calculation results of the model from the first iteration to the Qth iteration in the Xth training operation as the target calculation result.

[0294] In some other possible implementations, when there are multiple computing devices, after obtaining multiple computing results, the target computing result is determined from the multiple computing results according to the iteration progress of the multiple computing devices, and the model is iterated on according to the target computing result until the computing result of the model in the Rth iteration in the Xth training operation is obtained, and then the iteration continues.

[0295] The following explanation uses the multiple calculation results obtained in Example 21 above as an example to illustrate the process of determining the target calculation result. If the multiple calculation results obtained by one of the computing devices include the calculation results of one or more linear layers in the model from the 2nd to the Qth iteration in the Xth training operation, and the latest iteration completed by each of the multiple computing devices is the Sth iteration in the Xth training operation (S is a positive integer), if S < Q, then each computing device uses the calculation result of the Sth iteration as the target calculation result, and then uses the calculation result of the Sth iteration to continue iterating the model until the calculation result of the Rth iteration is obtained. If S ≥ Q, then each computing device uses the calculation result of the Qth iteration as the target calculation result, and then uses the calculation result of the Qth iteration to continue iterating the model until the calculation result of the Rth iteration is obtained.

[0296] In summary, this technical solution provides a method for recovering models in distributed training scenarios. When multiple computing devices are used to train a model together, the different computing performance of each device leads to different iteration progresses. This technical solution obtains the iteration progress of different computing devices and retrains the model based on the iteration progress of the lagging computing device, thereby ensuring data consistency among all computing devices.

[0297] It should be noted that the execution process of step S201 and step S202 can be carried out in parallel. Specifically, each time the computing device obtains an intermediate result, it calculates the corresponding result based on that intermediate result.

[0298] In some possible implementations, the Xth training operation includes N iterations. X It depends on the frequency of anomalies in the computing device and is negatively correlated with the frequency of anomalies.

[0299] In one specific implementation, the operating status of the computing device during the process before training the model is monitored, the frequency of anomalies during the operation of the computing device before training the model is statistically analyzed based on the monitoring results, and then N is determined based on the frequency of anomalies. X The specific value. If the abnormal frequency is greater than or equal to the first threshold, then N... X Set to less than or equal to the second threshold; if the abnormal frequency is less than the first threshold, then N is set to... X Set to be greater than the second threshold. Both the first and second thresholds are determined by the user.

[0300] In another specific implementation, the operating status of the computing device during any training operation (such as the Mth training operation) prior to the Xth training operation is monitored. Based on the monitoring results, the frequency of anomalies in the computing device during the execution of that training operation is statistically analyzed, and then N is determined based on the frequency of anomalies. X The specific value. If the abnormal frequency is greater than or equal to the first threshold, then N... X Set to less than or equal to the second threshold; if the abnormal frequency is less than the first threshold, then N is set to... X Set to be greater than the second threshold.

[0301] In another specific implementation, the operating status of the computing device is monitored during multiple training operations (such as the Mth and M+1th training operations) prior to the execution of the Xth training operation. Based on the monitoring results, the frequency of anomalies in the computing device during the execution of these training operations is statistically analyzed, and then N is determined based on the frequency of anomalies. X The specific value. If the abnormal frequency is greater than or equal to the first threshold, then N... X Set to less than or equal to the second threshold; if the abnormal frequency is less than the first threshold, then N is set to... X Set to be greater than the second threshold.

[0302] In another specific implementation, the operating status of the computing device is monitored during the process before training the model, and the operating status of the computing device during any or multiple training operations before executing the Xth training operation. Based on the monitoring results, the frequency of anomalies of the computing device during the monitored process is statistically analyzed, and then N is determined based on the frequency of anomalies. X The specific value. If the abnormal frequency is greater than or equal to the first threshold, then N... X Set to less than or equal to the second threshold; if the abnormal frequency is less than the first threshold, then N is set to... X Set to be greater than the second threshold.

[0303] It should be understood that the number of iterations included in different training operations can be set to different values ​​or the same value, and this application does not make specific limitations.

[0304] The above determines NX The numerical method is to start with N X Before the next iteration, N is determined based on the fault status of the computing device. X The numerical values ​​are adaptively adjusted, which essentially changes the frequency at which the calculation results are saved. In scenarios where the resources used to store the model are the same as those used to train the model, adaptively adjusting the frequency of saving the calculation results helps balance the efficiency of the training model and the speed of model recovery.

[0305] In summary, when a training model malfunctions, the model training method provided in this application can use the first calculation result and intermediate results generated in multiple iterations after the generation of the first calculation result to recover the calculation results in the multiple iterations after the generation of the first calculation result. Then, the model is trained again based on the recovered calculation results until the calculation results before the failure are restored, and training continues. Since the process of using the first calculation result and multiple intermediate results to calculate multiple calculation results corresponding one-to-one with the multiple intermediate results is much shorter than the existing scheme of periodically saving the calculation results of the model at different training stages and retraining the model using the latest saved calculation results to obtain the calculation results before the failure, this technical solution can reduce the computational and time overhead of restoring training, thus reducing the time cost of the training task. In addition, existing schemes that save the calculation results of the model in each iteration can provide the latest calculation results to restore the calculation results before the failure, but require a huge amount of data to be stored. In contrast, this technical solution uses the first calculation result and the intermediate results generated in multiple iterations after the first calculation result to calculate the calculation results in multiple iterations after the first calculation result is generated. This also achieves the effect of restoring the calculation results before the failure. Therefore, this technical solution achieves calculation instead of storage.

[0306] In a specific scenario, this specifically refers to performing N operations on the model. X In the scenario where an anomaly occurs during the Rth iteration of the Nth iteration, where the first calculation result is the result of the model's calculation in the 1st iteration of the Xth training operation, and multiple intermediate results, including intermediate results from the 2nd to the Qth iterations of the Xth training operation, this technical solution can use the model in N... X The model is recovered using the calculation results of the first iteration and the intermediate results from the second to the Qth iteration in N. X The calculation results from the second to the Qth iteration in the second iteration can then be used to apply the model to N. X In this iteration, any calculation result from the second to the Qth iteration continues to iterate the model until the calculation result before the anomaly is recovered (i.e., the model is in N). XThe calculation result of the Rth iteration in the nth iteration). Since the process of calculating the result of the (q+1)th iteration using the calculation result of the qth iteration and the intermediate result of the (q+1)th iteration is much shorter than the process of directly using the calculation result of the qth iteration to perform a complete iteration of the model to obtain the calculation result of the (q+1)th iteration, this technical solution restores the model in N X The computation speed of the second to Qth iterations in this iteration is much faster than that of the existing scheme. The latest saved computation results for the model in N... X Given the calculation results of the first iteration in the next iteration, the model is used in N. X In this iteration, the calculation result of the first iteration is used to perform Q-1 consecutive complete iterations on the model to recover the model in N. X The process of calculating the results from the second to the Qth iteration in the next iteration. Therefore, this technical solution can reduce the computational and time overhead of resuming training and reduce the time cost of the training task.

[0307] In another specific scenario, it specifically refers to performing N on the model. X In the scenario where an anomaly occurs during the Rth iteration in the nth iteration, even if the model is not stored in N... X In the next iteration, the calculation result of the first iteration can still be used to recover the model by using the stored calculation results from previous iterations. Specifically, in the first calculation result, the model is in N... M The calculation results of the first iteration in the next iteration, including multiple intermediate results: the model in N M In the second iteration to the Nth iteration M Intermediate results of the Nth iteration, and the model at N X The intermediate results from the 1st to the Qth iteration in the Nth iteration, and N M In the Nth iteration M The next iteration is N. X In the case of the iteration preceding the first iteration in the next iteration, this technical solution can be used when the model is in N M In the Nth iteration M The intermediate results of the next iteration are used to calculate the model in N. M In the Nth iteration M After the calculation results of the nth iteration, since N M In the Nth iteration M The next iteration is N. X The iteration preceding the first iteration in the next iteration, therefore the model can be used again in N. M In the Nth iteration M The calculation results of the next iteration and the model in N X The intermediate results of the first iteration in the next iteration are used to calculate the model in N. XThe calculation result of the first iteration in the next iteration can then be used sequentially with N. X The intermediate results from the second to the Qth iteration in the next iteration are used to recover the model in N. X The calculation results from the 2nd to the Qth iteration in this iteration. If the model is not stored in N... X The intermediate result of the first iteration in the nth iteration is then used in the model in N. M In the Nth iteration M The intermediate results of the next iteration are used to calculate the model in N. M In the Nth iteration M After calculating the results of the next iteration, the model needs to be used in N... M In the Nth iteration M The calculation results of the second iteration are used to perform a complete iteration of the model to obtain the model's performance in N. X The calculation result of the first iteration in the next iteration. Therefore, compared to not storing the model in N... X The intermediate results of the first iteration in the next iteration, when the model is stored in N X When using the intermediate results of the first iteration in the next iteration, the time and computational costs of restoring the model are both smaller.

[0308] See Figure 5 , Figure 5 This is a schematic diagram of the structure of a storage device provided in this application. The storage device 400 can be used to implement the aforementioned... Figure 3 The model storage method. For example... Figure 5 As shown, the storage device 400 includes a first storage unit 401 and a second storage unit 402.

[0309] The first storage unit 401 is used to store the model in N. M The calculation result of the first iteration in the next iteration;

[0310] The second storage unit 402 is used to store the model in N. M The intermediate result of the Kth iteration in the kth iteration;

[0311] The second storage unit 402 is also used to store the model in N, based on storing the intermediate results of the Kth iteration. M The intermediate result of the (K+1)th iteration in the next iteration;

[0312] The second storage unit 402 is also used to store the intermediate results of the Kth iteration and the intermediate results of the (K+1)th iteration, and to store the model in N. M The intermediate result of the (K+2)th iteration in the next iteration;

[0313] The total number of iterations of the model is N, N M ≤N, N MBoth N and K are positive integers. The intermediate results of the Kth iteration are used to calculate the result of the Kth iteration. The result of the Kth iteration is the final output of the model at the end of the Kth iteration. The amount of data in the intermediate results of the Kth iteration is less than the amount of data in the result of the Kth iteration. K ≥ 2, and K is an integer.

[0314] In some possible implementations, the computation result of the first iteration includes the optimizer state, the model weights and biases, and the optimizer state is used to update the model weights and biases; the intermediate result of the Kth iteration includes the weight gradient and bias gradient of the model in the Kth iteration, or the intermediate result of the Kth iteration includes the loss of the model in the Kth iteration, or the intermediate result of the Kth iteration includes the input data of the loss function of the model in the Kth iteration.

[0315] In some possible implementations, the second storage unit 402 is also used for: storing the model in N M In the Nth iteration M Based on the intermediate results of the Nth iteration, the storage model is in N X The intermediate results of the Pth iteration in the nth iteration; in the model stored in N M In the Nth iteration M The intermediate results of the next iteration and the model in N X Based on the intermediate results of the Pth iteration in the Nth iteration, the storage model is in N X The intermediate result of the (P+1)th iteration in the nth iteration. Where N M In the Nth iteration M The next iteration is N. X The iteration preceding the first iteration in the next iteration. The model in N... M In the Nth iteration M The intermediate results of the second iteration are used to calculate the model in N. M In the Nth iteration M The calculation results of the Nth iteration. The model in N M In the Nth iteration M The calculation results of the next iteration and the model in N X The intermediate results of the first iteration in the next iteration are used to calculate the model in N. X The result of the first iteration in the next iteration. N M <N, N X <N, N X P is a positive integer, P≥1, and P is an integer.

[0316] In some possible implementations, the first storage unit 401 is also used to store the model in N. X The calculation result of the first iteration in the next iteration. Where N... M In the Nth iteration MThe execution time of this iteration is earlier than N. X The execution time of the first iteration in the next iteration. The first storage unit 401 is also used to store the successful model in N. X Given the calculation results of the first iteration in the next iteration, delete the model in N. M The calculation result of the first iteration in the next iteration. The second storage unit 402 is also used to store the successful model in N. X Given the calculation results of the first iteration in the next iteration, delete the model in N. M Intermediate results of each iteration starting from the second iteration in the next iteration.

[0317] In some possible implementations, N X The value of is negatively correlated with the anomaly frequency of the computing device. Here, the anomaly frequency is the N-order frequency of the model implemented on the computing device. M The frequency of anomalies is determined during the iteration process, and / or the frequency of anomalies is determined during the operation of the computing device before the model is trained.

[0318] Both the first storage unit 401 and the second storage unit 402 can be implemented in software, hardware, or a combination of both. For example, the implementation of the first storage unit 401 will be described below. Similarly, the implementation of the second storage unit 402 can refer to the implementation of the first storage unit 401.

[0319] As an example of a software functional unit, the first storage unit 401 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Further, the aforementioned computing instance may be one or more. For example, the first storage unit 401 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0320] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0321] As an example of a hardware functional unit, the first storage unit 401 may include at least one computing device, such as a server. Alternatively, the first storage unit 401 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0322] The multiple computing devices included in the first storage unit 401 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the first storage unit 401 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the first storage unit 401 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0323] It should be noted that, in other embodiments, the first storage unit 401 can be used to execute any step in the model storage method, and the second storage unit 402 can be used to execute any step in the model storage method. The steps implemented by the first storage unit 401 and the second storage unit 402 can be specified as needed. The storage device 400 can achieve all its functions by implementing different steps in the model storage method through the first storage unit 401 and the second storage unit 402 respectively.

[0324] See Figure 6 , Figure 6 This is a schematic diagram of the structure of a computing device provided in this application. The computing device 500 can be used to implement the aforementioned... Figure 4 Model training methods. For example... Figure 6 As shown, the computing device 500 includes a result acquisition unit 501 and a recovery unit 502.

[0325] The result acquisition unit 501 is used to acquire the first calculation result and multiple intermediate results in the event of an anomaly in the training model.

[0326] The recovery unit 502 is used to calculate multiple calculation results based on the first calculation result and multiple intermediate results, and then continue to iterate the model based on any one of the multiple calculation results.

[0327] The first calculation result is the final output of the model at the end of one iteration. Multiple intermediate results are generated in subsequent iterations after the first calculation result, and there is a one-to-one correspondence between the intermediate results and the iterations. The calculation result indicates the final output of the model at the end of the corresponding iteration. The amount of data in the intermediate results is less than the amount of data in the corresponding calculation result.

[0328] In some possible implementations, the computation results include the optimizer state, model weights, and biases. The optimizer state is used to update the model weights and biases. Intermediate results include the model's weight gradients and bias gradients in the corresponding iteration, or the model's loss in the corresponding iteration, or the input data of the model's loss function in the corresponding iteration.

[0329] In some possible implementations, the first calculation result is N. X The calculation results of the first iteration in the next iteration, including multiple intermediate results, include the model in N. X The intermediate results from the second to the Qth iteration in this iteration. Recovery unit 502 is specifically used for: when performing N iterations on the model... X In the case of an anomaly occurring during the Rth iteration, the result of the (q+1)th iteration is calculated by sequentially using the calculation result of the qth iteration and the intermediate result of the (q+1)th iteration, following the order of q values ​​from smallest to largest. This yields the model's result in N... X The calculation results from the second to the Qth iteration in the next iteration are then used to calculate the model in N. X In this iteration, any calculation result from the second to the Qth iteration is used to iterate the model again, thereby obtaining the model in N. X The calculation result of the Rth iteration in the nth iteration. Where N X R, Q, and q are all positive integers, where R > 2 and Q < R.

[0330] In some possible implementations, the recovery unit 502 is specifically used to continue iterating the model using the calculation results of the Qth iteration.

[0331] In some possible implementations, the recovery unit 502 is specifically used to: complete N on multiple computing devices X In the case of the S-th iteration in the 1st iteration, if S < Q, the model is iterated again using the calculation result of the S-th iteration; if S ≥ Q, the model is iterated again using the calculation result of the Q-th iteration. Multiple computing devices are used to train the model, and S is a positive integer.

[0332] In some possible implementations, when performing N on the model X In the case of an anomaly occurring during the Rth iteration in the nth iteration, the first calculation result is the model in N. M The calculation results of the first iteration in the next iteration, including multiple intermediate results: the model in N M In the second iteration to the Nth iteration M Intermediate results of the Nth iteration, and the model at N X The intermediate results from the 1st to the Qth iteration in this iteration. Where N... M In the Nth iteration M The next iteration is N. X The iteration preceding the first iteration in the next iteration. N X R, Q, N M All are positive integers, R > 2, Q < R.

[0333] Both the result acquisition unit 501 and the recovery unit 502 can be implemented in software, hardware, or a combination of both. For example, the implementation of the result acquisition unit 501 will be described below. Similarly, the implementation of the recovery unit 502 can be referenced to that of the result acquisition unit 501.

[0334] As an example of a software functional unit, the result acquisition unit 501 may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, or a container. Further, the aforementioned computing instance may be one or more. For example, the result acquisition unit 501 may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0335] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0336] As an example of a hardware functional unit, the result acquisition unit 501 may include at least one computing device, such as a server. Alternatively, the result acquisition unit 501 may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0337] The multiple computing devices included in the result acquisition unit 501 can be distributed in the same region or in different regions. Similarly, the multiple computing devices included in the result acquisition unit 501 can be distributed in the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the result acquisition unit 501 can be distributed in the same Virtual Private Cloud (VPC) or in multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0338] It should be noted that, in other embodiments, the result acquisition unit 501 can be used to execute any step in the model storage method, and the second recovery unit 502 can be used to execute any step in the model storage method. The steps implemented by the result acquisition unit 501 and the recovery unit 502 can be specified as needed. The result acquisition unit 501 and the recovery unit 502 respectively implement different steps in the model storage method to realize all the functions of the computing device 500.

[0339] This application also provides a chip, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor performs the aforementioned operations. Figure 3 The steps in the model storage method are executed by the computing device. Alternatively, the processor is used to execute the aforementioned steps. Figure 4 The steps in the model training method performed by computing devices are described below for brevity. These steps will not be elaborated upon here. The processor can be a CPU, or a computing device such as a GPU, DPU, NPU, TPU, XPU, SoC, offload card, or accelerator card.

[0340] See Figure 7 , Figure 7 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application. Figure 7 As shown, the computing device 600 provided in this application includes: a bus 601, a processor 602, a memory 603, and a communication interface 604. The processor 602, the memory 603, and the communication interface 604 communicate with each other via the bus 601. The computing device 600 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in the computing device 600.

[0341] Bus 601 can be a Peripheral Component Interconnect Express (PCIe) bus, an Extended Industry Standard Architecture (EISA) bus, a Unified Bus (Ubus or UB), a Compute Express Link (CXL) bus, a Cache Coherent Interconnect for Accelerators (CCIX) bus, etc. The Unified Bus is also known as the Lingqu Bus. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 7 The bus 601 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 601 may include a path for transmitting information between various components of the computing device 600 (e.g., memory 603, processor 602, communication interface 604).

[0342] The processor 602 may include any one or more of the following computing devices: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP) or digital signal processor (DSP), ASIC, FPGA, CPLD, NPU, SoC, offload card, accelerator card, etc.

[0343] Memory 603 may include volatile memory, such as random access memory (RAM). Processor 602 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD). Furthermore, memory 603 may also be implemented using storage class memory (SCM), phase change memory (PCM), or other types of storage media.

[0344] It is worth noting that the same type of storage medium can be configured in the same computing device to realize the function of memory 603, or two or more types of storage media can be configured to realize the function of memory 603. This application does not limit this.

[0345] The memory 603 stores executable program code, and the processor 602 executes the executable program code to implement the functions of the aforementioned first storage unit 401 and second storage unit 402, thereby achieving the aforementioned... Figure 3 The model storage method. That is, the memory 603 stores the data used for execution. Figure 3 Instructions for the model storage method.

[0346] Alternatively, the memory 603 stores executable code, and the processor 602 executes the executable program code to implement the functions of the aforementioned result acquisition unit 501 and recovery unit 502, thereby performing the aforementioned functions respectively. Figure 4 The model training method. That is, the memory 603 stores the data used for execution. Figure 4 Instructions for model training methods.

[0347] The communication interface 604 uses transceiver units such as, but not limited to, network interface cards and transceivers to enable communication between the computing device 600 and other computing devices or communication networks.

[0348] As one possible implementation, the computing device 600 may also include a chip, which includes a processor and a power supply circuit. The power supply circuit supplies power to the processor, and the processor performs the aforementioned operations. Figure 3 The operation steps performed by the computing device in the model storage method, or the aforementioned Figure 4 The model training method involves operations performed by computing devices. These operations can be implemented using a CPU, or through computing devices or AI chips such as GPUs, DPUs, NPUs, TPUs, XPUs, SoCs, offloading cards, and accelerator cards.

[0349] As one possible implementation, the computing device 600 may include multiple types of processors 602, that is, the computing device 600 is a heterogeneous device. For example, the computing device 600 includes a CPU and a GPU, and the aforementioned functions can be executed by at least one of the processors 602. Figure 3 The operation steps performed by the computing device in the model storage method, or the aforementioned Figure 4 The operational steps performed by the computing device in the model training method are omitted here for brevity.

[0350] See Figure 8 , Figure 8This is a schematic diagram of a computing device cluster provided in an embodiment of this application. The computing device cluster provided in this embodiment includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0351] like Figure 8 As shown, the computing device cluster includes at least one computing device 600. The memory 603 in one or more computing devices 600 of the computing device cluster may store the same memory for performing the aforementioned tasks. Figure 3 The instructions for the steps executed by the computing device in the model storage method, or instructions for performing the aforementioned Figure 4 The instructions for the steps executed by the computing device in the model training method.

[0352] In some possible implementations, the memory 603 of one or more computing devices 600 in the computing device cluster may also store memory for executing the aforementioned commands. Figure 3 The model storage method contains partial instructions for steps executed by a computing device. In other words, a combination of one or more computing devices 600 can jointly execute instructions for performing the aforementioned steps. Figure 3 The instructions for the steps executed by the computing device in the model storage method.

[0353] It should be noted that the memory 603 in different computing devices 600 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 5 The instructions stored in the memory 603 of different computing devices 600 can implement the functions of one or more of the first memory unit 401 and the second memory unit 402.

[0354] In some other possible implementations, the memory 603 of one or more computing devices 600 in the computing device cluster may also store memory for performing the aforementioned tasks. Figure 4 The instructions for the steps executed by the computing device in the model training method. In other words, a combination of one or more computing devices 600 can jointly execute the instructions for performing the aforementioned steps. Figure 4 The instructions for the steps executed by the computing device in the model training method.

[0355] It should be noted that the memory 603 in different computing devices 600 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 6The computing device 500 has some of the functions. That is, the instructions stored in the memory 603 in different computing devices 600 can realize the functions of one or more units in the result acquisition unit 501 and the recovery unit 502.

[0356] See Figure 9 , Figure 9 This is a schematic diagram of another computing device cluster structure provided in an embodiment of this application. In some possible implementations, one or more computing devices in the computing device cluster can be connected via a network. The network can be a wide area network (WAN) or a local area network (LAN), etc. Figure 9 As shown, the two computing devices 600A and 600B are connected via a network. Specifically, they are connected to the network through the communication interfaces in each computing device.

[0357] In some possible implementations, the memory 603 in computing device 600A stores instructions for executing the function of the first storage unit 401. Simultaneously, the memory 603 in computing device 600B stores instructions for executing the function of the second storage unit 402. This connection method may be considered because the model storage method provided in this application requires storing a large amount of computational result data, therefore delegating the function implemented by the first storage unit 401 to computing device 600A for execution.

[0358] In some possible implementations, the memory 603 in computing device 600A stores instructions for performing the function of the result acquisition unit 501. Simultaneously, the memory 603 in computing device 600B stores instructions for performing the function of the recovery unit 502. This connection method can be considered because the model training method provided in this application requires a large number of matrix operations, therefore delegating the function implemented by the recovery unit 502 to computing device 600B.

[0359] It should be understood that Figure 9 The functions of computing device 600A shown can also be performed by multiple computing devices 600. Similarly, the functions of computing device 600B can also be performed by multiple computing devices 600.

[0360] This application also provides another computing device cluster. The connection relationships between the computing devices in this computing device cluster can be similarly referred to... Figure 8 and Figure 9 The connection method of the computing device cluster is different. In that, the memory 603 of one or more computing devices 600 in this computing device cluster can store the same memory for executing the aforementioned... Figure 3 The instructions for the steps executed by the computing device in the model storage method, or instructions for performing the aforementioned Figure 4 The instructions for the steps executed by the computing device in the model training method.

[0361] In some possible implementations, the memory 603 of one or more computing devices 600 in the computing device cluster may also store memory for executing the aforementioned commands. Figure 3 The partial instructions of the steps executed by the computing device in the model storage method, or, for executing the aforementioned Figure 4 The instructions for the steps executed by the computing device in the model training method. In other words, a combination of one or more computing devices 600 can jointly execute the instructions for performing the aforementioned steps. Figure 3 The instructions for the steps executed by the computing device in the model storage method, or instructions for performing the aforementioned Figure 4 The instructions for the steps executed by the computing device in the model training method.

[0362] It should be noted that the memory 603 in different computing devices 600 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 5 The storage device 400 performs some of the functions. That is, the instructions stored in the memory 603 in different computing devices 600 can implement the functions of one or more of the first storage unit 401 and the second storage unit 402.

[0363] Alternatively, the memory 603 in different computing devices 600 within the computing device cluster can store different instructions, each used to execute the aforementioned instructions. Figure 6 The computing device 500 has some of the functions. That is, the instructions stored in the memory 603 in different computing devices 600 can realize the functions of one or more units in the result acquisition unit 501 and the recovery unit 502.

[0364] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform the aforementioned... Figure 3 The steps performed by the computing device in the model storage method, or the aforementioned steps. Figure 4 The steps performed by the computing device in the model training method.

[0365] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any usable medium that a computing device can store, or a data storage device such as a data center containing one or more usable media. The usable medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that direct a computing device to perform the aforementioned actions. Figure 3The steps performed by the computing device in the model storage method, or the aforementioned steps. Figure 4 The steps performed by the computing device in the model training method.

[0366] It should be understood that in the embodiments of this application, "when," "...when," and "if" all refer to the device making corresponding processing under certain objective circumstances, and are not time-limited, nor do they require the device to make a judgment action, nor do they imply any other limitations.

[0367] It should be understood that in the embodiments of this application, "simultaneous" does not necessarily require that they occur at the same time, minute, second, or even at the same moment. When the times of occurrence are slightly different, they can also be understood as occurring "simultaneously".

[0368] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A model training method, characterized in that, The method includes: In the event of an anomaly during model training, a first calculation result and multiple intermediate results are obtained. The first calculation result is the final output of the model at the end of one iteration, and the multiple intermediate results are generated in multiple iterations after the first calculation result is generated. The multiple intermediate results correspond one-to-one with the multiple iterations. Multiple calculation results are calculated based on the first calculation result and the multiple intermediate results, wherein there is a one-to-one correspondence between the multiple calculation results and the multiple intermediate results, and the calculation results are used to indicate the final output of the model at the end of the corresponding iteration. The amount of data in the intermediate results is less than the amount of data in the corresponding calculation results. The model is iterated on based on any one of the multiple calculation results.

2. The method according to claim 1, characterized in that, The calculation results include the optimizer state, the weights and biases of the model, and the optimizer state is used to update the weights and biases of the model. The intermediate results include the weight gradients and bias gradients of the model in the corresponding iteration, or the intermediate results include the loss of the model in the corresponding iteration, or the intermediate results include the input data of the loss function of the model in the corresponding iteration.

3. The method according to claim 1 or 2, characterized in that, The first calculation result is N X The calculation result of the first iteration in the N iteration, the plurality of intermediate results including the model in the N X The intermediate results from the 2nd to the Qth iteration in the current iteration. In the model, the N X In the event of an anomaly during the Rth iteration, the calculation of multiple results based on the first calculation result and the multiple intermediate results includes: Following the order of q values ​​from smallest to largest, the calculation result of the (q+1)th iteration is obtained by sequentially using the calculation result of the q-th iteration and the intermediate result of the (q+1)-th iteration, thereby obtaining the model in the N... X The calculation results from the 2nd to the Qth iteration in the current iteration, where N X R, Q, and q are all positive integers, where R > 2 and Q < R; The step of iterating the model based on any one of the multiple calculation results includes: Based on the model in N X In the next iteration, any calculation result from the second to the Qth iteration is used to iterate the model, thereby obtaining the model in the Nth iteration. X The calculation result of the Rth iteration in the nth iteration.

4. The method according to claim 3, characterized in that, The model is based on the N X In this iteration, any calculation result from the second to the Qth iteration will continue to iterate the model, including: The model is iterated again using the results of the Qth iteration.

5. The method according to claim 3, characterized in that, The model is based on the N X In this iteration, any calculation result from the second to the Qth iteration will continue to iterate the model, including: The N was completed on multiple computing devices. X In the case of the S-th iteration in the nth iteration If S < Q, continue iterating the model using the calculation result of the Sth iteration; If S≥Q, the model is iterated again using the result of the Qth iteration; The plurality of computing devices are all used to train the model, and S is a positive integer.

6. The method according to claim 1 or 2, characterized in that, After performing N on the model X In the event of an anomaly occurring during the Rth iteration in the nth iteration, the first calculation result is the model in N M The calculation result of the first iteration in the N iteration, the plurality of intermediate results include: the model in the N M In the second iteration to the Nth iteration M The intermediate results of the Nth iteration, and the model in the Nth iteration X The intermediate results from the 1st to the Qth iteration in the N iteration; wherein, the N M In the Nth iteration M The next iteration is the N X The iteration preceding the first iteration in the next iteration, N X R, Q, N M All are positive integers, R > 2, Q < R.

7. A model storage method, characterized in that, The method includes: Storage model in N M The calculation result of the first iteration in the next iteration; Store the model in N M The intermediate result of the Kth iteration in the kth iteration; Based on storing the intermediate results of the Kth iteration, the model is stored in the Nth iteration. M The intermediate result of the (K+1)th iteration in the next iteration; Based on storing the intermediate results of the Kth iteration and the (K+1)th iteration, the model is stored in the Nth iteration. M The intermediate result of the (K+2)th iteration in the next iteration; The total number of iterations of the model is N, where N M ≤N, N M N and N are both positive integers. The intermediate result of the Kth iteration is used to calculate the result of the Kth iteration. The result of the Kth iteration is the final output of the model at the end of the Kth iteration. The amount of data in the intermediate result of the Kth iteration is less than the amount of data in the result of the Kth iteration. K ≥ 2, and K is an integer.

8. The method according to claim 7, characterized in that, The calculation result of the first iteration includes the optimizer state, the weights and biases of the model, and the optimizer state is used to update the weights and biases of the model. The intermediate result of the Kth iteration includes the weight gradient and bias gradient of the model in the Kth iteration, or the intermediate result of the Kth iteration includes the loss of the model in the Kth iteration, or the intermediate result of the Kth iteration includes the input data of the loss function of the model in the Kth iteration.

9. The method according to claim 7 or 8, characterized in that, The method further includes: The model is stored in N. M In the Nth iteration M Based on the intermediate results of the Nth iteration, the model is stored. X The intermediate result of the Pth iteration in the nth iteration; The model is stored in N. M In the Nth iteration M The intermediate results of the Nth iteration and the model in the Nth iteration X Based on the intermediate results of the Pth iteration in the Nth iteration, the model is stored in the Nth iteration. X The intermediate result of the (P+1)th iteration in the next iteration; Wherein, the N M In the Nth iteration M The next iteration is the N X The iteration preceding the first iteration in the next iteration, the model in the N M In the Nth iteration M The intermediate results of the next iteration are used to calculate the model in the Nth iteration. M In the Nth iteration M The calculation results of the Nth iteration, the model in the Nth iteration M In the Nth iteration M The calculation results of the Nth iteration and the model in the Nth iteration X The intermediate results of the first iteration in the next iteration are used to calculate the model in the N... X The calculation result of the first iteration in the next iteration, N M <N, N X <N, N X P is a positive integer, P≥1, and P is an integer.

10. The method according to any one of claims 7-9, characterized in that, The method further includes: Store the model in N X The calculation result of the first iteration in the next iteration, wherein N M In the Nth iteration M The execution time of this iteration is earlier than that of N. X The execution time of the first iteration in the next iteration; The model was successfully stored in N. X Given the calculation result of the first iteration in the next iteration, delete the model in the Nth iteration. M The calculation results of the first iteration in the second iteration and the intermediate results of each iteration starting from the second iteration.

11. The method according to claim 9 or 10, characterized in that, The N X The value of is negatively correlated with the anomaly frequency of the computing device, wherein the anomaly frequency is the N value of the model implemented on the computing device. M The abnormal frequency is determined during the iteration process, and / or is determined during the operation prior to training the model on the computing device.

12. A computing device, characterized in that, include: Result acquisition unit and recovery unit The result acquisition unit is used to acquire a first calculation result and multiple intermediate results when an anomaly occurs in the training model. The first calculation result is the final output of the model at the end of one iteration, and the multiple intermediate results are generated in multiple iterations after the first calculation result is generated. The multiple intermediate results correspond one-to-one with the multiple iterations. The recovery unit is used to calculate multiple calculation results based on the first calculation result and the multiple intermediate results, wherein the multiple calculation results correspond one-to-one with the multiple intermediate results, the calculation results are used to indicate the final output of the model at the end of the corresponding iteration, and the data volume of the intermediate results is less than the data volume of the corresponding calculation results; The recovery unit is also used to continue iterating the model based on any one of the plurality of calculation results.

13. The apparatus according to claim 12, characterized in that, The calculation results include the optimizer state, the weights and biases of the model, and the optimizer state is used to update the weights and biases of the model. The intermediate results include the weight gradients and bias gradients of the model in the corresponding iteration, or the intermediate results include the loss of the model in the corresponding iteration, or the intermediate results include the input data of the loss function of the model in the corresponding iteration.

14. The apparatus according to claim 12 or 13, characterized in that, The first calculation result is N X The calculation result of the first iteration in the N iteration, the plurality of intermediate results including the model in the N X The intermediate results from the 2nd to the Qth iteration in the current iteration. The recovery unit is specifically used for performing the N operation on the model. X In the case of an anomaly occurring during the Rth iteration, the calculation result of the (q+1)th iteration is obtained by sequentially using the calculation result of the qth iteration and the intermediate result of the (q+1)th iteration, following the order of q values ​​from smallest to largest. This yields the model in the Nth iteration. X The calculation results from the second to the Qth iteration in the next iteration are then used to calculate the Nth iteration based on the model. X In the next iteration, any calculation result from the second to the Qth iteration is used to iterate the model, thereby obtaining the model in the Nth iteration. X The calculation result of the Rth iteration in the nth iteration, where N X R, Q, and q are all positive integers, where R > 2 and Q < R.

15. The apparatus according to claim 14, characterized in that, The recovery unit is specifically used to continue iterating the model using the calculation results of the Qth iteration.

16. The apparatus according to claim 14, characterized in that, The recovery unit is specifically used for: completing the N operation on multiple computing devices. X In the case of the S-th iteration in the S-th iteration, if S < Q, the calculation result of the S-th iteration is used to continue iterating the model; If S≥Q, the model is iterated again using the calculation result of the Qth iteration; wherein, all of the plurality of computing devices are used to train the model, and S is a positive integer.

17. The apparatus according to claim 12 or 13, characterized in that, After performing N on the model X In the event of an anomaly occurring during the Rth iteration in the nth iteration, the first calculation result is the model in N M The calculation result of the first iteration in the N iteration, the plurality of intermediate results include: the model in the N M In the second iteration to the Nth iteration M The intermediate results of the Nth iteration, and the model in the Nth iteration X The intermediate results from the 1st to the Qth iteration in the N iteration; wherein, the N M In the Nth iteration M The next iteration is the N X The iteration preceding the first iteration in the next iteration, N X R, Q, N M All are positive integers, R > 2, Q < R.

18. A storage device, characterized in that, include: First storage unit and second storage unit, The first storage unit is used to store the model in N M The calculation result of the first iteration in the next iteration; The second storage unit is used to store the model in the N M The intermediate result of the Kth iteration in the kth iteration; The second storage unit is further configured to store, based on storing the intermediate results of the Kth iteration, the model in the Nth iteration. M The intermediate result of the (K+1)th iteration in the next iteration; The second storage unit is further configured to store, in addition to storing the intermediate results of the Kth iteration and the intermediate results of the (K+1)th iteration, the model in the Nth iteration. M The intermediate result of the (K+2)th iteration in the next iteration; The total number of iterations of the model is N, where N M ≤N, N M N and N are both positive integers. The intermediate result of the Kth iteration is used to calculate the result of the Kth iteration. The result of the Kth iteration is the final output of the model at the end of the Kth iteration. The amount of data in the intermediate result of the Kth iteration is less than the amount of data in the result of the Kth iteration. K ≥ 2, and K is an integer.

19. The apparatus according to claim 18, characterized in that, The calculation result of the first iteration includes the optimizer state, the weights and biases of the model, and the optimizer state is used to update the weights and biases of the model. The intermediate result of the Kth iteration includes the weight gradient and bias gradient of the model in the Kth iteration, or the intermediate result of the Kth iteration includes the loss of the model in the Kth iteration, or the intermediate result of the Kth iteration includes the input data of the loss function of the model in the Kth iteration.

20. The apparatus according to claim 18 or 19, characterized in that, The second storage unit is further used for: The model is stored in N. M In the Nth iteration M Based on the intermediate results of the Nth iteration, the model is stored. X The intermediate result of the Pth iteration in the nth iteration; The model is stored in N. M In the Nth iteration M The intermediate results of the Nth iteration and the model in the Nth iteration X Based on the intermediate results of the Pth iteration in the Nth iteration, the model is stored in the Nth iteration. X The intermediate result of the (P+1)th iteration in the next iteration; Wherein, the N M In the Nth iteration M The next iteration is the N X The iteration preceding the first iteration in the next iteration, the model in the N M In the Nth iteration M The intermediate results of the next iteration are used to calculate the model in the Nth iteration. M In the Nth iteration M The calculation results of the Nth iteration, the model in the Nth iteration M In the Nth iteration M The calculation results of the Nth iteration and the model in the Nth iteration X The intermediate results of the first iteration in the next iteration are used to calculate the model in the N... X The calculation result of the first iteration in the next iteration, N M <N, N X <N, N X P is a positive integer, P≥1, and P is an integer.

21. The apparatus according to claim 20, characterized in that, The N X The value of is negatively correlated with the anomaly frequency of the computing device, wherein the anomaly frequency is the N value of the model implemented on the computing device. M The abnormal frequency is determined during the iteration process, and / or is determined during the operation prior to training the model on the computing device.

22. A chip, characterized in that, The chip includes a processor and a power supply circuit, the power supply circuit being used to supply power to the processor, the processor being used to perform the operation steps of the method as described in any one of claims 1 to 6, or to perform the operation steps of the method as described in any one of claims 7 to 11.

23. A computing device, characterized in that, The computing device includes a processor and memory; The processor is configured to execute instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1 to 6, or to perform the method as described in any one of claims 7 to 11.

24. A computing device cluster, characterized in that, It includes multiple computing devices, each of which includes a processor and memory; The processors of the plurality of computing devices are configured to execute instructions stored in the memory of the plurality of computing devices, such that the cluster of computing devices performs the method as described in any one of claims 1 to 6, or performs the method as described in any one of claims 7 to 11.

25. A computer program product containing instructions, characterized in that, When the instructions are executed by the computing device, the computing device performs the method as described in any one of claims 1 to 6, or performs the method as described in any one of claims 7 to 11.

Images