Model training method and device and computer equipment
By obtaining integer gradient data from the GPU and updating model parameters in deep learning model training, the problem of low data transfer efficiency between devices is solved and the model training efficiency is improved.
Patent Information
- Application Number
- CN202411057279.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-02
- Publication Date
- 2024-12-03
AI Technical Summary
In deep learning model training, data transfer efficiency between different devices is low, resulting in low model training efficiency.
By obtaining integer-type gradient data of this round from the graphics processor GPU, and determining the next round model parameters based on the gradient data and model parameters of this round, sending it to the GPU for forward propagation and back propagation, thereby improving the data Transmission efficiency.
This improves the data transmission efficiency between the GPU and the CPU, reduces the data transmission amount, and thus improves the efficiency of model training.
Smart Images

Figure CN119067173A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of artificial intelligence technology, and in particular to a model training method, apparatus and computer equipment. Background Art
[0002] As the scale and complexity of deep learning models continue to grow, video memory management becomes particularly important in deep learning tasks. Optimizing video memory usage can not only improve computing efficiency, but also reduce hardware costs and energy consumption.
[0003] In traditional technology, different parts of the model are usually placed on different computing devices to reduce the memory requirements of a single device. However, the efficiency of transferring data between different devices is relatively low, resulting in low model training efficiency. Summary of the invention
[0004] Based on this, it is necessary to provide a model training method, device and computer equipment to address the above technical problems, which can improve the efficiency of data transmission between different devices and thus improve the efficiency of model training.
[0005] In a first aspect, the present application provides a model training method, applied to a central processing unit (CPU), the method comprising:
[0006] Obtaining the current round of gradient data of the training model from the graphics processing unit (GPU); wherein the data format of the current round of gradient data is an integer type;
[0007] Determining the model parameters of the next round of the training model according to the current round gradient data and the current round model parameters of the training model;
[0008] The next round model parameters are sent to the GPU, so that the GPU performs forward propagation and back propagation based on the next round model parameters to obtain the next round gradient data of the training model.
[0009] In one embodiment, determining the next round model parameters of the training model according to the current round gradient data and the current round model parameters of the training model includes:
[0010] Determine the target data format according to the current round model accuracy, target model accuracy, target training times and actual training times of the training model;
[0011] Determine whether the data formats of the current round of gradient data and the current round of model parameters are both in the target data format;
[0012] If not, the data format of the current round gradient data and / or the current round model parameters is converted into the target data format, and the next round model parameters of the training model are determined based on the current round gradient data and the current round model parameters in the target data format.
[0013] In one embodiment, determining the target data format according to the current round model accuracy, target model accuracy, target training times and actual training times includes:
[0014] The difference between the model accuracy of the current round and the target model accuracy is taken as the first difference;
[0015] Taking the difference between the target number of training times and the actual number of training times as the second difference;
[0016] If the ratio between the first difference and the second difference is greater than a preset single-round threshold, determining that the target data format is an integer type;
[0017] If the ratio between the first difference and the second difference is less than or equal to the preset single-round threshold, it is determined that the target data format is a floating point type.
[0018] In one embodiment, determining the target data format according to the current round model accuracy, target model accuracy, target training times and actual training times includes:
[0019] Obtaining a predicted number of training times according to the difference between the model accuracy of the current round and the target model accuracy; wherein the predicted number of training times is used to represent the number of training times that need to be performed to achieve the target model accuracy;
[0020] If the difference between the target training times and the actual training times is greater than the predicted training times, determining that the target data format is an integer type;
[0021] If the difference between the target training times and the actual training times is less than or equal to the predicted training times, the target data format is determined to be a floating point type.
[0022] In one embodiment, sending the next round model parameters to the GPU includes:
[0023] If the target data format is an integer, the next round model parameters are sent to the GPU;
[0024] If the target data format is a floating point type, the next round of model parameters are converted to integer type and sent to the GPU.
[0025] In a second aspect, the present application provides a model training method applied to a graphics processor (GPU), the method comprising:
[0026] Get the next round of model parameters of the training model sent by the CPU;
[0027] Forward propagation and back propagation are performed based on the next round model parameters to obtain the next round gradient data of the training model.
[0028] In one embodiment, performing forward propagation and back propagation based on the next round model parameters to obtain the next round gradient data of the training model includes:
[0029] If the data format of the next round model parameter is an integer type, converting the data format of the next round model parameter to a floating point type;
[0030] Forward propagation and back propagation are performed according to the next round model parameters converted to floating point type to obtain the next round gradient data of the training model.
[0031] In a third aspect, the present application further provides a model training device, the device being configured on a central processing unit (CPU), the device comprising:
[0032] A first acquisition module is used to acquire the current round gradient data of the training model from the graphics processing unit GPU; wherein the data format of the current round gradient data is an integer type;
[0033] A determination module, used to determine the model parameters of the next round of the training model according to the current round gradient data and the current round model parameters of the training model;
[0034] A sending module is used to send the next round model parameters to the GPU, so that the GPU performs forward propagation and back propagation based on the next round model parameters to obtain the next round gradient data of the training model.
[0035] In a fourth aspect, the present application further provides a model training device, the device being configured on a graphics processor (GPU), the device comprising:
[0036] The second acquisition module is used to obtain the next round of model parameters of the training model sent by the CPU;
[0037] The training module is used to perform forward propagation and backward propagation based on the next round model parameters to obtain the next round gradient data of the training model.
[0038] In a fifth aspect, the present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the processor implements the steps of any method described in the first aspect, or implements the steps of any method described in the second aspect.
[0039] In a sixth aspect, the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of any method described in the first aspect are implemented, or the steps of any method described in the second aspect are implemented. In a seventh aspect, the present application further provides a computer program product, wherein the computer program product includes a computer program, wherein when the computer program is executed by a processor, the steps of any method described in the first aspect are implemented, or the steps of any method described in the second aspect are implemented.
[0040] The above-mentioned model training method, device and computer equipment obtain the current round gradient data of the training model from the graphics processing unit GPU; the data format of the obtained current round gradient data is integer type; and the next round model parameters of the training model are determined according to the current round gradient data and the current round model parameters of the training model; and then the next round model parameters are sent to the GPU, so that the GPU performs forward propagation and backward propagation based on the next round model parameters to obtain the next round gradient data of the training model. In the above scheme, when each round of gradient data is propagated between the GPU and the CPU, the gradient data of each round is integer type, which reduces the amount of data transmission, improves the efficiency of data transmission, and thus improves the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS
[0041] Figure 1 An application environment diagram of a model training method in an embodiment;
[0042] Figure 2 A schematic diagram of a flow chart of a model training method in one embodiment;
[0043] Figure 3 A schematic diagram of a process for determining model parameters for the next round of training a model in one embodiment;
[0044] Figure 4 A schematic diagram of a process for determining a target data format in one embodiment;
[0045] Figure 5 A schematic diagram of a flow chart for determining a target data format in another embodiment;
[0046] Figure 6 A schematic diagram of a flow chart of a model training method in another embodiment;
[0047] Figure 7 A schematic diagram of a process for obtaining the next round of sub-gradient data of a training model in one embodiment;
[0048] Figure 8 is a structural block diagram of a model training device in one embodiment;
[0049] Fig. 9 is a structural block diagram of a model training device in another embodiment;
[0050] Fig.10 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0051] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0052] The model training method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, it is specifically applied to the application scenario of distributed training of large-scale models. Among them, the forward propagation and back propagation of model training are performed by the Graphic Processing Unit (GPU). Specifically, the model parameters sent by the Central Processing Unit (CPU) can be dequantized, that is, data conversion is performed to obtain floating-point model parameters; and forward propagation and back propagation are performed according to the floating-point model parameters to obtain floating-point gradient data; and the obtained gradient data is quantized, that is, data conversion is performed, and transmitted to the CPU in an integer format, and the CPU performs optimizer parameter update and model parameter update. Specifically, the CPU can first dequantize the gradient data to obtain floating-point gradient data; and update the parameters and optimizer parameters of the model according to the floating-point gradient data; and then convert the updated model parameters into integers and pass them to the GPU for the next round of model training. In this way, the data format of the data transmitted between the GPU and the CPU is an integer type, which can reduce the amount of data transmission, improve the data transmission efficiency, and thus improve the model training efficiency.
[0053] The model training method in the embodiment of the present application is executed by the GPU and the CPU. The forward propagation and back propagation of the specific model training are executed by the GPU, and the parameter update of the model is executed by the CPU.
[0054] In one embodiment, Figure 2 As shown, a model training method is provided, which is described by taking the method applied to a CPU as an example, and includes the following steps:
[0055] S201, obtaining the current round gradient data of the training model from the graphics processing unit GPU.
[0056] It is understandable that large-scale models need to store a large amount of intermediate calculation results during training. For example, Transformer models require a large amount of memory during training. In order to speed up model training, the forward propagation and backpropagation of model training can be performed by the GPU, and the model parameter update part of model training can be performed by the CPU.
[0057] Based on this, the current round gradient data of the training model can be obtained from the GPU, where the data format of the current round gradient data is an integer type, such as the data format is int8. It should be noted that in order to ensure the accuracy of the model, the gradient data obtained by the GPU is usually floating point type, for example, the gradient data format is usually fp16; transmitting floating point data will reduce the data transmission efficiency, so the GPU can convert the data format of the current round gradient data to an integer type before transmitting the current round gradient data to the CPU, and then transmit the integer type current round gradient data to the CPU.
[0058] S202, determining the model parameters of the next round of the training model according to the gradient data of this round and the model parameters of the training model of this round.
[0059] Exemplarily, the parameters of the optimizer can be updated according to the gradient data of this round, such as updating the learning rate and momentum of the optimizer; and the model parameters can be updated according to the gradient data of this round, the optimizer parameters and the model parameters of this round to obtain the model parameters of the next round of the training model.
[0060] S203, sending the next round of model parameters to the GPU, so that the GPU performs forward propagation and back propagation based on the next round of model parameters to obtain the next round of gradient data of the training model.
[0061] Exemplarily, the data format of the next round model parameters can be converted into an integer type, and the integer type next round model parameters can be sent to the GPU, so that the GPU performs forward propagation and back propagation based on the next round model parameters to obtain the next round gradient data of the training model.
[0062] The above model training method obtains the current round gradient data of the training model from the graphics processing unit GPU; the data format of the obtained current round gradient data is integer type; and the next round model parameters of the training model are determined according to the current round gradient data and the current round model parameters of the training model; and then the next round model parameters are sent to the GPU, so that the GPU performs forward propagation and backward propagation based on the next round model parameters to obtain the next round gradient data of the training model. In the above scheme, when each round of gradient data is propagated between the GPU and the CPU, the gradient data of each round is integer type, which reduces the amount of data transmission, improves the efficiency of data transmission, and thus improves the efficiency of model training.
[0063] In some optional implementations, see Figure 3 , Figure 3 A flowchart for determining the next round of model parameters of a training model is provided, which specifically includes the following steps:
[0064] S301, determining the target data format according to the model accuracy of the training model in this round, the target model accuracy, the target number of training times and the actual number of training times.
[0065] Exemplarily, the model accuracy of the current round of training model can be determined based on the difference between the predicted value and the actual value; the target model accuracy and the target number of training times can be set in advance based on actual needs; the actual number of training times is the total number of training times up to this time.
[0066] It should be noted that the distance between the current round training accuracy of the model and the target model training accuracy can be determined based on the current round model accuracy, target model accuracy, target training times and actual training times of the training model, and then the target data format can be determined based on the distance between the current round training accuracy of the model and the target model training accuracy. Among them, the target data format is the data format of the current round gradient data and the current round model parameters used to determine the next round model parameters.
[0067] S302, determine whether the data formats of the current round of gradient data and the current round of model parameters are both in the target data format; if not, execute S303, and if so, execute S304.
[0068] It can be understood that when determining the next round model parameters of the training model based on the current round gradient data and the current round model parameters, the data formats of the current round gradient data and the current round model parameters are required to be the target data format. Therefore, it can be determined whether the data formats of the current round gradient data and the current round model parameters are both in the target data format. If so, execute S304, otherwise execute S303.
[0069] S303, converting the data format of the current round gradient data and / or the current round model parameters into the target data format, and determining the next round model parameters of the training model according to the current round gradient data and the current round model parameters in the target data format.
[0070] Exemplarily, if the data formats of the current round gradient data and the current round model parameters are not both in the target data format, the data format of the current round gradient data and / or the current round model parameters can be converted into the target data format, that is, the data format of the current round gradient data and / or the current round model parameters that are not in the target data format are converted into the target data format; and the next round model parameters of the training model are determined based on the current round gradient data and the current round model parameters in the target data format.
[0071] S304, determining the next round model parameters of the training model according to the current round gradient data and the current round model parameters in the target data format.
[0072] Exemplarily, if the data formats of the current round gradient data and the current round model parameters are both in the target data format, the next round model parameters of the training model can be determined directly based on the current round gradient data and the current round model parameters in the target data format.
[0073] In the embodiment of the present application, the target data format is determined according to the model accuracy of the current round of the training model, the target model accuracy, the target number of training times and the actual number of training times; specifically, when the model training accuracy is sufficient to reach the target model accuracy in the predicted remaining number of training times, the target data format can be determined to be an integer type, which can speed up the model training speed; when the model training accuracy is insufficient to reach the target model accuracy in the predicted remaining number of training times, the target data format can be determined to be a floating point type, which can speed up the model training speed. In this way, the embodiment of the present application can ensure the training accuracy of the model on the one hand, and improve the training accuracy of the model on the other hand.
[0074] In some optional implementations, see Figure 4 , Figure 4 A flowchart for determining a target data format is provided, which specifically includes the following steps:
[0075] S401: The difference between the model accuracy of this round and the target model accuracy is taken as the first difference.
[0076] Exemplarily, the accuracy of the model in this round may be subtracted from the accuracy of the target model, and the obtained difference may be used as the first difference.
[0077] S402: Taking the difference between the target number of training times and the actual number of training times as the second difference.
[0078] Exemplarily, the target number of training times may be subtracted from the actual number of training times, and the obtained difference may be used as the second difference.
[0079] S403: If the ratio between the first difference and the second difference is greater than a preset single-round threshold, determine that the target data format is an integer type.
[0080] Exemplarily, the preset single-round threshold can be determined according to the target model accuracy and the target number of training times; if the ratio between the first difference and the second difference is greater than the preset single-round threshold, it is considered that the accuracy of the model in this round meets the accuracy requirement, and the target data format can be determined as an integer type, for example, the integer type is int8, so that the model parameters of the next round are determined using the integer type gradient data of this round and the model parameters of this round, which can not only ensure the model training accuracy, but also speed up the data processing efficiency.
[0081] S404: If the ratio between the first difference and the second difference is less than or equal to a preset single-round threshold, determine that the target data format is a floating point type.
[0082] Exemplarily, if the ratio between the first difference and the second difference is less than or equal to the preset single-round threshold, it is considered that the accuracy of the model in this round does not meet the accuracy requirement, and the target data format can be determined as a floating point type, for example, the floating point type can be fp32; in this way, the model parameters of the next round are determined using the floating point gradient data of this round and the model parameters of this round, which can further improve the model training accuracy.
[0083] The embodiment of the present application provides an achievable method for determining the target data format, which can not only ensure the training accuracy of the model but also improve the data processing efficiency.
[0084] In some optional implementations, see Figure 5 , Figure 5 Another flow chart for determining the target data format is provided, which specifically includes the following steps:
[0085] S501, obtaining the predicted number of training times according to the difference between the model accuracy of this round and the target model accuracy.
[0086] For example, based on a preset algorithm, the number of prediction training times required to reach the target model accuracy can be predicted according to the difference between the model accuracy of this round and the target model accuracy. The predicted number of training times is used to represent the number of training times required to reach the target model accuracy.
[0087] S502: If the difference between the target training times and the actual training times is greater than the predicted training times, the target data format is determined to be an integer.
[0088] Exemplarily, the target number of training times is subtracted from the actual number of training times, and the difference is used as the actual remaining number of training times. If the actual remaining number of training times is greater than the predicted number of training times, it means that the accuracy of the model training in this round has met the accuracy requirements, and the target data format can be determined as an integer. Determining the target data format as an integer can improve data processing efficiency.
[0089] S503: If the difference between the target training times and the actual training times is less than or equal to the predicted training times, determine that the target data format is a floating point type.
[0090] Furthermore, if the actual remaining number of training times is less than or equal to the predicted number of training times, it means that the accuracy of the model training in this round does not meet the accuracy requirements, and the target data format can be determined as a floating point type. Determining the target data format as a floating point type can improve the model training accuracy.
[0091] The embodiment of the present application provides an achievable method for determining the target data format, which can not only ensure the training accuracy of the model but also improve the data processing efficiency.
[0092] In some optional implementations, if the target data format is an integer type, the next round model parameters can be directly sent to the GPU. If the target data format is a floating point type, the next round model parameters can be converted to integer type before being sent to the GPU. For example, if the floating point type is fp32, the next round model parameters in fp32 format can be converted to fp16 format, and then fp16 is quantized to obtain integer type next round model parameters, such as int8 format.
[0093] In an embodiment of the present application, before sending the next round of model parameters to the GPU, the data format of the next round of model parameters is first adjusted to an integer type. The data transmission efficiency of transmitting integer type data is higher than the data transmission efficiency of transmitting floating-point type data, thereby improving the data transmission efficiency between the GPU and the CPU.
[0094] In some optional implementations, see Figure 6 , Figure 6 Another model training method is provided. Taking this method applied to GPU as an example, it includes the following steps:
[0095] S601, obtaining the next round model parameters of the training model sent by the CPU.
[0096] Exemplarily, in the above embodiment, the CPU sends the next round model parameters of the training model to the GPU, so that the GPU can obtain the next round model parameters of the training model sent by the CPU.
[0097] S602, forward propagation and back propagation are performed based on the next round of model parameters to obtain the next round of gradient data of the training model.
[0098] Furthermore, the GPU can perform forward propagation and back propagation based on the next round of model parameters to obtain the next round of gradient data for the training model.
[0099] In an embodiment of the present application, a model training method is provided, in which the forward propagation and back propagation of the model training are performed by a GPU, which can improve the utilization rate of the GPU and thus improve the efficiency of the model training.
[0100] Furthermore, in the above embodiment, the data format of the next round of gradient data of the training model can be an integer type. In order to improve data transmission efficiency, the CPU needs to convert the data format of the next round of gradient data of the training model into an integer type before sending the next round of gradient data of the training model to the GPU to improve data transmission efficiency.
[0101] Based on this, see Figure 7 , Figure 7 A flowchart for obtaining the next round of subgradient data for a training model is provided, which specifically includes the following steps:
[0102] S701: If the data format of the next round model parameter is an integer type, convert the data format of the next round model parameter to a floating point type.
[0103] Exemplarily, if the data format of the next round model parameters is integer type, the data format of the next round model parameters can be converted to floating point type, so as to train the model according to the next round model parameters with floating point data format to improve the training accuracy of the model.
[0104] S702, performing forward propagation and backward propagation according to the next round model parameters converted to floating point type to obtain the next round gradient data of the training model.
[0105] In some optional implementations, forward propagation and back propagation can be performed based on the next round model parameters converted to floating point type to obtain the next round gradient data of the training model, thereby improving the accuracy of the next round gradient data obtained.
[0106] In an embodiment of the present application, after obtaining the integer type of next round model parameters, the next round model parameters are first converted to floating point type, and the next round gradient data is determined based on the floating point type of next round model parameters, which can improve the accuracy of the obtained next round gradient data and improve the training accuracy of the model.
[0107] In the above embodiments, the computational complexity of forward propagation and backward propagation is approximately O(MB), the computational complexity of parameter update is O(B), and the precision conversion from fp32 to fp16 is O(M), where M is the number of parameters of the model and B is the number of samples. In order not to reduce the data processing efficiency, forward propagation and backward propagation are placed in the GPU. Parameter update and precision conversion not only have small computational complexity but also need to be related to the state of the optimizer, so they are placed on the CPU. The state of the optimizer is also placed in the CPU. In this way, the entire process is to first perform forward and backward calculations on the GPU, pass the gradient data to the CPU, update the model parameters, and then pass the updated model parameters to the GPU to facilitate the next round of forward propagation and backward propagation.
[0108] In the embodiments of the present application, transformer structure models with 13B parameters and 33B parameters were used for training, and the training time of the models of the same size using quantization modules and dequantization modules on a single card with 80G video memory was compared. It was found that the training time of the 13B model was 5% longer after adding quantization and dequantization modules than without these two modules, and the training time was 73% faster after adding quantization and dequantization modules to the 33B model. It can be seen that the communication performance between the CPU and the GPU has a great impact on the model training efficiency when the model scale is large. In the above embodiments, integer data is transmitted between the CPU and the GPU, which improves the data transmission efficiency and thus improves the model training efficiency.
[0109] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0110] Based on the same inventive concept, the embodiment of the present application also provides a model training device for implementing the model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more model training device embodiments provided below can refer to the limitations on the model training method above, and will not be repeated here.
[0111] In one embodiment, Figure 8 As shown, a model training device is provided, the device is configured on a central processing unit (CPU), and includes:
[0112] The first acquisition module 10 is used to obtain the current round gradient data of the training model from the graphics processing unit GPU; wherein the data format of the current round gradient data is an integer type;
[0113] A determination module 20, configured to determine the model parameters of the next round of the training model according to the current round gradient data and the current round model parameters of the training model;
[0114] The sending module 30 is used to send the next round model parameters to the GPU, so that the GPU performs forward propagation and backward propagation based on the next round model parameters to obtain the next round gradient data of the training model.
[0115] The above-mentioned model training device obtains the current round gradient data of the training model from the graphics processing unit GPU; the data format of the obtained current round gradient data is integer type; and according to the current round gradient data and the current round model parameters of the training model, the next round model parameters of the training model are determined; and then the next round model parameters are sent to the GPU, so that the GPU performs forward propagation and backward propagation based on the next round model parameters to obtain the next round gradient data of the training model. In the above scheme, when each round of gradient data is transmitted between the GPU and the CPU, the gradient data of each round is integer type, which reduces the amount of data transmission, improves the efficiency of data transmission, and thus improves the efficiency of model training.
[0116] In one embodiment, the determination module 20 specifically includes:
[0117] A first determining unit is used to determine a target data format according to the current round model accuracy, target model accuracy, target training times and actual training times of the training model;
[0118] A judging unit, used to judge whether the data formats of the current round of gradient data and the current round of model parameters are both in the target data format;
[0119] The second determination unit is used to convert the data format of the current round gradient data and / or the current round model parameters into the target data format if not, and determine the next round model parameters of the training model according to the current round gradient data and the current round model parameters in the target data format.
[0120] In one embodiment, the determination module 20 is specifically configured to:
[0121] The difference between the model accuracy of this round and the target model accuracy is taken as the first difference; the difference between the target number of training times and the actual number of training times is taken as the second difference; if the ratio between the first difference and the second difference is greater than the preset single-round threshold, the target data format is determined to be an integer; if the ratio between the first difference and the second difference is less than or equal to the preset single-round threshold, the target data format is determined to be a floating point type.
[0122] In one embodiment, the determination module 20 is specifically configured to:
[0123] According to the difference between the model accuracy of this round and the target model accuracy, the predicted number of training times is obtained; wherein the predicted number of training times is used to characterize the number of training times required to achieve the target model accuracy; if the difference between the target number of training times and the actual number of training times is greater than the predicted number of training times, the target data format is determined to be an integer type; if the difference between the target number of training times and the actual number of training times is less than or equal to the predicted number of training times, the target data format is determined to be a floating point type.
[0124] In one embodiment, the sending module 30 is specifically used for:
[0125] If the target data format is an integer, the next round model parameters are sent to the GPU; if the target data format is a floating point, the next round model parameters are converted to integers and then sent to the GPU.
[0126] In one embodiment, Fig. 9 As shown, another model training device is provided, wherein the device is configured on a graphics processor GPU, and comprises:
[0127] The second acquisition module 40 is used to obtain the next round of model parameters of the training model sent by the CPU;
[0128] The training module 50 is used to perform forward propagation and backward propagation based on the next round model parameters to obtain the next round gradient data of the training model.
[0129] The above-mentioned model training device performs the forward propagation and back propagation of the model training by the GPU, which can improve the utilization rate of the GPU and further improve the efficiency of the model training.
[0130] In one embodiment, the training module 50 is specifically used to:
[0131] If the data format of the next round model parameters is an integer, the data format of the next round model parameters is converted to a floating point type; forward propagation and backward propagation are performed according to the next round model parameters converted to a floating point type to obtain the next round gradient data of the training model.
[0132] Each module in the above-mentioned model training device can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory in the computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0133] In one embodiment, a computer device is provided. The computer device may include a CPU and a GPU. The internal structure diagram thereof may be as follows: Fig.10 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data for training models. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a model training method is implemented.
[0134] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0135] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0136] Obtain the current round of gradient data of the training model from the graphics processor GPU; wherein the data format of the current round of gradient data is an integer type;
[0137] Determine the next round of model parameters of the training model based on the current round of gradient data and the current round of model parameters of the training model;
[0138] The next round of model parameters are sent to the GPU so that the GPU performs forward propagation and back propagation based on the next round of model parameters to obtain the next round of gradient data of the training model.
[0139] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0140] Determine the target data format according to the current round model accuracy, target model accuracy, target training times and actual training times of the training model; determine whether the data formats of the current round gradient data and the current round model parameters are both in the target data format; if not, convert the data format of the current round gradient data and / or the current round model parameters into the target data format, and determine the next round model parameters of the training model according to the current round gradient data and the current round model parameters in the target data format.
[0141] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0142] The difference between the model accuracy of this round and the target model accuracy is taken as the first difference; the difference between the target number of training times and the actual number of training times is taken as the second difference; if the ratio between the first difference and the second difference is greater than the preset single-round threshold, the target data format is determined to be an integer; if the ratio between the first difference and the second difference is less than or equal to the preset single-round threshold, the target data format is determined to be a floating point.
[0143] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0144] According to the difference between the model accuracy of this round and the target model accuracy, the predicted number of training times is obtained; wherein the predicted number of training times is used to characterize the number of training times required to achieve the target model accuracy; if the difference between the target number of training times and the actual number of training times is greater than the predicted number of training times, the target data format is determined to be an integer type; if the difference between the target number of training times and the actual number of training times is less than or equal to the predicted number of training times, the target data format is determined to be a floating point type.
[0145] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0146] If the target data format is integer, the next round of model parameters will be sent to the GPU; if the target data format is floating point, the next round of model parameters will be converted to integers and sent to the GPU.
[0147] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the following steps are implemented:
[0148] Get the next round of model parameters of the training model sent by the CPU;
[0149] Forward propagation and back propagation are performed based on the next round of model parameters to obtain the next round of gradient data for the training model.
[0150] In one embodiment, when the processor executes the computer program, the processor further implements the following steps:
[0151] If the data format of the next round model parameters is integer type, the data format of the next round model parameters is converted to floating point type; forward propagation and back propagation are performed according to the next round model parameters converted to floating point type to obtain the next round gradient data of the training model.
[0152] In one embodiment, a computer readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0153] Obtain the current round of gradient data of the training model from the graphics processor GPU; wherein the data format of the current round of gradient data is an integer type;
[0154] Determine the next round of model parameters of the training model based on the current round of gradient data and the current round of model parameters of the training model;
[0155] The next round of model parameters are sent to the GPU so that the GPU performs forward propagation and back propagation based on the next round of model parameters to obtain the next round of gradient data of the training model.
[0156] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0157] Determine the target data format according to the current round model accuracy, target model accuracy, target training times and actual training times of the training model; determine whether the data formats of the current round gradient data and the current round model parameters are both in the target data format; if not, convert the data format of the current round gradient data and / or the current round model parameters into the target data format, and determine the next round model parameters of the training model according to the current round gradient data and the current round model parameters in the target data format.
[0158] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0159] The difference between the model accuracy of this round and the target model accuracy is taken as the first difference; the difference between the target number of training times and the actual number of training times is taken as the second difference; if the ratio between the first difference and the second difference is greater than the preset single-round threshold, the target data format is determined to be an integer; if the ratio between the first difference and the second difference is less than or equal to the preset single-round threshold, the target data format is determined to be a floating point.
[0160] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0161] According to the difference between the model accuracy of this round and the target model accuracy, the predicted number of training times is obtained; wherein the predicted number of training times is used to characterize the number of training times required to achieve the target model accuracy; if the difference between the target number of training times and the actual number of training times is greater than the predicted number of training times, the target data format is determined to be an integer type; if the difference between the target number of training times and the actual number of training times is less than or equal to the predicted number of training times, the target data format is determined to be a floating point type.
[0162] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0163] If the target data format is integer, the next round of model parameters will be sent to the GPU; if the target data format is floating point, the next round of model parameters will be converted to integers and sent to the GPU.
[0164] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented:
[0165] Get the next round of model parameters of the training model sent by the CPU;
[0166] Forward propagation and back propagation are performed based on the next round of model parameters to obtain the next round of gradient data for the training model.
[0167] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0168] If the data format of the next round model parameters is integer type, the data format of the next round model parameters is converted to floating point type; forward propagation and back propagation are performed according to the next round model parameters converted to floating point type to obtain the next round gradient data of the training model.
[0169] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0170] Obtain the current round of gradient data of the training model from the graphics processor GPU; wherein the data format of the current round of gradient data is an integer type;
[0171] Determine the next round of model parameters of the training model based on the current round of gradient data and the current round of model parameters of the training model;
[0172] The next round of model parameters are sent to the GPU so that the GPU performs forward propagation and back propagation based on the next round of model parameters to obtain the next round of gradient data of the training model.
[0173] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0174] Determine the target data format according to the current round model accuracy, target model accuracy, target training times and actual training times of the training model; determine whether the data formats of the current round gradient data and the current round model parameters are both in the target data format; if not, convert the data format of the current round gradient data and / or the current round model parameters into the target data format, and determine the next round model parameters of the training model according to the current round gradient data and the current round model parameters in the target data format.
[0175] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0176] The difference between the model accuracy of this round and the target model accuracy is taken as the first difference; the difference between the target number of training times and the actual number of training times is taken as the second difference; if the ratio between the first difference and the second difference is greater than the preset single-round threshold, the target data format is determined to be an integer; if the ratio between the first difference and the second difference is less than or equal to the preset single-round threshold, the target data format is determined to be a floating point.
[0177] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0178] According to the difference between the model accuracy of this round and the target model accuracy, the predicted number of training times is obtained; wherein the predicted number of training times is used to characterize the number of training times required to achieve the target model accuracy; if the difference between the target number of training times and the actual number of training times is greater than the predicted number of training times, the target data format is determined to be an integer type; if the difference between the target number of training times and the actual number of training times is less than or equal to the predicted number of training times, the target data format is determined to be a floating point type.
[0179] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0180] If the target data format is integer, the next round of model parameters will be sent to the GPU; if the target data format is floating point, the next round of model parameters will be converted to integers and sent to the GPU.
[0181] In one embodiment, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the following steps:
[0182] Get the next round of model parameters of the training model sent by the CPU;
[0183] Forward propagation and back propagation are performed based on the next round of model parameters to obtain the next round of gradient data for the training model.
[0184] In one embodiment, when the computer program is executed by a processor, the following steps are also implemented:
[0185] If the data format of the next round model parameters is integer type, the data format of the next round model parameters is converted to floating point type; forward propagation and back propagation are performed according to the next round model parameters converted to floating point type to obtain the next round gradient data of the training model.
[0186] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0187] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0188] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A model training method, characterized in that: Applied to a central processing unit (CPU), the method comprises: Obtaining the current round of gradient data of the training model from the graphics processing unit (GPU); wherein the data format of the current round of gradient data is an integer type; Determining the model parameters of the next round of the training model according to the current round gradient data and the current round model parameters of the training model; The next round model parameters are sent to the GPU, so that the GPU performs forward propagation and back propagation based on the next round model parameters to obtain the next round gradient data of the training model.
2. The method according to claim 1, characterized in that: Determining the next round model parameters of the training model according to the current round gradient data and the current round model parameters of the training model includes: Determine the target data format according to the current round model accuracy, target model accuracy, target training times and actual training times of the training model; Determine whether the data formats of the current round of gradient data and the current round of model parameters are both in the target data format; If not, the data format of the current round gradient data and / or the current round model parameters is converted into the target data format, and the next round model parameters of the training model are determined based on the current round gradient data and the current round model parameters in the target data format.
3. The method according to claim 2, characterized in that The determining of the target data format according to the model accuracy of the current round, the target model accuracy, the target number of training times and the actual number of training times includes: The difference between the model accuracy of the current round and the target model accuracy is taken as the first difference; Taking the difference between the target number of training times and the actual number of training times as the second difference; If the ratio between the first difference and the second difference is greater than a preset single-round threshold, determining that the target data format is an integer type; If the ratio between the first difference and the second difference is less than or equal to the preset single-round threshold, it is determined that the target data format is a floating point type.
4. The method according to claim 2, characterized in that: The determining of the target data format according to the model accuracy of the current round, the target model accuracy, the target number of training times and the actual number of training times includes: Obtaining a predicted number of training times according to the difference between the model accuracy of the current round and the target model accuracy; wherein the predicted number of training times is used to represent the number of training times that need to be performed to achieve the target model accuracy; If the difference between the target training times and the actual training times is greater than the predicted training times, determining that the target data format is an integer type; If the difference between the target training times and the actual training times is less than or equal to the predicted training times, the target data format is determined to be a floating point type.
5. The method according to claim 2, characterized in that: The sending the next round model parameters to the GPU includes: If the target data format is an integer, the next round model parameters are sent to the GPU; If the target data format is a floating point type, the next round of model parameters are converted to integer type and sent to the GPU.
6. A model training method, characterized in that: Applied to a graphics processor (GPU), the method comprises: Get the next round of model parameters of the training model sent by the CPU; Forward propagation and back propagation are performed based on the next round model parameters to obtain the next round gradient data of the training model.
7. The method according to claim 6, characterized in that The forward propagation and the backward propagation are performed based on the next round model parameters to obtain the next round gradient data of the training model, including: If the data format of the next round model parameter is an integer type, converting the data format of the next round model parameter to a floating point type; Forward propagation and back propagation are performed according to the next round model parameters converted to floating point type to obtain the next round gradient data of the training model.
8. A model training device, characterized in that: The device is configured on a central processing unit (CPU), and comprises: A first acquisition module is used to acquire the current round gradient data of the training model from the graphics processing unit GPU; wherein the data format of the current round gradient data is an integer type; A determination module, used to determine the model parameters of the next round of the training model according to the current round gradient data and the current round model parameters of the training model; A sending module is used to send the next round model parameters to the GPU, so that the GPU performs forward propagation and back propagation based on the next round model parameters to obtain the next round gradient data of the training model.
9. A model training device, characterized in that: The device is configured on a graphics processor (GPU), and comprises: The second acquisition module is used to obtain the next round of model parameters of the training model sent by the CPU; The training module is used to perform forward propagation and backward propagation based on the next round model parameters to obtain the next round gradient data of the training model.
10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the processor implements the steps of the method according to any one of claims 1 to 5, or implements the steps of the method according to any one of claims 6 to 7.
