Model training method, task processing method, edge end device, medium and product

Abstract

The invention belongs to the technical field of deep learning, and discloses a model training method, a task processing method, an edge end device, a medium and a product, and the method comprises the steps: dividing model parameters in a two-dimensional matrix form in a deep neural network model into update stopping parameters and update continuing parameters after entering a current training round; deleting the historical gradient information of the parameters which stop updating, and updating the parameters which continue to be updated based on the historical gradient information of the parameters which continue to be updated; calculating a current loss value of the deep neural network model after parameter updating of the current training round is completed; determining whether training is completed by using the current loss value; if so, ending the training; and if not, entering the next training round. The historical gradient information of the parameters which stop updating is deleted to reduce video memory occupation, and the calculation amount is reduced only by updating the parameters which continue to be updated. According to the invention, on the premise that privacy protection is not violated, effective fine tuning of the deep neural network model of the edge end equipment is realized.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to model training methods, task processing methods, edge devices, media and products. Background Art

[0002] Edge devices (DNN, Deep Neural Network) have limited computing resources, such as CPU, GPU, and memory. These limitations make DNN training at the edge extremely challenging. In addition, the energy supply of edge devices is relatively limited, and long training processes may cause the battery to run out quickly. These hardware and energy limitations make it difficult for edge devices to take on the heavy task of DNN training.

[0003] Although edge devices are resource-constrained, fine-tuning DNN models to adapt to the needs of specific users or environments is necessary in some application scenarios, such as personalized recommendations and localized services. However, the fine-tuning process often requires collecting and processing sensitive user data, which conflicts with the increasingly stringent privacy protection regulations and users' concerns about personal privacy.

[0004] How to achieve effective fine-tuning of edge DNN without violating privacy protection is a technical problem that technicians in this field urgently need to solve. Summary of the invention

[0005] The purpose of this application is to provide a model training method, a task processing method, an edge device, a medium, and a product to train the DNN directly on the edge device, thereby achieving effective fine-tuning of the DNN and meeting the privacy protection requirements.

[0006] In order to solve the above technical problems, this application provides the following technical solutions:

[0007] A model training method, applied to an edge device, comprising:

[0008] After entering the current training round, the model parameters in the form of two-dimensional matrices in the deep neural network model are divided into stop updating parameters and continue updating parameters;

[0009] Deleting the historical gradient information of the stop updating parameter, and updating the continue updating parameter based on the historical gradient information of the continue updating parameter;

[0010] After completing the parameter update of the current training round, calculating the current loss value of the deep neural network model;

[0011] Use the current loss value to determine whether training is complete;

[0012] If yes, the training ends; if no, the next training round begins.

[0013] Preferably, updating the continued updating parameter based on the historical gradient information of the continued updating parameter comprises:

[0014] Finding historical gradient information of the continuously updated parameters;

[0015] If not found, using the continued updating parameter and gradient to determine the historical gradient information of the continued updating parameter;

[0016] The continuous updating parameter is updated based on the historical gradient information.

[0017] Preferably, using the continuously updated parameter and the gradient to determine the historical gradient information of the continuously updated parameter includes:

[0018] The historical gradient information of the continuously updated parameter is determined by utilizing the parameter update rule and combining the continuously updated parameter and the gradient.

[0019] Preferably, calculating the current loss value of the deep neural network model includes:

[0020] Using the predicted probability obtained from training and the training samples, calculate the cross entropy loss value;

[0021] Calculating a sparsity loss value based on a parameter state of the neural network model;

[0022] The cross entropy loss value and the sparsity loss value are superimposed to obtain the current loss value.

[0023] Preferably, the calculating of the sparsity loss value based on the parameter state of the neural network model includes:

[0024] Obtaining the total number of model parameters;

[0025] Obtaining parameter values ​​of the model parameters, and determining a target number of non-zero model parameters based on the parameter values;

[0026] Calculating the ratio of the target number to the total number;

[0027] The ratio is determined as the sparsity loss value.

[0028] Preferably, the cross entropy loss value is calculated using the predicted probability obtained through training and the training samples, including:

[0029] Obtaining the sample quantity, category quantity and true label of the training sample;

[0030] Obtaining the predicted probability using a normalization layer in the deep neural network model;

[0031] The cross entropy loss value is calculated by using a cross entropy loss function and combining the number of samples, the number of categories, the true label and the predicted probability.

[0032] Preferably, deleting the historical gradient information of stopping updating the parameter includes:

[0033] The first-order moment estimate and the second-order moment estimate of the stop updating parameter are set to zero.

[0034] Preferably, the model parameters in the form of two-dimensional matrices in the deep neural network model are divided into stop updating parameters and continue updating parameters, including:

[0035] Adding auxiliary parameters corresponding to the model parameters; wherein the auxiliary parameters are structured representations of the model parameters;

[0036] determining a control state of the model parameter using the auxiliary parameter;

[0037] By using the control state, the model parameters are divided into stop updating parameters and continue updating parameters.

[0038] Preferably, using the auxiliary parameter to determine the control state of the model parameter includes:

[0039] If the auxiliary parameter is less than or equal to zero, determining that the control state of the corresponding model parameter is to stop updating;

[0040] If the auxiliary parameter is greater than zero, it is determined that the control state of the corresponding model parameter is to continue updating.

[0041] Preferably, updating the continued updating parameter based on the historical gradient information of the continued updating parameter comprises:

[0042] Using the auxiliary parameters to obtain a sparse structure of the gradient;

[0043] Structural simplification of parameter matrices and gradients;

[0044] When performing general matrix multiplication, the simplified parameter matrix and gradient are used to perform calculations to obtain calculation results;

[0045] The calculation result is padded with zeros to keep the output dimension unchanged.

[0046] Preferably, the parameter matrix and gradient are structurally simplified, including:

[0047] Obtaining the sparse types corresponding to the parameter matrix and the gradient respectively;

[0048] If the sparse type is a row sparse parameter, the columns at the sparse positions are compressed;

[0049] If the sparse type is a column sparse parameter, the rows at the sparse position are compressed;

[0050] After completing row and column compression, the simplified parameter matrix and gradient are obtained.

[0051] A model training device, comprising:

[0052] A parameter classification module is used to classify the model parameters in the form of two-dimensional matrices in the deep neural network model into parameters to be stopped from updating and parameters to be continued to update after entering the current training round;

[0053] A parameter updating module, used for deleting the historical gradient information of the stop updating parameter, and updating the continue updating parameter based on the historical gradient information of the continue updating parameter;

[0054] A loss calculation module, used to calculate the current loss value of the deep neural network model after completing the parameter update of the current training round;

[0055] The direction judgment module is used to use the current loss value to determine whether the training is completed; if yes, the training is ended; if not, the next training round is entered.

[0056] A task processing method, applied to an edge device, comprising:

[0057] Get the deep neural network model from the server;

[0058] Loading the deep neural network model to a local training platform;

[0059] Loading the private data required for training to the local training platform, so that the local training platform can train and fine-tune the deep neural network model based on the private data according to the above-mentioned model training method to obtain a target model;

[0060] Obtain a target task, and process the target task using the target model to obtain a processing result.

[0061] A task processing device, applied to an edge device, comprising:

[0062] Model acquisition module, used to obtain the deep neural network model from the server;

[0063] A model loading module, used to load the deep neural network model into a local training platform;

[0064] A model training module, used to load the private data required for training into the local training platform, so that the local training platform can train and fine-tune the deep neural network model based on the private data according to the model training method as described above to obtain a target model;

[0065] The task processing module is used to obtain the target task, process the target task using the target model, and obtain a processing result.

[0066] An edge device, comprising:

[0067] Memory for storing computer programs;

[0068] A processor is used to implement the steps of the above-mentioned model training method when executing the computer program, or to implement the steps of the above-mentioned model training method when executing the computer program.

[0069] A readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the above-mentioned model training method, or, when executed by a processor, implements the steps of the above-mentioned task processing method.

[0070] A computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of the above-mentioned model training method or the steps of the above-mentioned task processing method.

[0071] The method provided by the embodiment of the present application is applied in the edge device. After entering the current training round, the model parameters in the form of a two-dimensional matrix in the deep neural network model are divided into stop updating parameters and continue updating parameters; the historical gradient information of the stop updating parameters is deleted, and the continue updating parameters are updated based on the historical gradient information of the continue updating parameters; after completing the parameter update of the current training round, the current loss value of the deep neural network model is calculated; the current loss value is used to determine whether the training is completed; if yes, the training is ended; if not, the next training round is entered.

[0072] In this application, in order to avoid violating privacy protection, the deep neural network model is trained directly in the edge device. In order to realize the training of the deep neural network model in the edge device, after determining the current training round, the model parameters in the two-dimensional matrix form of the deep neural network model are first divided into stop updating parameters and continue updating parameters. The historical gradient information of the stop updating parameters is deleted, and only the parameters are updated for the continue updating parameters. Then the current loss value is calculated, that is, the loss value situation is determined to determine the next training direction. Because the model parameters in the two-dimensional matrix form are first classified before the parameter update, and the historical gradient information of the stop updating parameters is deleted, on the one hand, the video memory occupancy can be effectively reduced, and on the other hand, only the update processing is performed for the continue updating parameters, and the amount of calculation can also be greatly reduced. In this way, the training of the deep neural network model can be realized in the edge device with limited resources. Training the deep neural network in the edge device means that the privacy data information required for training can be directly obtained locally, without the need to propagate the privacy data information or collect it across devices, which may violate privacy protection. This application can achieve effective fine-tuning of the deep neural network model of the edge device without violating privacy protection.

[0073] Correspondingly, the embodiments of the present application also provide a task processing method, a model training device, a task processing device, an edge device, a readable storage medium and a computer program product corresponding to the above-mentioned model training method, which have the above-mentioned technical effects and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the related technologies, the drawings required for use in the embodiments or the related technical descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0075] Figure 1 This is a flowchart of an implementation of a model training method in an embodiment of the present application;

[0076] Figure 2 A schematic diagram of the amount of calculation in a model training method in an embodiment of the present application;

[0077] Figure 3 This is a schematic diagram of the structure of a model training device in an embodiment of the present application;

[0078] Figure 4 This is a schematic diagram of the structure of an edge device in an embodiment of the present application;

[0079] Figure 5 This is a schematic diagram of a specific structure of an edge device in an embodiment of the present application;

[0080] Figure 6 This is a flowchart of an implementation method of a task processing method in an embodiment of the present application;

[0081] Figure 7 It is a structural diagram of a task processing device in an embodiment of the present application. DETAILED DESCRIPTION

[0082] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0083] The training process of deep neural networks is usually concentrated on large servers with sufficient resources. However, although this centralized training method is more efficient, it also brings problems such as high data transmission costs, poor real-time performance, and difficulty in privacy protection. In contrast, edge devices such as smartphones and IoT (Internet of Things) devices, although they play an important role in reasoning (i.e. model application), are rarely involved in training.

[0084] The model training method proposed in this application can realize the training of DNN models on edge devices. For details, please refer to Figure 1 , Figure 1 This is a flow chart of a model training method in an embodiment of the present application. The method can be applied to edge devices (such as smart phones, IoT devices, etc.). The method includes the following steps:

[0085] S101. After entering the current training round, the model parameters in the form of two-dimensional matrices in the deep neural network model are divided into stop updating parameters and continue updating parameters.

[0086] In this embodiment, the deep neural network model is specifically a model with model parameters in the form of a two-dimensional matrix, for example, a model with a Transformer, which is a model that uses an attention mechanism to improve the model training speed. For a model with a Transformer structure, the main parameters of the model are composed of a two-dimensional matrix, so the model parameters that need to be updated at the current moment can be found by making a structured selection of the matrix blocks, while retaining the structured model parameter representation, which is conducive to subsequent accelerated calculations on the Jetson platform.

[0087] In practical applications, the deep neural network model can be specifically used as a model for common deep task analysis and processing such as image classification processing, text classification processing, and video processing.

[0088] It is understandable that the deep neural network model often needs to be iterated through multiple training rounds. In this embodiment, each training iteration can be performed according to Figure 1 Follow the steps shown.

[0089] After entering the current training, the model parameters in the two-dimensional matrix form in the deep neural network model are first divided into stop updating parameters and continue updating parameters.

[0090] For example, if there are Nall model parameters in the form of two-dimensional matrices in the deep neural network model, these Nall model parameters are classified one by one to determine the parameters that need to be updated in this round of training, that is, continue to update the parameters, and the parameters that do not need to be updated in this round of training, that is, stop updating the parameters.

[0091] In specific implementation, the classification of model parameters can be determined by random selection, or the classification can be determined based on the value of the model parameters.

[0092] In a specific implementation of the present application, before executing step S101, the following steps may also be performed:

[0093] Send model acquisition request to large server;

[0094] A deep neural network model that receives feedback from a large server;

[0095] Load the deep neural network model to the local training platform;

[0096] The private data required for training is loaded into the local training platform so that the local training platform can train and fine-tune the deep neural network model based on the private data.

[0097] That is to say, in this embodiment, the deep neural network model can be pre-trained on a large server for general purposes. If an edge device needs to run the deep neural network model, it can be obtained from the large server and further fine-tuned based on local privacy data and in combination with a local training platform (such as the Jetson platform, i.e., an embedded artificial intelligence computing platform). In this way, local privacy data protection is achieved and effective fine-tuning of the model is achieved. For specific information on how to obtain privacy data locally and how to train and fine-tune the model based on these privacy data, please refer to the model training implementation, which will not be described in detail here.

[0098] Among them, the details of training fine-tuning can be referred to Figure 1 The operation steps shown will not be repeated here one by one.

[0099] S102: Delete the historical gradient information of the stop updating parameter, and update the continue updating parameter based on the historical gradient information of the continue updating parameter.

[0100] The update of model parameters needs to be based on the current gradient and historical gradient information. For the historical gradient information of the parameters that do not need to be updated, they can be deleted directly to reduce the memory usage. It is understandable that with the iteration of training rounds, the parameters that need to be updated will become fewer and fewer, the memory usage will become lower and lower, and the computing resources required for training will also become lower and lower. The historical gradient information of the parameters that continue to be updated needs to be retained, and the parameters that continue to be updated are updated based on the historical gradient information.

[0101] Specifically, the Adam optimizer can dynamically adjust the learning rate of each parameter by estimating the first-order moment (mean) and second-order moment (variance) of the gradient. For the update of model parameters at time step t, that is, for the update process of continuing to update parameters, the details are as follows:

[0102] First calculate the gradient :

[0103] , where Loss is the loss function, Represents the derivative of Loss with respect to W, t represents the current moment, and t-1 represents the previous moment.

[0104] Update the matrix estimation section and the second-order matrix estimate :

[0105] ;

[0106] Among them, m represents the estimated value of the first-order moment, v represents the estimated value of the second-order moment, and β1 and β2 are manually set weight parameters.

[0107] The formula for updating the continuous update parameters is:

[0108] ,in, is the learning rate, is a small constant that prevents the denominator from being zero.

[0109] In a specific implementation of the present application, deleting the historical gradient information of stopping updating the parameters includes:

[0110] Set the first-order moment estimate and the second-order moment estimate of the parameter that stops updating to zero. That is, for the parameter that stops updating, set its first-order moment estimate and the second-order moment estimate to zero, thereby reducing the memory usage. If the parameter that stops updating no longer has historical gradient information, it can be skipped.

[0111] In a specific implementation of the present application, the continued update parameter is updated based on the historical gradient information of the continued update parameter, including:

[0112] Find the historical gradient information to continue updating the parameters;

[0113] If not found, the historical gradient information of the parameters to be updated is determined by using the parameters and gradients to be updated continuously;

[0114] The parameters are updated based on the historical gradient information.

[0115] The historical gradient information of the parameters to be updated is determined by using the parameters and gradients to be updated continuously, including:

[0116] The parameter update rule is used in combination with the continued update of the parameters and the gradient to determine the historical gradient information for the continued update of the parameters.

[0117] For ease of description, the above steps are combined for explanation below.

[0118] Considering different training rounds, the classification of whether to continue training and updating for the same model parameter may be different. For example, in the previous round of training, model parameter a is determined as a parameter to stop updating, and in the next round of training, model parameter a is determined as a parameter to continue updating. In this way, in each round of training, some parameters to continue updating may appear, and their corresponding historical gradient information has been deleted. In order to allow the model parameters in this case to be updated, the missing historical gradient information can be restored in this embodiment.

[0119] Specifically, when it is necessary to restore historical gradient information, assuming that the current parameter is , the current gradient is , the current information can be used to approximate the historical gradient. For example, the current gradient and the parameters at the previous moment can be used to estimate the historical gradient. Assume that the parameters at the previous moment are , the historical gradient can be estimated using the following formula :

[0120] ,in, Represents the i-th parameter in the DNN model The gradient value at time t; Indicates the matrix parameter for the model i , the second-order moment estimate retained at the previous moment; β1 is the manually set weight parameter; α is the learning rate; represents a small constant that prevents the denominator from being zero.

[0121] That is, in this embodiment, the items in the Adam (Adaptive Moment Estimation Optimizer) update rule are used to restore the historical gradient.

[0122] After the historical gradient information is restored, the parameters can be updated based on the restored historical gradient information. In other words, even if the historical gradient information is deleted in the previous round, the model parameters can continue to be updated, and deleting the historical gradient information will not have a substantial negative impact on the parameter update.

[0123] S103. After completing the parameter update of the current training round, calculate the current loss value of the deep neural network model.

[0124] After completing the update process of all the parameters, the current loss value of the deep neural network model can be calculated.

[0125] In a specific implementation of the present application, calculating the current loss value of the deep neural network model includes:

[0126] Using the predicted probability obtained from training and the training samples, calculate the cross entropy loss value;

[0127] Based on the parameter state of the neural network model, the sparsity loss value is calculated;

[0128] Superimpose the cross entropy loss value and the sparsity loss value to get the current loss value.

[0129] Among them, based on the parameter state of the neural network model, the sparsity loss value is calculated, including:

[0130] Get the total number of model parameters;

[0131] Obtain parameter values ​​of model parameters, and determine a target number of non-zero model parameters based on the parameter values;

[0132] Calculate the ratio of the target number to the total number;

[0133] The ratio is determined as the sparsity loss value.

[0134] Among them, the cross entropy loss value is calculated using the predicted probability obtained from training and the training samples, including:

[0135] Get the number of samples, number of categories and true labels of training samples;

[0136] Use the normalization layer in the deep neural network model to obtain the prediction probability;

[0137] The cross entropy loss function is used to calculate the cross entropy loss value by combining the number of samples, the number of categories, the true label and the predicted probability.

[0138] For ease of description, the above steps are combined for explanation below.

[0139] Taking the deep neural network model for text classification tasks as an example, the definition of the loss function and the calculation of the current loss value are explained. For deep neural network models for other types of tasks, you can refer to this and will not go into details here.

[0140] Definition of loss function:

[0141] For text classification tasks, cross entropy loss can be used to calculate the loss value:

[0142] , where N is the number of samples and C is the number of categories. is the true label (0 or 1, indicating whether sample i belongs to category j, is the model’s predicted probability (usually calculated via a softmax (normalization layer) function).

[0143] In order to reduce the number of model parameters to be modified in each iteration, a sparsity loss is added in this embodiment:

[0144] , where Nnonzero represents the number of non-zero model parameters and Nall represents the number of all model parameters.

[0145] The overall loss function during model training is:

[0146] ,

[0147] In this way, it is possible to achieve, during training, on the one hand, ensuring the accuracy of the model (i.e., the continuous reduction of cross entropy), and on the other hand, ensuring the sparsity of the model (i.e., the continuous reduction of sparsity loss).

[0148] S104. Determine whether the training is completed using the current loss value.

[0149] If the judgment result is yes, the operation of step S105 is executed, and if the judgment result is no, the operation of step S106 is executed.

[0150] The threshold for ending training can be determined according to actual needs, and the current loss value can be compared with the threshold to determine whether the training is completed. Of course, the training can also be determined based on the trend of the loss value.

[0151] S105. If yes, end the training; S106. If no, proceed to the next training round.

[0152] After the training is completed, a deep neural network model that matches the current edge device can be obtained. After the personalized fine-tuning of the model is realized, the deep neural network model can be directly run locally to process the corresponding deep learning tasks, so that the processing can be adapted to the local area, and there is no need to worry about privacy protection data leakage. If the training is not completed, the next training round can be entered, that is, after determining to enter the next training round, step S101 is performed again to classify the current model parameters, and the model parameters are trained and updated according to different classifications.

[0153] The method provided by the embodiment of the present application is applied in the edge device. After entering the current training round, the model parameters in the form of a two-dimensional matrix in the deep neural network model are divided into stop updating parameters and continue updating parameters; the historical gradient information of the stop updating parameters is deleted, and the continue updating parameters are updated based on the historical gradient information of the continue updating parameters; after completing the parameter update of the current training round, the current loss value of the deep neural network model is calculated; the current loss value is used to determine whether the training is completed; if yes, the training is ended; if not, the next training round is entered.

[0154] In this application, in order to avoid violating privacy protection, the deep neural network model is trained directly in the edge device. In order to realize the training of the deep neural network model in the edge device, after determining the current training round, the model parameters in the two-dimensional matrix form of the deep neural network model are first divided into stop updating parameters and continue updating parameters. The historical gradient information of the stop updating parameters is deleted, and only the parameters are updated for the continue updating parameters. Then the current loss value is calculated, that is, the loss value situation is determined to determine the next training direction. Because the model parameters in the two-dimensional matrix form are first classified before the parameter update, and the historical gradient information of the stop updating parameters is deleted, on the one hand, the video memory occupancy can be effectively reduced, and on the other hand, only the update processing is performed for the continue updating parameters, and the amount of calculation can also be greatly reduced. In this way, the training of the deep neural network model can be realized in the edge device with limited resources. Training the deep neural network in the edge device means that the privacy data information required for training can be directly obtained locally, without the need to propagate the privacy data information or collect it across devices, which may violate privacy protection. This application can achieve effective fine-tuning of the deep neural network model of the edge device without violating privacy protection.

[0155] It should be noted that, based on the above embodiments, the embodiments of the present application also provide corresponding improved solutions. In the preferred / improved embodiments, the same steps or corresponding steps as those in the above embodiments can be referenced to each other, and the corresponding beneficial effects can also be referenced to each other, which will not be described one by one in the preferred / improved embodiments of this article.

[0156] In a specific implementation of the present application, the model parameters in the form of a two-dimensional matrix in the deep neural network model are divided into stop updating parameters and continue updating parameters, including:

[0157] Add auxiliary parameters corresponding to the model parameters; wherein the auxiliary parameters are structured representations of the model parameters;

[0158] Determine the control state of model parameters using auxiliary parameters;

[0159] Using the control state, the model parameters are divided into stop updating parameters and continue updating parameters.

[0160] The control state of the model parameters is determined by using auxiliary parameters, including:

[0161] If the auxiliary parameter is less than or equal to zero, the control state of the corresponding model parameter is determined to stop updating;

[0162] If the auxiliary parameter is greater than zero, the control state of the corresponding model parameter is determined to continue updating.

[0163] For edge description, the above steps are combined and explained below.

[0164] In order to control the scale of parameters to be trained and updated in the model, the selection of effective parameters is achieved by designing auxiliary parameters and gating functions. At the same time, in order to reduce the number of auxiliary parameters and facilitate efficient execution on the actual hardware platform, the auxiliary parameters are represented as a structured representation of the model parameters.

[0165] For some W in the model parameter W ij , add auxiliary parameter W ij ', and through the gating function, the corresponding parameters are sparsely controlled.

[0166]

[0167] Where g(*) is a step function, which is used to control whether the corresponding parameter stops updating; i and j represent the i-th model parameter matrix of the model, and j represents the j-th row (or j-th column) of the parameter matrix; M ij Indicates the control status of the corresponding parameter (M ij =0, the corresponding parameter stops updating; M ij =1, the corresponding parameters continue to be updated).

[0168] In a specific implementation of the present application, the continued update parameter is updated based on the historical gradient information of the continued update parameter, including:

[0169] Using auxiliary parameters to obtain the sparse structure of gradients;

[0170] Structural simplification of parameter matrices and gradients;

[0171] When performing general matrix multiplication, the simplified parameter matrix and gradient are used to perform calculations to obtain calculation results;

[0172] The calculation results are padded with zeros to keep the output dimension unchanged.

[0173] For ease of description, the above steps are combined for explanation below.

[0174] After the gradient is calculated, for the structured sparse parameter matrix, the gradient and the parameter matrix are multiplied during back propagation. Since the gradient is sparse, there are invalid calculations in this matrix multiplication (for example, the model parameter W and the gradient G are multiplied by W*G. Since G is row or column sparse, some elements in the calculation result are fixed to 0, and the effective calculation amount of some vector multiplications is reduced).

[0175] For this reason, in this embodiment, the computational task based on matrix re-stitching is simplified. Figure 2 , through the current structured sparse auxiliary parameter W ij ', get the sparse structure of the current gradient G; then simplify the parameter matrix W and the gradient G (fix the black blocks to 0); in the GEMM calculation, directly calculate the simplified W and G, and after obtaining the calculation results, perform zero padding (such as the black blocks shown in the calculation results are the zero padding positions) to achieve the invariance of the output dimension.

[0176] In a specific implementation of the present application, the parameter matrix and the gradient are structurally simplified, including:

[0177] Get the sparse types corresponding to the parameter matrix and gradient respectively;

[0178] If the sparse type is a row sparse parameter, the columns at the sparse position are compressed;

[0179] If the sparse type is a column sparse parameter, the rows at the sparse position are compressed;

[0180] After completing row and column compression, the simplified parameter matrix and gradient are obtained.

[0181] For ease of description, the above steps are combined for explanation below.

[0182] For example, for a 3x3 model parameter matrix W, during the model pruning process, the row and column sparsity parameters obtained may be: , , then corresponding to W, the first row and the second column are reset to zero, that is, the corresponding mask is (0 means the parameter is set to zero; 1 means the parameter may not be zero).

[0183] Accelerated matrix multiplication based on sparse structure, pseudo code of structured sparse matrix multiplication:

[0184] Step 1: Get the model parameter sequence number indices based on the row and column sparsity parameters obtained by model pruning;

[0185] Step 2: According to the complete model parameters W and model parameter serial number indices, extract the non-zero parameters, re-concatenate the matrix, and obtain the compressed model parameters W_S;

[0186] Step 3: Modify the GEMM interface call according to W_S and the model parameter sequence number indices: For row-sparse parameters, compress the columns of the corresponding positions of input X; for column-sparse parameters, compress the rows of the corresponding positions of input X, and fill the final result with zeros to ensure the invariance of the calculation result dimension. In this way, useless element multiplication and addition operations can be avoided without affecting the calculation of other layers.

[0187] Corresponding to the above method embodiment, the embodiment of the present application also provides a model training device, and the model training device described below and the model training method described above can refer to each other.

[0188] See also Figure 3 As shown, the device includes the following modules:

[0189] The parameter classification module 101 is used to classify the model parameters in the form of two-dimensional matrices in the deep neural network model into parameters to be stopped from updating and parameters to be continued to update after entering the current training round;

[0190] A parameter updating module 102, configured to delete the historical gradient information of the stopped updating parameters, and update the continued updating parameters based on the historical gradient information of the continued updating parameters;

[0191] The loss calculation module 103 is used to calculate the current loss value of the deep neural network model after completing the parameter update of the current training round;

[0192] The direction judgment module 104 is used to use the current loss value to determine whether the training is completed; if yes, the training is ended; if not, the next training round is entered.

[0193] The device provided by the embodiment of the present application is applied in the edge device. After entering the current training round, the model parameters in the form of a two-dimensional matrix in the deep neural network model are divided into stop updating parameters and continue updating parameters; the historical gradient information of the stop updating parameters is deleted, and the continue updating parameters are updated based on the historical gradient information of the continue updating parameters; after completing the parameter update of the current training round, the current loss value of the deep neural network model is calculated; the current loss value is used to determine whether the training is completed; if yes, the training is ended; if not, the next training round is entered.

[0194] In this application, in order to avoid violating privacy protection, the deep neural network model is trained directly in the edge device. In order to realize the training of the deep neural network model in the edge device, after determining the current training round, the model parameters in the two-dimensional matrix form of the deep neural network model are first divided into stop updating parameters and continue updating parameters. The historical gradient information of the stop updating parameters is deleted, and only the parameters are updated for the continue updating parameters. Then the current loss value is calculated, that is, the loss value situation is determined to determine the next training direction. Because the model parameters in the two-dimensional matrix form are first classified before the parameter update, and the historical gradient information of the stop updating parameters is deleted, on the one hand, the video memory occupancy can be effectively reduced, and on the other hand, only the update processing is performed for the continue updating parameters, and the amount of calculation can also be greatly reduced. In this way, the training of the deep neural network model can be realized in the edge device with limited resources. Training the deep neural network in the edge device means that the privacy data information required for training can be directly obtained locally, without the need to propagate the privacy data information or collect it across devices, which may violate privacy protection. This application can achieve effective fine-tuning of the deep neural network model of the edge device without violating privacy protection.

[0195] In a specific implementation of the present application, the parameter updating module is specifically used to find the historical gradient information for continuing to update the parameters;

[0196] If not found, the historical gradient information of the parameters to be updated is determined by using the parameters and gradients to be updated continuously;

[0197] The parameters are updated based on the historical gradient information.

[0198] In a specific implementation of the present application, the parameter updating module is specifically used to utilize the parameter updating rule and combine the continued updating of the parameter and the gradient to determine the historical gradient information of the continued updating of the parameter.

[0199] In a specific implementation of the present application, the loss calculation module is specifically used to calculate the cross entropy loss value using the prediction probability and training samples obtained through training;

[0200] Based on the parameter state of the neural network model, the sparsity loss value is calculated;

[0201] Superimpose the cross entropy loss value and the sparsity loss value to get the current loss value.

[0202] In a specific implementation of the present application, the loss calculation module is specifically used to obtain the total number of model parameters;

[0203] Obtain parameter values ​​of model parameters, and determine a target number of non-zero model parameters based on the parameter values;

[0204] Calculate the ratio of the target number to the total number;

[0205] The ratio is determined as the sparsity loss value.

[0206] In a specific implementation of the present application, the loss calculation module is specifically used to obtain the sample quantity, category quantity and true label of the training sample;

[0207] Use the normalization layer in the deep neural network model to obtain the prediction probability;

[0208] The cross entropy loss function is used to calculate the cross entropy loss value by combining the number of samples, the number of categories, the true label and the predicted probability.

[0209] In a specific implementation of the present application, the parameter updating module is specifically used to set the first-order moment estimation and the second-order moment estimation of the stopped parameter updating to zero.

[0210] In a specific implementation of the present application, the parameter classification module is specifically used to add auxiliary parameters corresponding to the model parameters; wherein the auxiliary parameters are structured representations of the model parameters;

[0211] Determine the control state of model parameters using auxiliary parameters;

[0212] Using the control state, the model parameters are divided into stop updating parameters and continue updating parameters.

[0213] In a specific implementation of the present application, the parameter classification module is specifically used to determine that the control state of the corresponding model parameter is to stop updating if the auxiliary parameter is less than or equal to zero;

[0214] If the auxiliary parameter is greater than zero, the control state of the corresponding model parameter is determined to continue updating.

[0215] In a specific implementation of the present application, the parameter updating module is specifically used to obtain the sparse structure of the gradient using the auxiliary parameters;

[0216] Structural simplification of parameter matrices and gradients;

[0217] When performing general matrix multiplication, the simplified parameter matrix and gradient are used to perform calculations to obtain calculation results;

[0218] The calculation results are padded with zeros to keep the output dimension unchanged.

[0219] In a specific implementation of the present application, the parameter updating module is specifically used to obtain the sparse types corresponding to the parameter matrix and the gradient respectively;

[0220] If the sparse type is a row sparse parameter, the columns at the sparse position are compressed;

[0221] If the sparse type is a column sparse parameter, the rows at the sparse position are compressed;

[0222] After completing row and column compression, the simplified parameter matrix and gradient are obtained.

[0223] Corresponding to the above method embodiment, the embodiment of the present application also provides a task processing method. The task processing method described below and the model training method described above can refer to each other.

[0224] Please refer to Figure 6 ,The model training method, applied to edge devices, includes:

[0225] S201. Obtain a deep neural network model from a server.

[0226] Specifically, the edge device can send a model acquisition request to the large server, and then receive the deep neural network model fed back by the large server. For example, when the edge device needs to process a text-type deep task, it can obtain the deep neural network model for processing text that has been trained on the server from the server; when the edge device needs to process an image-type deep task, it can obtain the deep neural network model for processing images that has been trained on the server from the server. In other words, the edge device needs to process which type of deep task, and it will obtain the deep neural network model that has been trained on the server from the server to process that type of deep task.

[0227] S202: Load the deep neural network model to the local training platform.

[0228] The local training platform may be a Jetson platform. After the model is loaded, proceed to step S203.

[0229] S203. Load the privacy data required for training into the local training platform, so that the local training platform can train and fine-tune the deep neural network model based on the privacy data according to the above-mentioned model training method to obtain the target model.

[0230] Among them, fine-tuning the model can be specifically referred to Figure 1 The steps shown are processed and will not be described in detail here.

[0231] The privacy data refers to the user's sensitive data. For details on how to obtain the privacy data locally and how to train and fine-tune the model based on the privacy data, please refer to the model training implementation, which will not be described in detail here.

[0232] S204: Obtain the target task, and process the target task using the target model to obtain a processing result.

[0233] After achieving local privacy data protection and effectively fine-tuning the model, the target task is obtained and processed based on the target model to obtain accurate processing results.

[0234] It can be seen that this method can not only achieve fine-tuning of the model based on the limited resources of edge devices, but also protect privacy data, thereby obtaining more accurate processing results.

[0235] Corresponding to the above method embodiment, the embodiment of the present application further provides a task processing device. The task processing device described below and the task processing method described above can refer to each other.

[0236] Please refer to Figure 7 The task processing device is applied to an edge device and includes:

[0237] A model acquisition module 201 is used to acquire a deep neural network model from a server;

[0238] A model loading module 202 is used to load the deep neural network model into a local training platform;

[0239] Model training module 203, used to load the private data required for training into the local training platform, so that the local training platform can train and fine-tune the deep neural network model based on the private data according to the model training method as described above to obtain a target model;

[0240] The task processing module 204 is used to obtain the target task, process the target task using the target model, and obtain a processing result.

[0241] This device can not only fine-tune the model based on the limited resources of edge devices, but also protect privacy data, thereby obtaining more accurate processing results.

[0242] Corresponding to the above method embodiment, the embodiment of the present application also provides an edge device. The edge device described below and the model training method described above can refer to each other.

[0243] See also Figure 4 As shown, the edge device includes:

[0244] A memory 332, for storing computer programs;

[0245] The processor 322 is used to implement the steps of the model training method of the above method embodiment when executing a computer program, or to implement the steps of the model training method as described above when executing a computer program.

[0246] For details, please refer to Figure 5 , Figure 5 A specific structural diagram of an edge device provided for this embodiment, the edge device may have relatively large differences due to different configurations or performances, and may include one or more processors (central processing units, CPU) (for example, one or more processors) and a memory 332, and the memory 332 stores one or more computer programs 342 or data 344. Among them, the memory 332 can be a temporary storage or a permanent storage. The program stored in the memory 332 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the data processing device. Furthermore, the processor 322 can be configured to communicate with the memory 332 to execute a series of instruction operations in the memory 332 on the edge device 301.

[0247] The edge device 301 may also include one or more power supplies 326 , one or more wired or wireless network interfaces 350 , one or more input and output interfaces 358 , and / or one or more operating systems 341 .

[0248] The steps in the model training method described above can be implemented by the structure of the edge device.

[0249] Specifically, the following steps can be implemented in the edge device:

[0250] After entering the current training round, the model parameters in the form of two-dimensional matrices in the deep neural network model are divided into stop updating parameters and continue updating parameters;

[0251] Delete the historical gradient information of the stopped updating parameters, and update the continued updating parameters based on the historical gradient information of the continued updating parameters;

[0252] After completing the parameter update of the current training round, calculate the current loss value of the deep neural network model;

[0253] Use the current loss value to determine whether training is complete;

[0254] If yes, the training ends; if no, the next training round begins.

[0255] Wherein, based on the historical gradient information of the continued update parameter, the continued update parameter is updated, including:

[0256] Find the historical gradient information to continue updating the parameters;

[0257] If not found, the historical gradient information of the parameters to be updated is determined by using the parameters and gradients to be updated continuously;

[0258] The parameters are updated based on the historical gradient information.

[0259] The historical gradient information of the parameters to be updated is determined by using the parameters and gradients to be updated continuously, including:

[0260] The parameter update rule is used in combination with the continued update of the parameters and the gradient to determine the historical gradient information for the continued update of the parameters.

[0261] Among them, the current loss value of the deep neural network model is calculated, including:

[0262] Using the predicted probability obtained from training and the training samples, calculate the cross entropy loss value;

[0263] Based on the parameter state of the neural network model, the sparsity loss value is calculated;

[0264] Superimpose the cross entropy loss value and the sparsity loss value to get the current loss value.

[0265] Among them, based on the parameter state of the neural network model, the sparsity loss value is calculated, including:

[0266] Get the total number of model parameters;

[0267] Obtain parameter values ​​of model parameters, and determine a target number of non-zero model parameters based on the parameter values;

[0268] Calculate the ratio of the target number to the total number;

[0269] The ratio is determined as the sparsity loss value.

[0270] Among them, the cross entropy loss value is calculated using the predicted probability obtained from training and the training samples, including:

[0271] Get the number of samples, number of categories and true labels of training samples;

[0272] Use the normalization layer in the deep neural network model to obtain the prediction probability;

[0273] The cross entropy loss function is used to calculate the cross entropy loss value by combining the number of samples, the number of categories, the true label and the predicted probability.

[0274] Among them, the historical gradient information of stopping updating parameters is deleted, including:

[0275] Set the first-order moment estimate and the second-order moment estimate to zero to stop updating the parameters.

[0276] Among them, the model parameters in the form of two-dimensional matrix in the deep neural network model are divided into stop updating parameters and continue updating parameters, including:

[0277] Add auxiliary parameters corresponding to the model parameters; wherein the auxiliary parameters are structured representations of the model parameters;

[0278] Determine the control state of model parameters using auxiliary parameters;

[0279] Using the control state, the model parameters are divided into stop updating parameters and continue updating parameters.

[0280] The control state of the model parameters is determined by using auxiliary parameters, including:

[0281] If the auxiliary parameter is less than or equal to zero, the control state of the corresponding model parameter is determined to stop updating;

[0282] If the auxiliary parameter is greater than zero, the control state of the corresponding model parameter is determined to continue updating.

[0283] Wherein, based on the historical gradient information of the continued update parameter, the continued update parameter is updated, including:

[0284] Using auxiliary parameters to obtain the sparse structure of gradients;

[0285] Structural simplification of parameter matrices and gradients;

[0286] When performing general matrix multiplication, the simplified parameter matrix and gradient are used to perform calculations to obtain calculation results;

[0287] The calculation results are padded with zeros to keep the output dimension unchanged.

[0288] Among them, the parameter matrix and gradient are structurally simplified, including:

[0289] Get the sparse types corresponding to the parameter matrix and gradient respectively;

[0290] If the sparse type is a row sparse parameter, the columns at the sparse position are compressed;

[0291] If the sparse type is a column sparse parameter, the rows at the sparse position are compressed;

[0292] After completing row and column compression, the simplified parameter matrix and gradient are obtained.

[0293] The steps in the model training method described above can be implemented by the structure of the edge device.

[0294] Specifically, the following steps can be implemented in the edge device:

[0295] Get the deep neural network model from the server;

[0296] Load the deep neural network model to the local training platform;

[0297] Load the private data required for training into the local training platform, so that the local training platform can train and fine-tune the deep neural network model based on the private data according to the above-mentioned model training method to obtain the target model;

[0298] Obtain the target task, process the target task using the target model, and obtain the processing result.

[0299] Corresponding to the above method embodiment, the embodiment of the present application also provides a readable storage medium. The readable storage medium described below and the model training method and task processing method described above can be referenced to each other.

[0300] A readable storage medium having a computer program stored thereon, which implements the steps of the model training method of the above-mentioned method embodiment when the computer program is executed by a processor, or implements the steps of the above-mentioned task processing method when the computer program is executed by a processor.

[0301] The readable storage medium may specifically be a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or other readable storage medium that can store program codes.

[0302] Corresponding to the above method embodiment, the embodiment of the present application also provides a computer program product. The computer program product described below and the model training method and task processing method described above can be referenced to each other.

[0303] A computer program product includes a computer program / instruction. When the computer program / instruction is executed by a processor, the steps of the above-mentioned model training method or the steps of the above-mentioned task processing method are implemented.

[0304] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0305] Those skilled in the art may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0306] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0307] Finally, it should be noted that, in this article, relationships such as first and second, etc. are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms include, include or any other variations are intended to cover non-exclusive inclusion, so that a process, method, article or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0308] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core idea of ​​the present application. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A model training method, characterized in that: Applied to edge devices, including: After entering the current training round, the model parameters in the form of two-dimensional matrices in the deep neural network model are divided into stop updating parameters and continue updating parameters; Deleting the historical gradient information of the stop updating parameter, and updating the continue updating parameter based on the historical gradient information of the continue updating parameter; After completing the parameter update of the current training round, calculating the current loss value of the deep neural network model; Use the current loss value to determine whether training is complete; If yes, the training ends; if no, the next training round begins.

2. The method according to claim 1, characterized in that Based on the historical gradient information of the continuously updated parameter, the continuously updated parameter is updated, including: Finding historical gradient information of the continuously updated parameters; If not found, using the continued updating parameter and gradient to determine the historical gradient information of the continued updating parameter; The continuous updating parameter is updated based on the historical gradient information.

3. The method according to claim 2, characterized in that Determining historical gradient information of the continuously updated parameter by using the continuously updated parameter and the gradient includes: The historical gradient information of the continuously updated parameter is determined by utilizing the parameter update rule and combining the continuously updated parameter and the gradient.

4. The method according to claim 1, characterized in that: Calculating the current loss value of the deep neural network model, including: Using the predicted probability obtained from training and the training samples, calculate the cross entropy loss value; Calculating a sparsity loss value based on a parameter state of the neural network model; The cross entropy loss value and the sparsity loss value are superimposed to obtain the current loss value.

5. The method according to claim 4, characterized in that The calculating of the sparsity loss value based on the parameter state of the neural network model includes: Obtaining the total number of model parameters; Obtaining parameter values ​​of the model parameters, and determining a target number of non-zero model parameters based on the parameter values; Calculating the ratio of the target number to the total number; The ratio is determined as the sparsity loss value.

6. The method according to claim 4, characterized in that Using the predicted probabilities obtained from training and the training samples, the cross entropy loss value is calculated, including: Obtaining the sample quantity, category quantity and true label of the training sample; Obtaining the predicted probability using a normalization layer in the deep neural network model; The cross entropy loss value is calculated by using a cross entropy loss function and combining the number of samples, the number of categories, the true label and the predicted probability.

7. The method according to claim 1, characterized in that Deleting the historical gradient information of the stopped updating parameter includes: The first-order moment estimate and the second-order moment estimate of the stop updating parameter are set to zero.

8. The method according to any one of claims 1 to 7, characterized in that: The model parameters in the form of two-dimensional matrices in the deep neural network model are divided into stop updating parameters and continue updating parameters, including: Adding auxiliary parameters corresponding to the model parameters; wherein the auxiliary parameters are structured representations of the model parameters; determining a control state of the model parameter using the auxiliary parameter; By using the control state, the model parameters are divided into stop updating parameters and continue updating parameters.

9. The method according to claim 8, characterized in that Determining the control state of the model parameter by using the auxiliary parameter includes: If the auxiliary parameter is less than or equal to zero, determining that the control state of the corresponding model parameter is to stop updating; If the auxiliary parameter is greater than zero, it is determined that the control state of the corresponding model parameter is to continue updating.

10. The method according to claim 8, characterized in that Based on the historical gradient information of the continuously updated parameter, the continuously updated parameter is updated, including: Using the auxiliary parameters to obtain a sparse structure of the gradient; Structural simplification of parameter matrices and gradients; When performing general matrix multiplication, the simplified parameter matrix and gradient are used to perform calculations to obtain calculation results; The calculation result is padded with zeros to keep the output dimension unchanged.

11. The method according to claim 10, characterized in that Simplify the structure of parameter matrices and gradients, including: Obtaining the sparse types corresponding to the parameter matrix and the gradient respectively; If the sparse type is a row sparse parameter, the columns at the sparse positions are compressed; If the sparse type is a column sparse parameter, the rows at the sparse position are compressed; After completing row and column compression, the simplified parameter matrix and gradient are obtained.

12. A task processing method, characterized in that: Applied to edge devices, including: Get the deep neural network model from the server; Loading the deep neural network model to a local training platform; Loading the privacy data required for training to the local training platform, so that the local training platform can train and fine-tune the deep neural network model based on the privacy data according to the model training method according to any one of claims 1 to 11 to obtain a target model; Obtain a target task, and process the target task using the target model to obtain a processing result.

13. An edge device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the model training method as described in any one of claims 1 to 11 when executing the computer program, or to implement the steps of the task processing method as described in claim 12 when executing the computer program.

14. A readable storage medium, characterized in that: The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the model training method as described in any one of claims 1 to 11, or, when executed by a processor, implements the steps of the task processing method as described in claim 12.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the model training method described in any one of claims 1 to 11 are implemented, or the steps of the above-mentioned task processing method described in claim 12 are implemented.

Images