Model pruning method, device, equipment and storage medium

Abstract

The present application provides a model pruning method, apparatus, device and storage medium, the method comprising: obtaining random initialization parameters and multiple training tasks of a model; performing multiple rounds of training on the model using training samples corresponding to each of the multiple training tasks, and obtaining first model parameters and second model parameters corresponding to the model after two different rounds of training. Determine a first shielding matrix corresponding to the first parameter matrix of the target layer in the first model parameters, and a second shielding matrix corresponding to the second parameter matrix of the target layer in the second model parameters; if the similarity between the first shielding matrix and the second shielding matrix is ​​greater than a set threshold, prune the randomly initialized parameters according to the shielding matrices corresponding to the parameter matrices of each layer in the second model parameters to obtain third model parameters. This solution reduces the computational overhead of model training.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a model pruning method, device, equipment and storage medium. Background Art

[0002] User terminal devices, such as smartphones, tablets, and embedded devices, are becoming an indispensable part of people's daily lives. This causes user data to accumulate on user terminal devices, thus creating a need to train deep learning models using user data collected locally on terminal devices.

[0003] For reasons of data security and privacy protection, and due to the limited amount of user data and computing power of terminal devices, how to use this user data to train deep learning models on user terminal devices has become a difficult problem that needs to be solved urgently. Summary of the Invention

[0004] Embodiments of the present invention provide a model pruning method, apparatus, device, and storage medium for reducing the computational overhead required for terminal-side model training.

[0005] In a first aspect, an embodiment of the present invention provides a model pruning method applied to a user terminal device, the method comprising:

[0006] Obtaining random initialization parameters of a model and a plurality of training tasks for training the model;

[0007] Performing multiple rounds of training on the model using training samples corresponding to each of the multiple training tasks to obtain first model parameters and second model parameters corresponding to the model after two different rounds of training, respectively, wherein the first model parameters and the second model parameters each include a parameter matrix of each layer in the model;

[0008] determining a first shielding matrix corresponding to a first parameter matrix of a target layer in the first model parameters, and a second shielding matrix corresponding to a second parameter matrix of the target layer in the second model parameters;

[0009] If the similarity between the first shielding matrix and the second shielding matrix is ​​greater than a set threshold, the randomly initialized parameters are pruned according to the shielding matrices corresponding to the parameter matrices of each layer in the second model parameters to obtain third model parameters.

[0010] In a second aspect, an embodiment of the present invention provides a model pruning device, applied to a user terminal device, comprising:

[0011] An acquisition module, configured to acquire random initialization parameters of a model and a plurality of training tasks for training the model;

[0012] a pre-training module, configured to perform multiple rounds of training on the model using training samples corresponding to each of the multiple training tasks, so as to obtain first model parameters and second model parameters corresponding to the model after two different rounds of training, respectively, wherein the first model parameters and the second model parameters each include a parameter matrix of each layer in the model;

[0013] A pruning module is used to determine a first shielding matrix corresponding to the first parameter matrix of the target layer in the first model parameters, and a second shielding matrix corresponding to the second parameter matrix of the target layer in the second model parameters; if the similarity between the first shielding matrix and the second shielding matrix is ​​greater than a set threshold, the randomly initialized parameters are pruned according to the shielding matrices corresponding to the parameter matrices of each layer in the second model parameters to obtain third model parameters.

[0014] In a third aspect, an embodiment of the present invention provides a user terminal device, comprising: a memory, a processor, and a communication interface; wherein, the memory stores executable code, and when the executable code is executed by the processor, the processor executes the model pruning method described in the first aspect.

[0015] In a fourth aspect, an embodiment of the present invention provides a non-transitory machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of a user terminal device, the processor can at least implement the model pruning method described in the first aspect.

[0016] In an embodiment of the present invention, during the process of pre-training a model using a meta-learning approach (i.e., obtaining model initialization parameters with better performance), the model parameters are first randomly initialized to obtain the model's random initialization parameters. Subsequently, the model is trained for multiple rounds using training samples corresponding to multiple training tasks to obtain first and second model parameters corresponding to the model after two different rounds of training, respectively. The first and second model parameters each include the parameter matrices of each layer in the model. A first masking matrix corresponding to the first parameter matrix of the target layer in the first model parameters is determined. Similarly, a second masking matrix corresponding to the second parameter matrix of the target layer in the second model parameters is determined, and the similarity between the first and second masking matrices is calculated. If the similarity between the first and second masking matrices is greater than a set threshold, it indicates that the parameter matrix of the target layer of the model has stabilized. At this point, training can be paused to prune the randomly initialized parameters based on the masking matrices corresponding to the parameter matrices of each layer in the second model parameters, thereby obtaining the model's third model parameters. At this point, the model with the third model parameters is a sparse model. Afterwards, the model is continuously trained based on the third model parameters to obtain target initialization parameters of the model when the training reaches convergence.

[0017] During the above training process, it is only necessary to monitor the changes in the pruning position of a specific layer in the model (reflected by the shielding matrix). When the pruning position of this layer stabilizes, the randomly initialized parameters can be pruned based on this pruning position to obtain a sparse parameter as the starting point for continued training. Sparse parameters require less storage and computing resources. Using this as a starting point for model parameter training can complete model training faster and at a lower computational cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following is a brief introduction to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0019] Figure 1 A flowchart of a model pruning method provided by an embodiment of the present invention;

[0020] Figure 2 A schematic diagram of the execution process of a model pruning method provided by an embodiment of the present invention;

[0021] Figure 3 A schematic diagram of an application of a model pruning method provided by an embodiment of the present invention;

[0022] Figure 4A schematic diagram of another application of a model pruning method provided by an embodiment of the present invention;

[0023] Figure 5 A schematic diagram of an application of a model pruning method provided by an embodiment of the present invention;

[0024] Figure 6 A schematic structural diagram of a model pruning device provided in an embodiment of the present invention;

[0025] Figure 7 A schematic diagram of the structure of a user terminal device provided in this embodiment;

[0026] Figure 8 This is a schematic diagram of the structure of another user terminal device provided in this embodiment. DETAILED DESCRIPTION

[0027] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0028] In addition, the step sequence in the following method embodiments is only an example and not a strict limitation.

[0029] The problem of few-shot learning arises in many real-world scenarios. For example, the user data collected on user terminals is relatively limited, yet there is a need to train a neural network model based on this small amount of user data to provide a specific service. Another example is the limited medical records for rare diseases, yet there is a need to train a neural network model based on this small amount of information to provide disease prediction capabilities.

[0030] Meta-learning, also known as "learning to learn," leverages past knowledge and experience to guide the learning of new tasks, enabling the network to learn. It is a common approach for solving small-sample problems. Common meta-learning methods include the MAML algorithm and the Reptile algorithm.

[0031] First, let's briefly introduce meta-learning. In contrast, traditional machine learning involves manually adjusting model parameters before directly training a neural network model for a specific task. Meta-learning, on the other hand, first trains a model using other tasks to obtain optimal initial model parameters, then trains on a specific task to obtain the model parameters for that task. Therefore, meta-learning involves training a model on a large number of tasks, obtaining an initialized model (i.e., obtaining optimal model initialization parameters), and then using a small amount of data to more quickly learn on new tasks. This initialized model has the ability to converge quickly on unknown tasks.

[0032] In machine learning, the unit of training is the sample data corresponding to a task. This sample data is used to train the model, and can be divided into a training set, a test set, and a validation set. In meta-learning, the unit of training is the task, which is generally divided into two types: training tasks and test tasks. Many training tasks are prepared for learning, with the goal of learning optimal hyperparameters (better initialization parameters). Test tasks use the hyperparameters learned from the training tasks to train the specific task. The sample data for each task in a training task is divided into a support set and a query set; the data for a test task is divided into a training set and a test set.

[0033] For example: Task 1 is a speech recognition task, Task 2 is an image recognition task, etc., Task 100 is a text classification task, Task 101 is different from the previous 100 tasks. The training tasks are these 100 different tasks, and the test task is the 101st task.

[0034] In meta-learning, the dependence on the amount of sample data corresponding to each task is small, meaning the number of sample data corresponding to each task can be small. In summary, the meta-learning training process first requires randomly initializing the model parameters. Then, using these randomly initialized parameters as a starting point, the model is iteratively trained on multiple training tasks until convergence. This results in good initialization parameters for the model. Subsequently, using these good initialization parameters as a starting point, the model is trained on new tasks to obtain model parameters suitable for the new task.

[0035] In the traditional meta-learning process described above, training the model from randomly initialized model parameters to convergence actually consumes a large amount of computing resources and takes a long time. To this end, a pruning strategy has been considered to reduce the computational overhead of model training and improve model training efficiency.

[0036] A basic pruning solution includes the following steps: 1) first randomly initialize the model parameters; 2) then perform meta-learning training until the model converges; 3) determine the masking matrix based on the model parameters at the time of model convergence to prune the entire model; 4) retrain to restore the model performance.

[0037] Among them, the mask matrix is ​​used to record the pruned parameter positions in the model parameter matrix. It is a matrix composed of 0 and 1. 0 represents that the parameter value at the corresponding position is pruned, and 1 represents that the parameter value at the corresponding position is retained. In fact, a model will have many parameter matrices. For example, each layer in the model will correspond to a parameter matrix. The above-mentioned model pruning process is actually based on the parameter matrix corresponding to each layer in the model to determine the mask matrix corresponding to each layer, and then prune the parameter matrix of the corresponding layer based on each mask matrix to subtract the parameter value in the parameter matrix corresponding to the 0 value position in the mask matrix (set to 0).

[0038] While model pruning (step 4) can reduce computational overhead and enable faster model convergence by continuing model training based on pruned model parameters, this approach suffers from the high computational cost of meta-learning and very slow convergence during step 2) of training the model from randomly initialized parameters to convergence, making the overall pruning process computationally intensive. Traditional model pruning techniques in step 3) are designed for general machine learning tasks and are not necessarily suitable for meta-learning scenarios.

[0039] Based on this, traditional pruning solutions are not applicable in some situations that are more sensitive to computational overhead, such as when performing model training on user terminal devices with limited computing power and storage capacity. In particular, in scenarios where model training is performed using user data collected on user terminal devices, for the sake of user data security and privacy, the need to reduce computational overhead is more urgent to complete model training locally on user terminal devices based on meta-learning.

[0040] Based on this, the embodiment of the present invention provides an efficient model pruning scheme suitable for the meta-learning process based on the Lottery Ticket Hypothesis (LTH) theory, which can effectively reduce computational overhead and improve model training efficiency. The scheme provided by the example of the present invention performs model pruning on the meta-learning model to obtain a sparse model with fast convergence capability, thereby enabling deep learning model training on the end-side user terminal device. It should be noted that the model pruning scheme provided by the embodiment of the present invention can be applied to the process of model training using the meta-learning training method, and can also be applied to the model training process of other training methods.

[0041] The lottery hypothesis states that: given a randomly initialized deep learning model, which contains at least one sub-model, it can be trained to achieve the same effect (accuracy / error rate, etc.) as the original model on a given target learning task. This sub-model is the winning lottery ticket, that is, the winning sub-model. Among them, this winning model is actually the model after some parameter values ​​in the model parameter matrix of the original model are set to 0 (that is, the model after deleting some parameter values). In fact, the model parameter matrix is ​​also commonly referred to as the model weight matrix. Of course, in fact, a model will have many parameter matrices. For example, each layer in the model will correspond to a parameter matrix. In the absence of ambiguity, the purpose of model training is actually to obtain model parameters. Therefore, the randomly initialized model mentioned in the embodiment of the present invention is also equivalent to the randomly initialized parameters of the model.

[0042] The inventors have verified that the lottery ticket hypothesis still holds true in meta-learning. Therefore, during model pruning, finding the winning sub-model can be the goal of model pruning.

[0043] The following specifically describes the model pruning method proposed in an embodiment of the present invention in combination with the lottery hypothesis theory.

[0044] Figure 1 A flow chart of a model pruning method provided by an embodiment of the present invention is as follows: Figure 1 As shown, the method includes the following steps:

[0045] 101. Obtain random initialization parameters of the model and multiple training tasks for training the model.

[0046] 102. Use training samples corresponding to multiple training tasks to perform multiple rounds of training on the model to obtain first model parameters and second model parameters corresponding to the models after two different rounds of training, respectively. The first model parameters and the second model parameters both include parameter matrices of each layer in the model.

[0047] 103. Determine a first shielding matrix corresponding to the first parameter matrix of the target layer in the first model parameters, and a second shielding matrix corresponding to the second parameter matrix of the target layer in the second model parameters; if the similarity between the first shielding matrix and the second shielding matrix is ​​greater than a set threshold, prune the randomly initialized parameters according to the shielding matrices corresponding to the parameter matrices of each layer in the second model parameters to obtain third model parameters.

[0048] The trained model in the embodiment of the present invention may be a neural network model with a certain structure, such as a convolutional neural network model, a recurrent neural network model, a long short-term memory neural network model, and the like.

[0049] In an optional embodiment, the trained model can be trained using a meta-learning approach. When the model is trained using a meta-learning approach, the meta-learning training process can be divided into two stages: a pre-training stage and a training stage for the target task. In the pre-training stage, the model needs to be trained using training samples corresponding to multiple training tasks, with the goal of obtaining a better initialization parameter. In other words, the pre-training stage needs to complete the optimization of the model from random initialization parameters to better initialization parameters. In this way, when the model is trained for the target task, it can start with a better model initialization parameter, thereby achieving rapid convergence of the model.

[0050] Therefore, when using meta-learning to train the model, after step 103, the following steps can also be performed:

[0051] 104. Train the model based on the third model parameters to obtain target initialization parameters of the model when the training reaches convergence.

[0052] Generally speaking, the above-mentioned multiple training tasks and the target task are different tasks, but these tasks have certain similarities and conform to a certain set probability distribution.

[0053] Figure 1 The solution provided by the illustrated embodiment corresponds to the pre-training stage of meta-learning, that is, the above-mentioned target initialization parameters are the better initialization parameters finally obtained in the pre-training stage.

[0054] However, it should be noted that, as described above, the model pruning scheme provided in the embodiments of the present invention is applicable not only to the meta-learning training process, but also to other model training processes. In this case, the training samples used to train the model are not limited to corresponding to multiple training tasks, but can also correspond to only one task (such as an image classification task). Since the model requires multiple rounds of iterative training and the model parameters are randomly initialized, the first model parameters and the second model parameters can be the model parameters after any two adjacent or non-adjacent iteration rounds. The third model parameters correspond to the overall model parameters when the parameters corresponding to the target layer in the model tend to a stable state. Based on this, the execution of the model pruning scheme is to prune the randomly initialized model parameters based on the above-mentioned shielding matrix when the parameters corresponding to the target layer tend to a stable state, thereby obtaining a sparse model parameter—the third model parameter. After obtaining the third model parameter, if the model has not converged at this time, the iterative training of the model can continue based on the third model parameter.

[0055] As can be seen, regardless of whether or not meta-learning training is used, the execution of the model pruning solution is similar, and steps similar to steps 101-103 above need to be performed. Therefore, for ease of understanding and description, the following uses the meta-learning training method as an example for explanation.

[0056] In practical applications, multiple training tasks and their corresponding training samples are collected according to the needs of new target tasks.

[0057] For example, if the target task is a classification task, then the multiple training samples are multiple classification tasks different from the target task. For example, the multiple training tasks are: the classification task of classifying cats and dogs, the classification task of classifying dogs and wolves, and the classification task of classifying cats and tigers; the target task can be the task of classifying birds.

[0058] During the pre-training phase, the model parameters are first randomly initialized to obtain the model's randomly initialized parameters. In reality, model parameters are tensors composed of several parameter matrices. The model here refers to a deep neural network model. Structurally, it typically includes an input layer, several hidden layers, and an output layer. Each layer has a corresponding parameter matrix. Therefore, the model parameters are actually composed of the parameter matrices corresponding to each layer. Therefore, the model training process is essentially the process of determining the model parameters.

[0059] After obtaining the model's initialization parameters and multiple training tasks, the model is trained in multiple rounds of meta-learning based on the training samples corresponding to the multiple training tasks. After each round of training, the model parameters are updated.

[0060] The meta-learning training process can be implemented by referring to existing related technologies. The embodiments of the present invention will not be elaborated in detail. Only the following brief description is provided: training samples of multiple training tasks can be sampled. For each training task, starting from the same random initialization parameters, the test loss function generated after the training sample corresponding to each training task is calculated. The test loss functions corresponding to each of the multiple training tasks are weighted summed to obtain the total loss function corresponding to the entire training task. The model parameters are updated once based on the total loss function. This is repeated for multiple rounds.

[0061] A round interval k for reading model parameters can be pre-set. For example, k=1 means that the model parameters need to be obtained after each round of training; k=3 means that the model parameters need to be read every three rounds.

[0062] Based on the set value of k, the first and second model parameters corresponding to the models after two different rounds of training are obtained. For example, if k = 1, starting with randomly initialized parameters, the model parameters are updated after each round of training. The model parameters after the first update can be used as the first model parameters, and the model parameters after the second update can be used as the second model parameters. At this time, the first shielding matrix corresponding to the first parameter matrix of the target layer in the first model parameters and the second shielding matrix corresponding to the second parameter matrix of the target layer in the second model parameters are determined. The similarity between the two shielding matrices is calculated. If the similarity is less than the set threshold, the third round of training continues to obtain the model parameters after the third update. At this time, the second updated model parameters are used as the first model parameters and the third updated model parameters as the second model parameters. The corresponding shielding matrices and the similarity between the shielding matrices are calculated as described above. If the calculated similarity is greater than the set threshold, training is suspended.

[0063] In summary, the model parameters read at any two adjacent times are respectively taken as the first model parameters and the second model parameters. As mentioned above, these two model parameters include the parameter matrices corresponding to each layer in the model. However, first, it is only necessary to extract the parameter matrix corresponding to the set target layer from these two model parameters, which are respectively called the first parameter matrix and the second parameter matrix, determine the first shielding matrix corresponding to the first parameter matrix, and determine whether to proceed to the next round of training based on the similarity between the two shielding matrices.

[0064] Among them, determining the first shielding matrix corresponding to the first parameter matrix of the target layer in the first model parameter can be implemented as follows: sorting the parameter values ​​contained in the first parameter matrix of the target layer from large to small; determining the parameter values ​​that are ranked at the back of a set proportion, or determining the parameter values ​​that are less than the set value; and generating the first shielding matrix according to the corresponding positions of the determined parameter values ​​in the first parameter matrix.

[0065] The second shielding matrix is ​​generated in the same manner and will not be described in detail. The first parameter matrix and the second parameter matrix have the same dimension, so the first shielding matrix and the second shielding matrix also have the same dimension, which is equal to the dimension of the parameter matrix.

[0066] For example, assuming the first parameter matrix is ​​a 10*10 dimensional matrix, the 100 parameter values ​​are sorted from largest to smallest. For example, if the above-mentioned set ratio is 10%, the last 10 parameter values ​​are determined, and the target positions corresponding to these 10 parameter values ​​in the first parameter matrix are recorded. A first masking matrix of the same dimension as the first parameter matrix is ​​generated. In the first masking matrix, the element values ​​corresponding to these target positions are set to 0, and the element values ​​of other positions are set to 1.

[0067] After obtaining the first and second shielding matrices, the similarity between the two is calculated. For example, a preset distance type between the first and second shielding matrices is optionally determined as the similarity, where the preset distance type is, for example, Euclidean distance or Hamming distance. Alternatively, the differences in element values ​​at the same position in the two shielding matrices can be counted, the number of instances in which the element values ​​corresponding to the same position are identical in the two shielding matrices is determined, and the percentage of this number relative to the dimension of the shielding matrices is determined as the similarity between the two. Alternatively, the sum of the differences in the element values ​​at the same position in the two shielding matrices can be calculated as the similarity between the two.

[0068] If the similarity between the first shielding matrix and the second shielding matrix is ​​high, for example, greater than the set threshold, it is considered that the first parameter matrix of the target layer in the first model parameters and the second parameter matrix of the target layer in the second model parameters are highly similar, indicating that the training of the target layer is now stable and the training can be paused to use the shielding matrix corresponding to the parameter matrix of each layer in the second model parameters to prune the randomly initialized parameters.

[0069] Among them, the method for determining the shielding matrix corresponding to the parameter matrix of each layer in the second model parameters is the same as the method for determining the second shielding matrix corresponding to the second parameter matrix of the target layer, and will not be repeated here.

[0070] The above-mentioned pruning process can be: use the shielding matrix corresponding to the parameter matrix of a certain layer in the second model parameters to multiply the parameter matrix of the same layer in the random initialization parameters, so that multiple parameter values ​​in the parameter matrix of this layer in the random initialization parameters are set to 0, while other parameter values ​​remain unchanged, wherein the positions of these multiple parameter values ​​correspond to the positions where the element values ​​in the corresponding shielding matrix are 0.

[0071] The result of pruning the randomly initialized parameters is called the third model parameters. The parameter matrices of each layer in the third model parameters will contain many zero-valued elements, making the third model parameters sparse. Model training continues using the third model parameters as a starting point until the model converges. When the model converges, the target initialization parameters are obtained, completing the model pre-training phase.

[0072] It is understandable that since the third model parameters are sparse, when subsequent meta-learning training is performed based on the third model parameters, the 0-valued elements in each parameter matrix in the third model parameters do not need to participate in the relevant calculations in the subsequent training process, thereby reducing the computational overhead. Moreover, these 0-valued elements do not need to be stored. Only the values ​​of the non-zero-valued elements and their positions in the parameter matrix need to be stored, which reduces the storage capacity. Based on the sparse third model parameters, the model can be trained to convergence faster. Moreover, the target initialization parameters obtained when training to convergence are also sparse, that is, a sparse model can be obtained, which can enable the model to have faster convergence capabilities on subsequent target tasks.

[0073] In fact, the theoretical basis of the above-mentioned idea of ​​pruning the model is the lottery hypothesis. According to the lottery hypothesis, the original model (i.e., the model with randomly initialized parameters) has a sparse winning sub-model at the beginning. Therefore, pruning can be performed with the goal of finding this winning sub-model. At the same time, the inventors discovered through research that the shielding matrix corresponding to the parameter matrix of the first layer in the winning sub-model is the most important among all layers. That is, how to prune the parameter matrix of the first layer in the winning sub-model plays a very important role in the pruning process of the entire model. Therefore, the goal of finding the winning sub-model becomes finding the shielding matrix corresponding to the first layer of the winning sub-model. At the same time, the shielding matrix corresponding to the first layer stabilizes very quickly during the pre-training process and is often fixed in the early stages of training. Therefore, it also provides a possibility for reducing the computational cost of pre-training.

[0074] In summary, the embodiment of the present invention determines when to pause training and obtain the final sparse initialization model, that is, the final target initialization parameters of the model, by monitoring the stability of the shielding matrix corresponding to the first layer of the model. Based on this, the target layer mentioned above can be the first layer. Of course, in practical applications, the target layer can also be several layers including the first layer, such as the first layer and the second layer. However, the number of target layers is much smaller than the total number of layers included in the model. For example, for a deep neural network model, which consists of an input layer, several hidden layers, and an output layer, the first layer is the input layer.

[0075] To facilitate understanding of the pruning solution provided in the above embodiment, Figure 2 , the execution process of the above steps is simplified to make the solution more intuitive. Figure 2As shown in , the model parameters are first randomly initialized, and the randomly initialized deep neural network model is trained using a meta-learning algorithm. Let k represent the number of training rounds. In round k, the first layer of the deep neural network model (i.e., the parameter matrix of the first layer) is pruned to obtain the corresponding masking matrix for the first layer, namely, the masking matrix mk consisting of 0 / 1, where 0 indicates that the parameter value at that position has been pruned. The Hamming distance between the current masking matrix mk and mk-1 is calculated: Hamming(mk,mk-1), where mk-1 represents the masking matrix of the first layer calculated in round k-1. When this distance is less than a given threshold (indicated by a in the figure), training is terminated, indicating that the masking / pruning positions of the first layer of the model have stabilized. Subsequently, the entire model (referring to the initial randomly initialized model) is pruned based on the masking matrices corresponding to each layer in the model obtained in round k (including the masking matrix mk of the first layer) to obtain a sparse model. At this point, the model exhibits the characteristics of a meta-learning model, allowing for rapid convergence. It is also sparse, meaning that training requires relatively little computation. Subsequently, the model is retrained (meta-learning retraining) based on this sparsified model to restore its performance. Training continues using this sparsified model as a starting point until convergence and the final target initialization parameters are obtained.

[0076] like Figure 3 and Figure 4 As shown, the above-mentioned pruning scheme provided by the embodiment of the present invention can be executed on the user terminal device side, or on the server side or the cloud side. Since the above-mentioned pruning scheme is adopted in the above-mentioned model pre-training stage, the computing overhead and storage overhead can be greatly reduced, so it provides the possibility of completing the training of the model on the user terminal device. Of course, if the above-mentioned pre-training stage is executed on the cloud side or the server side, the model with the target initialization parameters obtained when the pre-training stage is completed can also be sent to the user terminal device, so that the user terminal device can continue to complete the training of the model under the target task for the local target task. At this time, the training of the model under the target task can be completed through the traditional machine learning training method.

[0077] In summary, regardless of whether the meta-learning pre-training phase is completed on the user terminal device or in the cloud, the user terminal device can eventually obtain a model with target initialization parameters. Afterwards, the user terminal device can train the model based on the training samples corresponding to the target task obtained locally to obtain the target parameters of the model under the target task.

[0078] Based on this, when the training samples corresponding to the target task collected in the user terminal device involve user privacy and security issues, it can be ensured that these training sample data do not leave the user terminal device, and can also meet the user's model training needs for the target task. Moreover, the computing and storage overhead brought by model training is effectively reduced.

[0079] The model pruning solution provided by the embodiment of the present invention can be applied to any scenario requiring model training with a small number of samples, and can adopt a meta-learning approach to achieve model training for the target task at a relatively low computational cost.

[0080] As mentioned above, the target task can be a classification task, such as image classification, speech classification, etc. In practical applications, for example, a user is a photography enthusiast who often takes photos of landscapes, animals, people, and other objects in daily life. They need to classify and store the photos they took. In this case, the user will have a target classification task: identifying which of multiple classification labels the objects in the photos belong to, and storing photos with the same classification label in a folder.

[0081] In this example scenario, multiple other classification tasks can be collected in advance as multiple training tasks. Based on the model pruning scheme introduced in the above embodiment, a model with target initialization parameters is obtained. The user's user terminal device then uses the model with target initialization parameters to continue training the model using the training samples corresponding to the above target classification task (which can be multiple photos marked with classification labels that have been stored locally), and finally obtains a model suitable for the target classification task. Afterwards, when the user takes another photo, the model can be called to complete the classification and storage of the photo.

[0082] In addition to classification tasks, target tasks can also be recommendation tasks, such as recommending items within a target category. These items can be commodities, films, TV series, songs, literary works, and so on. Accordingly, the multiple training tasks used for model meta-learning training can be recommendation tasks for items within various categories.

[0083] In the recommendation task scenario, the model that needs to be trained can be called a recommendation model. For example, if the project is a commodity, Figure 5 As shown, the multiple training tasks used for meta-learning training of the recommendation model may include, for example, the recommendation task of electronic products, the recommendation task of clothing products, and the recommendation task of food products. The target task may be the recommendation task of cosmetics products.

[0084] In the above example scenario, the training samples corresponding to each training task can be obtained by collecting the purchase, evaluation, attention, and shopping cart operations of the same user (the corresponding user when the user terminal device performs the complete model training process) or a large number of users (a large number of user purchase and other behavior records can be obtained when the model pre-training phase is performed by the e-commerce server). For each corresponding product category, the training samples for the target task can be obtained by collecting the above-mentioned various operation behavior information of the user corresponding to the user terminal device on the corresponding category of products.

[0085] In the example scenario of this recommendation task, a pre-collected recommendation task of multiple categories of items is used, and based on the model pruning scheme introduced in the above embodiment, a model with target initialization parameters is obtained. The model with target initialization parameters has good learning characteristics of the preferences of users with different characteristics for different categories of items. The user terminal device of a certain user then uses the corresponding training samples under the above target recommendation task to continue training the model with target initialization parameters, and finally obtains a model suitable for the target recommendation task of the user. The model is finally run in the user terminal device of the user, and can realize the recommendation service of the target category items corresponding to the target recommendation task for the user.

[0086] The above description only uses classification tasks and recommendation tasks as examples to illustrate application scenarios to which the solutions provided by the embodiments of the present invention can be applied, but the present invention is not limited thereto.

[0087] The following describes in detail the model pruning device of one or more embodiments of the present invention. Those skilled in the art will appreciate that these devices can be constructed using commercially available hardware components and configured according to the steps taught in this solution.

[0088] Figure 6 A schematic diagram of the structure of a model pruning device provided by an embodiment of the present invention is shown in FIG. Figure 6 As shown, the device includes: an acquisition module 11, a pre-training module 12, and a pruning module 13.

[0089] The acquisition module 11 is used to obtain random initialization parameters of the model and multiple training tasks for training the model.

[0090] The pre-training module 12 is used to use the training samples corresponding to each of the multiple training tasks to perform multiple rounds of training on the model to obtain the first model parameters and the second model parameters corresponding to the model after two different rounds of training, respectively. The first model parameters and the second model parameters both include the parameter matrix of each layer in the model.

[0091] The pruning module 13 is used to determine a first shielding matrix corresponding to the first parameter matrix of the target layer in the first model parameters, and a second shielding matrix corresponding to the second parameter matrix of the target layer in the second model parameters; if the similarity between the first shielding matrix and the second shielding matrix is ​​greater than a set threshold, the randomly initialized parameters are pruned according to the shielding matrices corresponding to the parameter matrices of each layer in the second model parameters to obtain third model parameters.

[0092] Optionally, the pre-training module 12 is further configured to train the model based on the third model parameters to obtain target initialization parameters of the model when the training reaches convergence.

[0093] Optionally, the device further includes: a target task training module, configured to train the model based on locally acquired training samples corresponding to the target task and the target initialization parameters, so as to obtain target parameters of the model under the target task.

[0094] Optionally, the multiple training tasks include multiple classification tasks, and the target task is a classification task different from the multiple classification tasks.

[0095] Optionally, the multiple training tasks include recommendation tasks for multiple categories of items, and the target task is a recommendation task for a target category item that is different from the multiple categories of items; the training samples corresponding to the target task include user usage information of the target category item.

[0096] Optionally, the target layer is the first layer in the model.

[0097] Optionally, the pruning module 13 is specifically configured to determine a preset type of distance between the first shielding matrix and the second shielding matrix as the similarity.

[0098] Optionally, the pruning module 13 is specifically used to: sort the parameter values ​​contained in the first parameter matrix of the target layer from large to small; determine the parameter values ​​that are ranked at the back of a set proportion, or determine the parameter values ​​that are less than a set value; and generate the first shielding matrix according to the corresponding positions of the determined parameter values ​​in the first parameter matrix.

[0099] Figure 6 The device shown can execute the steps executed by the user terminal device in the aforementioned embodiment. The detailed execution process and technical effects can be found in the description of the aforementioned embodiment and will not be repeated here.

[0100] In one possible design, the above Figure 6 The structure of the item recommendation device shown can be implemented as a user terminal device, such as a smart phone, PC, tablet computer, etc. Figure 7 As shown, the user terminal device may include: a processor 21, a memory 22, and a communication interface 23. The memory 22 stores executable code, and when the executable code is executed by the processor 21, the processor 21 can at least implement the model pruning method provided in the above embodiment.

[0101] Figure 8 A schematic diagram of the structure of another user terminal device provided in this embodiment, such as Figure 8 As shown, the user terminal device 800 may include one or more of the following components: a processing component 802, a memory 804, a power component 806, a multimedia component 808, an audio component 810, an input / output (I / O) interface 812, a sensor component 814, and a communication component 816.

[0102] The processing component 802 generally controls the overall operation of the user terminal device 800, such as operations associated with display, phone calls, data communications, camera operation, and recording. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the method steps performed by the user terminal device described above. In addition, the processing component 802 may include one or more modules to facilitate interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

[0103] The memory 804 is configured to store various types of data to support operations on the user terminal device 800. Examples of such data include instructions for any application or method operating on the user terminal device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0104] The power supply component 806 provides power to various components of the user terminal device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the user terminal device 800.

[0105] The multimedia component 808 includes a screen that provides an output interface between the user terminal device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor can not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the user terminal device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera can receive external multimedia data. Each front camera and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0106] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), which is configured to receive external audio signals when the user terminal device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 804 or transmitted via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0107] The input / output interface 812 provides an interface between the processing component 802 and peripheral interface modules, such as a keyboard, a click wheel, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0108] The sensor assembly 814 includes one or more sensors for providing various aspects of the status assessment of the user terminal device 800. For example, the sensor assembly 814 can detect the open / closed state of the user terminal device 800, the relative positioning of components, such as the display and keypad of the electronic device 800. The sensor assembly 814 can also detect changes in the position of the user terminal device 800 or a component of the user terminal device 800, the presence or absence of user contact with the user terminal device 800, the orientation or acceleration / deceleration of the user terminal device 800, and temperature changes of the user terminal device 800. The sensor assembly 814 can include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 814 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0109] The communication component 816 is configured to facilitate wired or wireless communication between the user terminal device 800 and other devices. The user terminal device 800 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G or 4G or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0110] In an exemplary embodiment, the user terminal device 800 can be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-mentioned method.

[0111] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is further provided, such as a memory 804 including instructions, and the instructions can be executed by the processor 820 of the user terminal device 800 to perform the above method. For example, the non-transitory computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0112] In addition, an embodiment of the present invention provides a non-temporary machine-readable storage medium, on which executable code is stored. When the executable code is executed by a processor of a user terminal device, the processor can at least implement the project recommendation method provided in the aforementioned embodiment.

[0113] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separate. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Those skilled in the art can understand and implement the present invention without inventive effort.

[0114] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by adding a necessary general hardware platform, and of course can also be implemented by a combination of hardware and software. Based on this understanding, the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a computer product. The present invention can take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0115] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A model pruning method, characterized in that: Applied to user terminals, including: Obtaining random initialization parameters of a model and a plurality of training tasks for training the model, wherein the plurality of training tasks include a plurality of classification tasks, and the classification tasks include at least one of the following: an image classification task and a speech classification task; Performing multiple rounds of training on the model using training samples corresponding to each of the multiple training tasks to obtain first model parameters and second model parameters corresponding to the model after two different rounds of training, respectively, wherein the first model parameters and the second model parameters each include a parameter matrix of each layer in the model; determining a first shielding matrix corresponding to a first parameter matrix of a target layer in the first model parameters, and a second shielding matrix corresponding to a second parameter matrix of the target layer in the second model parameters; If the similarity between the first shielding matrix and the second shielding matrix is ​​greater than a set threshold, pruning the randomly initialized parameters according to the shielding matrices corresponding to the parameter matrices of each layer in the second model parameters to obtain third model parameters; The determining of a first shielding matrix corresponding to a first parameter matrix of a target layer in the first model parameters and a second shielding matrix corresponding to a second parameter matrix of the target layer in the second model parameters comprises: Determine a parameter value in a first parameter matrix of a target layer in the first model parameters, the parameter value of which is less than a set value; generating the first shielding matrix according to positions in the first parameter matrix corresponding to parameter values ​​in the first parameter matrix that are smaller than a set value; Determine a parameter value in the second parameter matrix of the target layer in the second model parameter whose value is less than a set value; The second shielding matrix is ​​generated according to the positions in the second parameter matrix corresponding to the parameter values ​​in the second parameter matrix that are smaller than the set value.

2. The method according to claim 1, characterized in that The method further comprises: The model is trained based on the third model parameters to obtain target initialization parameters of the model when the training reaches convergence.

3. The method according to claim 2, characterized in that The method further comprises: The model is trained based on the locally acquired training samples corresponding to the target task and the target initialization parameters to obtain the target parameters of the model under the target task.

4. The method according to claim 3, characterized in that The target task is a classification task different from the multiple classification tasks; or The multiple training tasks include recommendation tasks for multiple categories of items, and the target task is a recommendation task for a target category item that is different from the multiple categories of items; the training samples corresponding to the target task include user operation behavior information on the target category item.

5. The method according to claim 1, wherein The target layer is the first layer in the model.

6. The method according to claim 1, characterized in that The method further comprises: A distance of a preset type between the first shielding matrix and the second shielding matrix is ​​determined as the similarity.

7. The method according to claim 1, characterized in that The determining of a first shielding matrix corresponding to a first parameter matrix of a target layer in the first model parameters comprises: Sorting the parameter values ​​contained in the first parameter matrix of the target layer from large to small; Determine the parameter value of the rear set ratio; The first shielding matrix is ​​generated according to the positions corresponding to the determined parameter values ​​in the first parameter matrix.

8. A model pruning device, characterized in that: include: An acquisition module, configured to acquire random initialization parameters of a model and a plurality of training tasks for training the model, wherein the plurality of training tasks include a plurality of classification tasks, and the classification tasks include at least one of the following: an image classification task and a speech classification task; a pre-training module, configured to perform multiple rounds of training on the model using training samples corresponding to each of the multiple training tasks, so as to obtain first model parameters and second model parameters corresponding to the model after two different rounds of training, respectively, wherein the first model parameters and the second model parameters each include a parameter matrix of each layer in the model; a pruning module configured to determine a first shielding matrix corresponding to a first parameter matrix of a target layer in the first model parameters, and a second shielding matrix corresponding to a second parameter matrix of the target layer in the second model parameters; If the similarity between the first shielding matrix and the second shielding matrix is ​​greater than a set threshold, pruning the randomly initialized parameters according to the shielding matrices corresponding to the parameter matrices of each layer in the second model parameters to obtain third model parameters; The pruning module is specifically configured to determine a parameter value in a first parameter matrix of a target layer in the first model parameter whose value is less than a set value; generating the first shielding matrix according to positions in the first parameter matrix corresponding to parameter values ​​in the first parameter matrix that are smaller than a set value; Determine a parameter value in the second parameter matrix of the target layer in the second model parameter whose value is less than a set value; The second shielding matrix is ​​generated according to the positions in the second parameter matrix corresponding to the parameter values ​​in the second parameter matrix that are smaller than the set value.

9. A user terminal device, characterized in that: include: A memory, a processor, and a communication interface; wherein the memory stores executable code, and when the executable code is executed by the processor, the processor executes the model pruning method according to any one of claims 1 to 7.

10. A non-transitory machine-readable storage medium, characterized in that The non-transitory machine-readable storage medium stores executable code, and when the executable code is executed by a processor of a user terminal device, the processor is caused to execute the model pruning method according to any one of claims 1 to 7.

Images