Method for training deep learning model, data processing method and device

By sharing the parameters of the deep learning model in the shared parameter pool, using pseudo-random functions to correlate the model parameters and locations, the problem of excessive parameters is solved, efficient compression and deployment of the model is achieved, and the computing power and memory of the terminal device are improved. use.

CN114462571BActive Publication Date: 2024-12-06IFLYTEK CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202111678978.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-12-31
Publication Date
2024-12-06
Estimated Expiration
2041-12-31

AI Technical Summary

Technical Problem

The excessive number of parameters in the deep learning model leads to high hardware configuration requirements, limiting its deployment and use efficiency on end devices.

Method used

By obtaining and updating the parameters of the deep learning model in the shared parameter pool, using pseudo-random functions to correlate the model parameters and the parameter pool locations, multiple model parameters share the same location, reducing the amount of parameters without affecting the model accuracy.

Benefits of technology

Without affecting the model accuracy, the parameters of the deep learning model are significantly compressed, improving the deployment and operation efficiency of the model on resource-limited devices, such as on mobile and embedded devices.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114462571B_ABST
    Figure CN114462571B_ABST
Patent Text Reader

Abstract

The embodiments of the present application provide a method for training a deep learning model, a data processing method and a device. By compressing at least part of the model parameters required by the deep learning model in a shared parameter pool, multiple model parameters can share the parameters stored in the same position in the shared parameter pool, and the parameter amount of the model can be compressed without affecting the accuracy of the model, thereby solving the technical problem of too many parameters of the deep learning model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computers, and specifically to a method for training a deep learning model, a data processing method and a device. Background Art

[0002] With the rapid development of deep learning (DL) technology, various deep learning models have been widely used in image classification, multi-target detection, semantic segmentation, and behavior recognition. However, deep learning models have a very complex structure and millions of parameters, and the hardware configuration required for model operation is very high. Excessive configuration requirements limit the use of deep learning models. Therefore, how to achieve compression of deep learning models has become an important issue that needs to be solved urgently. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a method for training a deep learning model, a data processing method and a device to solve the problem of excessive number of model parameters in the deep learning model and achieve compression of the deep learning model.

[0004] In a first aspect, an embodiment of the present application provides a data processing method for processing data using a deep learning model, the method comprising: obtaining a set of model parameters of the deep learning model in a shared parameter pool, the shared parameter pool comprising multiple positions storing parameter values, at least some of the positions in the shared parameter pool being respectively associated with model parameters in a subset of the set, so that model parameters in the same subset share the parameter value of the same position, and the subsets associated with different positions have no intersection; and inputting the data to be processed into the deep learning model using the set of model parameters to obtain an output result.

[0005] In some embodiments of the present application, each of the model parameters is associated with a position in the shared parameter pool through a pseudo-random function.

[0006] In certain embodiments of the present application, a set of model parameters of a deep learning model is obtained in a shared parameter pool, including: determining a pseudo-random seed; determining the associated position of each model parameter in the shared parameter pool based on the pseudo-random seed and the pseudo-random function; and obtaining the parameter value of each model parameter from the associated position of each model parameter.

[0007] In a second aspect, an embodiment of the present application provides a method for training a deep learning model, the method comprising: before each training of the deep learning model, obtaining a set of model parameters of the deep learning model in a shared parameter pool, the shared parameter pool comprising multiple positions storing parameter values, at least some of the positions in the shared parameter pool being respectively associated with model parameters in a subset of the set, so that the model parameters in the same subset share the parameter value of the same position, and the subsets associated with different positions have no intersection; training the deep learning model using the set of model parameters to obtain training results; and updating the parameter values ​​stored in each position in the shared parameter pool according to the training results.

[0008] In some embodiments of the present application, each model parameter is associated with a position in the shared parameter pool through a pseudo-random function.

[0009] In certain embodiments of the present application, it also includes: determining a shared parameter pool; initializing a deep learning model; determining a pseudo-random seed; and determining the associated positions of each model parameter of the deep learning model in the shared parameter pool based on the pseudo-random seed and the pseudo-random function.

[0010] In some embodiments of the present application, determining the shared parameter pool includes: instantiating the shared parameter pool; and initializing the shared parameter pool.

[0011] In some embodiments of the present application, the method further includes: in response to the deep learning model not converging, obtaining a model parameter set of the deep learning model in the updated shared parameter pool;

[0012] Training a deep learning model using the set of model parameters to obtain a training result; and

[0013] Update the parameter values ​​stored in each location in the shared parameter pool according to the training results.

[0014] In a third aspect, an embodiment of the present application provides a processing device for processing data using a deep learning model, the device comprising: a second acquisition module, used to obtain a set of model parameters of the deep learning model in a shared parameter pool, the shared parameter pool comprising multiple positions storing parameter values, at least some of the positions in the shared parameter pool are respectively associated with model parameters in a subset of the set, so that the model parameters in the same subset share the parameter value of the same position, and the subsets associated with different positions have no intersection; an output module, used to input the data to be processed into the deep learning model using the set of model parameters to obtain an output result.

[0015] In a fourth aspect, an embodiment of the present application provides a device for training a deep learning model, the device comprising:

[0016] A first acquisition module is used to acquire a set of model parameters of the deep learning model from a shared parameter pool before each training of the deep learning model, the shared parameter pool including a plurality of positions storing parameter values, at least some of the positions in the shared parameter pool are respectively associated with model parameters in a subset of the set, so that the model parameters in the same subset share the parameter value of the same position, and the subsets associated with different positions have no intersection;

[0017] A training module, used to train a deep learning model using a set of model parameters to obtain a training result; and

[0018] The update module is used to update the parameter values ​​stored in each position in the shared parameter pool according to the training results.

[0019] In a fifth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program, and the computer program is used to execute the data processing method of the first aspect above, and / or to execute the method of training a deep learning model of the second aspect above.

[0020] In a sixth aspect, an embodiment of the present application provides an electronic device comprising: a processor; and a memory for storing processor executable instructions, wherein the processor is used to execute the data processing method of the first aspect above, and / or to execute the method for training a deep learning model of the second aspect above.

[0021] The embodiments of the present application provide a method for training a deep learning model, a data processing method and a device. By compressing at least part of the model parameters required by the deep learning model in a shared parameter pool, multiple model parameters can share the parameters stored in the same position in the shared parameter pool, and the parameter amount of the model can be compressed without affecting the accuracy of the model, thereby solving the technical problem of too many parameters of the deep learning model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flowchart of a method for training a deep learning model provided by an exemplary embodiment of the present application.

[0023] Figure 2 It is a schematic diagram of the structure of a neural network of another exemplary embodiment of the present application.

[0024] Figure 3 It is a schematic diagram of a shared parameter pool of another exemplary embodiment of the present application.

[0025] Figure 4 It is a flowchart of a method for training a deep learning model provided by another exemplary embodiment of the present application.

[0026] Figure 5It is a flowchart of a method for determining a shared parameter pool provided by another exemplary embodiment of the present application.

[0027] Figure 6 It is a schematic diagram of obtaining a set of model parameters in a shared parameter pool provided by another exemplary embodiment of the present application.

[0028] Figure 7 It is a flowchart of a method for training a deep learning model provided by another exemplary embodiment of the present application.

[0029] Figure 8 It is a flowchart of a method for training a deep learning model provided by another exemplary embodiment of the present application.

[0030] Fig. 9 It is a flowchart of a data processing method provided by another exemplary embodiment of the present application.

[0031] Fig.10 It is a structural diagram of an apparatus for training a deep learning model provided by an exemplary embodiment of the present application.

[0032] Fig.11 It is a structural diagram of a data processing device provided by an exemplary embodiment of the present application.

[0033] Fig.12 It is a block diagram of an electronic device for training deep learning models and data processing provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0034] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] In recent years, with the development of data processing technology, deep learning has been widely used in the field of data processing. Neural networks (NN) are representative algorithms of deep learning. Models based on neural networks, especially convolutional neural networks (CNN), have been widely used in image classification, multi-target detection, semantic segmentation, behavior recognition and other fields due to their excellent performance. However, neural networks have complex structures and millions of parameters, which enable neural networks to simulate and abstract high-level patterns and extremely complex nonlinear relationships, making neural networks have better performance. However, when deploying and running such deep learning models on terminal devices, a large number of parameters will lead to excessive memory requirements for hardware devices. Taking the classification network as an example, the storage size of the trained AlexNet model is about 230M, and the storage size of the VGG16 model is about 527M. As a result, the existing neural network models have problems that are difficult to overcome, such as low efficiency, poor real-time performance and large memory usage. At the same time, because neural networks occupy a large amount of memory on the running device, they hinder their deployment on embedded computing devices with limited memory. In particular, it limits the deployment of convolutional neural networks on terminals with extremely limited hardware computing power and memory, such as mobile terminals, mobile phones, and wearable devices.

[0036] In a comparative example, model parameter compression is achieved through the idea of ​​weight sharing. This solution allows multiple weights in a neural network to share the same parameter value, thereby reducing the number of parameters and the actual storage requirements. Specifically, first, the hash map corresponding to each layer of network parameters is mapped to different hash buckets using a series of low-cost hash functions, and all connections in the same hash bucket are allowed to share the same parameter value. Secondly, during network training, all hash maps and parameters in the corresponding hash buckets are jointly optimized under constraints such as matrix orthogonality; finally, when the network is deployed, only the corresponding hash function and the parameters in the corresponding hash bucket need to be saved, and the corresponding parameters are retrieved and combined for reasoning through hash table lookup.

[0037] However, this proportional method assigns an independent hash map to be optimized to each layer of the neural network model. This operation will make the optimization process of parameters and hash matrices too complicated when the neural network structure is deepened. On the one hand, this approach cannot share cross-layer joint parameters between layers, which limits the overall compression ratio of the network and prevents extreme compression. On the other hand, multiple hash matrices impose a heavy burden on the training and joint optimization process of parameters, and it is difficult to guarantee convergence, resulting in this solution causing the network to eventually suffer from precision loss, and this situation is particularly prominent in some compact basic model structures (such as MobileNet models, EfficientNet models, etc.).

[0038] In other comparative examples, model compression technologies such as pruning, distillation, tensor low-rank decomposition, and quantization are used to solve the problem of the huge number of parameters of the neural network model, achieving the effect of model compression and acceleration. Pruning technology can be mainly divided into unstructured pruning and structured pruning technology, which respectively achieve the effect of reducing the number of parameters by removing some parameters in the network in a fine-grained and coarse-grained manner. Knowledge distillation relies on the transfer of knowledge (intermediate information) of the teacher model (full parameters) to guide the training of the student model (a small number of parameters) to achieve overall compression of the model. Tensor low-rank decomposition mostly uses matrix decomposition technologies such as SVD decomposition to decompose complex and high-dimensional matrix calculations into multiple low-dimensional matrix series calculations, thereby reducing the amount of network calculations and the amount of parameters required as a whole. Network quantization mainly reduces the amount of model parameters by quantizing floating-point weights to low-bit fixed-point weights (for example, 8 bits or even 1 bit) instead of directly compressing the model's parameters. Instead, it reduces the memory consumption required when the model is finally saved by limiting the accuracy of data storage. However, the compression effect of the above methods is limited.

[0039] In order to solve the technical problem of too many parameters of the deep learning model, the embodiment of the present application adopts parameter sampling and shared joint training methods to compress all model parameters of the deep learning model into a shared parameter pool without introducing additional auxiliary parameters and without increasing the difficulty of model optimization. Multiple model parameters can share the parameter values ​​stored in the same position in the parameter pool, which can compress the model parameter quantity without affecting the model accuracy.

[0040] The method of the embodiment of the present application can be applied to various deep learning frameworks, such as PaddlePaddle, Tensorflow, Caffe, Theano, MXNet, Torch, PyTorch, etc.

[0041] Figure 1It is a flowchart of a method for training a deep learning model provided by an exemplary embodiment of the present application. Figure 1 The method is performed by a computing device, such as a server. Figure 1 As shown, the method includes the following contents.

[0042] 110: Before each training of the deep learning model, multiple groups of model parameters of the neural network model are obtained in the shared parameter pool. The shared parameter pool includes multiple locations storing parameter values, and at least some of the locations in the shared parameter pool are respectively associated with model parameters in a subset of the set, so that the model parameters in the same subset share the parameter value of the same location, and the subsets associated with different locations have no intersection. In this embodiment, the framework of the deep learning model is an example of a neural network model, and the neural network is a complex network system formed by a large number of simple nodes that are widely connected to each other.

[0043] Figure 2 Schematic diagram of the structure of a neural network model of an exemplary embodiment of the present application. In one embodiment of the present application, the structure of the neural network model is as follows Figure 2 As shown, Figure 2 The neural network model shown has the following parts: input layer x, hidden layer (including the first hidden layer h1 and the second hidden layer h2), output layer y. Each node is a neuron, and each neuron in each layer is connected to the neurons in the next layer. The input layer x is the feature extracted by the neural network, and the output layer y is the desired result.

[0044] Figure 3 Schematic diagram of a shared parameter pool of an exemplary embodiment of the present application. Figure 3 As shown, the shared parameter pool 300 includes four positions. Each position of the shared parameter pool 300 stores a parameter value.

[0045] In one embodiment of the present application, Figure 2 The model parameters of multiple nodes of the neural network model in Figure 3 The positions of the shared parameter pool 300 in are associated by a pseudo-random function (PRF). Further, Figure 2 The model parameters of multiple nodes of the neural network model in Figure 3The position of the shared parameter pool 300 in the neural network model is associated with a pseudo-random seed (PRS, Pseudo Random Seed) and a pseudo-random function. That is to say, the model parameters of multiple nodes in the neural network model and the position of the shared parameter pool have a predetermined association relationship, and the association relationship can be determined in advance by a pseudo-random sampling function and a pseudo-random seed. In other words, the model parameters of each node correspond to an associated position, and the associated positions of different nodes can be the same. Each position in the shared parameter pool can be associated with the model parameters of one, two or more nodes. Specifically, multiple nodes corresponding to the same position can be regarded as a group of nodes, and each model parameter in the same group is respectively associated with the same position in the shared parameter pool, and multiple nodes in a group can share the parameters stored in the same position in the shared parameter pool. Among them, the nodes in each group can be one, two, three and more. When there are two or more nodes in a group, each node in the same group can be located at the same layer of the neural network or at different layers of the neural network.

[0046] Acquiring multiple model parameters of the neural network model is to obtain the parameters stored at the associated position of each node from the shared parameter pool. In one embodiment of the present application, Figure 2 and Figure 3 As shown, nodes a and f are a group, which are associated with position 0 in the shared parameter pool. To obtain the parameters of nodes a and f is to obtain the parameter values ​​1.6 stored in position 0 in the shared parameter pool. Nodes c, g, and h are another group, which are associated with position 1 in the shared parameter pool. To obtain the parameters of nodes c, g, and h is to obtain the parameter values ​​2.3 stored in position 1 in the shared parameter pool. Similarly, nodes b and d are another group and node e is another group. The nodes in these two groups also have predetermined associated positions. The model parameters in these two groups are obtained according to the associated positions.

[0047] 120: Train the deep learning model using the parameters to obtain training results.

[0048] Specifically, the neural network model uses the multiple model parameters determined by step 110 and uses the training set to train the neural network model. In one embodiment of the present application, the neural network model can be a model framework based on EfficientNet B0, which is used for ImageNet image classification tasks. The training set used to train the model is ILSVRC2012, in which the training data includes 1000 categories and 1.2 million data; the test data includes 1000 categories, 50 samples per category, and 5 million data. For example, the training data is used to train the neural network model using the forward method to obtain the training results.

[0049] 130: Update the parameter values ​​stored in each position in the shared parameter pool according to the training results.

[0050] Specifically, all the accumulated gradients are returned to the leaf node (each parameter position) through gradient backpropagation, and the optimizer is called to update the parameters stored in the shared parameter pool.

[0051] In this embodiment, the positions of multiple model parameters in the neural network model and the shared parameter pool are associated through a pseudo-random function, and the neural network model is updated by updating the parameters in the shared parameter pool. During the training process, multiple model parameters of the neural network model are obtained from various positions in the shared parameter pool according to the pseudo-random function, and multiple model parameters obtained from the same position can be updated at the same time, so that different nodes of the neural network model can share the parameters stored in one position in the shared parameter pool. In this embodiment of the application, all model parameters of the neural network model can be obtained in the shared parameter pool through a fixed pseudo-random function, and multiple model parameters across layers can be shared, which greatly compresses the neural network model and can achieve compression of the neural network model.

[0052] The training method of the embodiment of the present application can be used to train models of most deep learning frameworks. For example, the number of parameters of models such as MobileNet and EfficientNet frameworks can be compressed by 5-10 times.

[0053] The method shown in steps 110-130 of this example can be considered as an iterative process in the neural network model training process. In another embodiment of the present application, the difference from the previous embodiment is that a back propagation algorithm can also be used to implement an iterative process using part of the training data.

[0054] It should be understood that in the present embodiment, the set of model parameters obtained in the shared parameter pool refers to the parameters of all nodes in the neural network model. In another embodiment of the present application, the set of model parameters obtained in the shared parameter pool may also be the parameters of some nodes in a neural network model. For example, when the neural network model includes n hidden layers, multiple nodes of nx (x is an integer greater than 1) hidden layers have an associated relationship with the positions of the shared parameter pool. In another embodiment of the present application, the set of model parameters obtained in the shared parameter pool may also be the parameters of all nodes of the model obtained from two different shared parameter pools. For example, a part of the nodes of the neural network model are associated with the positions of the first shared parameter pool, and another part of the nodes of the neural network model are associated with the positions of the second shared parameter pool.

[0055] Figure 4FIG. 1 is a flow chart of a method for training a deep learning model provided by another exemplary embodiment of the present application. Figure 4 As shown, the training method of the neural network model of the present application includes the following steps:

[0056] 410: Determine the shared parameter pool.

[0057] Specifically, the size of the shared parameter pool is determined according to the desired compression ratio. For example, the size of the shared parameter pool is (m,).

[0058] Figure 5 FIG. 1 is a flow chart of a method for determining a shared parameter pool provided by another exemplary embodiment of the present application. Figure 5 As shown, determining the shared parameter pool includes the following steps:

[0059] 510: Instantiate the shared parameter pool.

[0060] Specifically, the shared parameter pool is instantiated as a one-dimensional tensor structure. Instantiating the shared parameter pool can facilitate the subsequent establishment of an association relationship between the positions in the shared parameter pool and the model parameters in the neural network model.

[0061] 520: Initialize the shared parameter pool.

[0062] In one embodiment of the present application, a convolution-like initialization method (e.g., Kaiming initialization) can be used to initialize the parameters stored in the shared parameter pool. In this embodiment, the shared parameter pool can be initialized using the initialization method in formula (1). Specifically, the shared parameter pool Params_pool instantiated in step 510 is equivalently mapped to a four-dimensional tensor form through a reshape operation, and then the standard deviation std and the boundary bound are calculated, and finally the parameters in the shared parameter pool Params_pool are sampled from the upper and lower bounds (-bound, bound), and reshaped back to the one-dimensional data structure of the original shared parameter pool.

[0063]

[0064] In this embodiment, the Kaiming initialization method is used to initialize the shared parameter pool, which has great advantages in convergence and can improve the training efficiency and training accuracy of the neural network model.

[0065] It should be understood that this embodiment is illustrated by taking the training of a convolutional neural network model as an example. In fact, different initialization methods of the shared parameter pool can be determined according to different model frameworks. For example, the parameters stored in the shared parameter pool can also be initialized using a fully connected initialization method.

[0066] 420: Initialize the deep learning model.

[0067] Specifically, different model frameworks and corresponding training sets can be determined according to the specific application scenarios of the deep learning model. In one embodiment of the present application, the neural network model is used for the ImageNet image classification task, and the model framework based on EfficientNet B0 can be determined according to the task requirements. The network structure of EfficientNet B0 is shown in Table 1, where Conv is a normal convolution structure, K3*3 represents the convolution kernel size (Kernel size), and MBConv is the basic block structure in the EfficientNet network. Each MBConv structure consists of 2 1*1Conv, a fully connected SE block and 1 Conv3*3.

[0068] Table 1 Efficient B0 network structure

[0069]

[0070] The corresponding neural network multiple model parameter lists are generated through the structure in Table 1, and the EfficientNet b0 network structure is instantiated.

[0071] 430: Determine the pseudo-random seed.

[0072] Specifically, the pseudo-random seed is used to determine the association between the position in the shared parameter pool and each model parameter of the neural network model.

[0073] 440: Establishing random associations between multiple positions and model parameters in a shared parameter pool according to a pseudo-random function.

[0074] Specifically, the pseudo-random number is continuously generated by iterating the pseudo-random seed and pseudo-random function determined in step 430 to determine an index (INDEX) for each model parameter, and the INDEX indicates the sampling position of each model parameter in the shared parameter pool. The sampling positions of different model parameters can be repeated.

[0075] Pseudo-random numbers are random number sequences calculated using a deterministic algorithm. Pseudo-random numbers are not truly random, but have statistical characteristics similar to random numbers, such as uniformity and independence. When the pseudo-random seed and sampling function remain unchanged, the generated pseudo-random numbers are fixed. In this embodiment, the pseudo-random function can specifically be the random number generation (random.choice) function that comes with Python, etc.

[0076] In one embodiment of the present application, in the initialization function of each operation with parameters, the INDEX (repeatable) corresponding to the number of parameters required by the function is sampled in the shared parameter pool. After the INDEX of each model parameter is determined, the INDEX will not be modified during the entire training process. That is, it is equivalent to binding a fixed INDEX to the model parameter. This INDEX is generated by a pseudo-random seed and a pseudo-random function. Therefore, during the entire training process, the model parameters of the neural network model are always stored in a fixed position in the shared parameter pool. When a certain model parameter is needed later, the INDEX of the model parameter is determined by the pseudo-random seed and the pseudo-random function and sampled in the shared parameter pool.

[0077] By determining the INDEX of the model parameters through pseudo-random seeds and pseudo-random functions, the storage of a large number of INDEXs can be avoided, and the effect of the neural network model during training and testing can be perfectly guaranteed to be completely consistent. Therefore, by storing and using the model parameters of the neural network model in the form of a shared parameter pool, the compression ratio of the parameters of the neural network model can be flexibly controlled.

[0078] 450: Obtain a set of model parameters of a deep learning model in a shared parameter pool.

[0079] Specifically, the pseudo-random seed and pseudo-random function determined according to steps 410-440 can determine the INDEX of each model parameter to determine the sampling position of each model parameter in the shared parameter pool, and obtain each model parameter from the sampling position in the shared parameter pool.

[0080] Figure 6 FIG. 1 is a schematic diagram of a scenario of obtaining a set of model parameters in a shared parameter pool provided by another exemplary embodiment of the present application. Figure 6 As shown, in one embodiment, the parameters of two convolutions in a convolutional neural network are obtained as an example. The neural network model is a conventional convolution Conv3*3. The size of the shared parameter pool 600 is (6,). The parameters of the two convolutions of the neural network model are obtained from the shared parameter pool 600, specifically the first convolution parameter matrix 610 and the second convolution parameter matrix 620. Specifically, the shapes of the two convolutions are both N in *N out *k*k(N in =1,N in =1, k=3), group is 1. The first convolution parameter matrix 610 and the second convolution parameter matrix 620 include 9 parameters respectively.

[0081] The parameters in the first convolution parameter matrix 610 and the second convolution parameter matrix 620 have been pre-determined by the pseudo-random seed and the pseudo-random function through steps 410-440. INDEX. In this step, the same pseudo-random seed and pseudo-random function as in step 440 are used to determine the INDEX of the two convolutions, which are the first index 611 and the second index 621. The first index 611 and the second index 621 can indicate the position of the 9 model parameters in each convolution in the shared parameter pool. According to the first index 611 and the second index 621, the parameters stored in different positions of the shared parameter pool are obtained from the shared parameter pool 600 to determine the parameter values ​​in the first convolution parameter matrix 610 and the second convolution parameter matrix 620.

[0082] In this embodiment, the shared parameter pool 600 is a 1-dimensional tensor of size (6,). The 9 parameters in each of the first convolution parameter matrix 610 and the second convolution parameter matrix 620 are all sampled and taken out according to the INDEX sampled and bound during the model instantiation process and then reshaped. In the subsequent training process, the sampled parameters are injected into the functional method for forward calculation to update the shared parameter pool. The total number of model parameters of the neural network model is 18. The 18 model parameters in the neural network model can be perfectly represented by a shared parameter pool of size (6,), which is equivalent to obtaining 18 model parameters using a shared parameter pool including 6 parameter values, achieving a three-fold compression effect on the parameter quantity.

[0083] The embodiment of the present application shares all parameters into a parameter pool (there is only one parameter space). When the network is constructed, a fixed parameter pool INDEX sampling (corresponding to a position in the parameter pool) is bound to the parameters corresponding to each convolution. A pseudo-random sampling method is used. By fixing the pseudo-random seed, the INDEX is fixed by fixed seed sampling during the network instantiation process, which can perfectly avoid the storage of a large number of INDEXs and perfectly ensure the consistency of the training and testing process. After that, the training of the neural network model is also calculated according to this index sampling parameter combination, so that the network can adapt to the sampled parameter sharing combination during the training process, so that only the parameters in the parameter pool need to be updated, so that the compression ratio can be flexibly controlled.

[0084] Figure 7 FIG. 1 is a flow chart of a method for training a deep learning model provided by another exemplary embodiment of the present application. Figure 7 As shown, the method for training a deep learning model includes the following contents.

[0085] 710: Obtain a set of model parameters of the deep learning model in a shared parameter pool.

[0086] 720: Train a deep learning model using a set of model parameters, and update parameter values ​​stored at each location in the shared parameter pool.

[0087] Specifically, each operation is reshaped through the bound INDEX sampling parameter and injected into the functional method for forward calculation to obtain the final output of the network. After that, all gradient accumulations are returned to the leaf node (each parameter position in the model) through gradient backpropagation, and the optimizer is called to update the parameters.

[0088] 730: Determine whether the preset number of iteration rounds has been reached.

[0089] If "yes", execute step 740. If "no", repeat step 710. Specifically, in response to the neural network model not reaching the predetermined number of iterations, repeat step 710.

[0090] 740: Use parameters in the shared parameter pool as deep learning.

[0091] Specifically, in response to the neural network model reaching a predetermined number of iterations, the parameters in the shared parameter pool are used as the neural network model.

[0092] Specifically, the shared parameter pool is saved. When the iteration reaches a preset number of rounds, the parameters in the shared parameter pool can be saved in a predetermined format. For example, the data in the shared parameter pool is saved in the form of a .pth file.

[0093] Figure 8 FIG. 1 is a flow chart of a method for training a deep learning model provided by another exemplary embodiment of the present application. Figure 8 As shown, the method for training a deep learning model includes the following contents.

[0094] 810: Instantiate the shared parameter pool.

[0095] Specifically, according to the required parameter restrictions, a 1-dimensional tensor structure is used to instantiate the parameter pool.

[0096] 820: Initialize the shared parameter pool.

[0097] Specifically, the Kaiming initialization method is adopted to initialize the parameters in the shared parameter pool with reference to the convolution form.

[0098] 830: Initialize the entire model.

[0099] Specifically, it includes determining a basic model structure list and a pseudo-random seed.

[0100] 840: Determine the association of model parameters and the location of the shared parameter pool.

[0101] A series of sampling INDEX in the parameter pool is bound to each operation (op) with parameters through the pseudo-random sampling tool (random.choice) and the pseudo-random seed.

[0102] 850: Instantiate the model.

[0103] Specifically, the model is instantiated using Dataset, Dataloader, Adam optimizer, Cosine Annealing LR and other methods. Specifically, Adam is used as the optimizer, and Cosine Annealing LR is used as the learning rate update strategy. After that, a batch (256 samples) of data is continuously taken from the Dataloader, and the model forward training process is called.

[0104] 860: Determine whether the preset number of iteration rounds has been reached.

[0105] Specifically, if the answer is “yes”, execute step 890 ; if the answer is “no”, execute step 870 .

[0106] 870: Training model.

[0107] Specifically, in the forward function, the corresponding parameter value of each operation is sampled from the shared parameter pool through the bound INDEX, and the parameter is injected into the calculation process of the operation using the functional calculation method.

[0108] 880: Update the shared parameter pool.

[0109] Specifically, the accumulated gradients are passed back to the shared parameter pool through the optimizer and updated, and the training-related variables are updated.

[0110] 890: Save the shared parameter pool.

[0111] Specifically, the shared parameter pool, pseudo-random seed, pseudo-random function and model framework are saved as the finalized neural network model.

[0112] Fig. 9 FIG. 1 is a flow chart of a data processing method provided by another exemplary embodiment of the present application. The data processing method of this embodiment processes data using a deep learning model trained by the method of the above embodiment. Fig. 9 As shown, the data processing method includes the following contents.

[0113] 910: Obtain a set of model parameters of the deep learning model in a shared parameter pool. The shared parameter pool includes multiple locations storing parameter values, and the set of model parameters includes multiple subsets. Model parameters in different subsets are respectively associated with different locations in the shared parameter pool, and model parameters in the same subset are respectively associated with the same location in the shared parameter pool, with parameter values ​​stored in the shared locations.

[0114] Specifically, the shared parameter pool, pseudo-random seed, and pseudo-random function in the embodiment of the present application are data saved after being updated by the method of the above embodiment. Specifically, the shared parameter pool, pseudo-random seed, and pseudo-random function can all be stored in a terminal, for example, the terminal can be a server for executing the data processing method of this embodiment.

[0115] Specifically, INDEX is sampled in the shared parameter pool. Since the pseudo-random seed is fixed, the INDEX sampled in the shared parameter pool for each operation with parameters in the network is also fixed, ensuring that the inference process is completely consistent with the training process.

[0116] Specifically, the position of each model parameter in the shared parameter pool is determined according to the pseudo-random seed and the pseudo-random function, and the values ​​of all parameters of the deep learning model are obtained from the shared parameter pool. The specific method of obtaining the set of model parameters of the deep learning model in the shared parameter pool can be referred to above. Figure 1-Figure 8 In order to avoid repetition, the description of the method provided in the corresponding embodiment will not be repeated here.

[0117] 920: Input the data to be processed into a deep learning model using a set of model parameters to obtain an output result.

[0118] Call the model forward operation process. During the operation, each operation is reshaped through the INDEX sampling parameter bound in the previous step and injected into the functional method for forward calculation to obtain the predicted output of the model.

[0119] An embodiment of the present application provides a data processing device, which achieves flexible compression control of the number of model parameters of the deep learning model by representing all parameters in the deep learning model only through a compact shared parameter pool sampling space, binds an index to each model parameter and samples the parameter from the shared parameter pool during the deep learning model training process, and optimizes the parameters in the shared parameter pool through joint gradient optimization. At the same time, in the process of instantiating the model construction, a fixed pseudo-random seed is used to ensure the consistency of the bound sampling indexes for training and testing through pseudo-random sampling, thereby avoiding the storage of a large number of index parameters.

[0120] Fig.10 is a schematic diagram of the structure of a device for training a deep learning model provided by an exemplary embodiment of the present application. Fig.10 As shown, the model training device 1000 includes: a first acquisition module 1010, a training module 1020 and an update module 1030.

[0121] The first acquisition module 1010 is used to obtain a set of model parameters of the deep learning model in a shared parameter pool before each training of the deep learning model. The shared parameter pool includes multiple positions storing parameter values. At least some of the positions in the shared parameter pool are respectively associated with model parameters in a subset of the set, so that the model parameters in the same subset share the parameter value of the same position, and subsets associated with different positions have no intersection.

[0122] The training module 1020 is used to train the deep learning model using the set of model parameters to obtain training results.

[0123] The updating module 1030 is used to update the parameter values ​​stored in each position in the shared parameter pool according to the training results.

[0124] An embodiment of the present application provides a device for training a deep learning model. By compressing at least part of the model parameters required by the deep learning model in a shared parameter pool, multiple model parameters can share the parameters stored in the same position in the shared parameter pool, thereby compressing the model parameter quantity without affecting the model accuracy, thereby solving the technical problem of too many parameters of the deep learning model.

[0125] According to an embodiment of the present application, each model parameter is associated with the position of the shared parameter pool through a pseudo-random function.

[0126] According to one embodiment of the present application, the first acquisition module 1010 is used to determine a pseudo-random seed; determine the associated position of each model parameter in the shared parameter pool based on the pseudo-random seed and the pseudo-random function; and obtain the parameter value of each model parameter from the associated position of each model parameter.

[0127] According to an embodiment of the present application, the model training device 1000 also includes: a determination module, an initialization module, a second determination module and an association module.

[0128] The first determining module is used to determine a shared parameter pool.

[0129] The initialization module is used to initialize the deep learning model.

[0130] The second determination module is used to determine the pseudo-random seed.

[0131] The association module is used to determine the associated positions of various model parameters of the deep learning model in the shared parameter pool according to the pseudo-random seed and the pseudo-random function.

[0132] According to an embodiment of the present application, the first determining unit 1040 is used to instantiate a shared parameter pool; and initialize the shared parameter pool.

[0133] According to an embodiment of the present application, the shared parameter pool is a one-dimensional tensor.

[0134] According to an embodiment of the present application, the model training device 1000 also includes a third determination module.

[0135] The third determination module is used to use the parameter values ​​in the shared parameter pool as the deep learning model in response to the deep learning model converging.

[0136] According to an embodiment of the present application, the model training device 1000 also includes: a third determination module, a fourth determination module and a storage module.

[0137] The third determination module is used to obtain a model parameter set of the deep learning model from the updated shared parameter pool in response to the deep learning model failing to converge.

[0138] The third determination module is used to train the deep learning model using the set of model parameters to obtain a training result.

[0139] The storage module is used to update the parameter values ​​stored in each location in the shared parameter pool according to the training results.

[0140] It should be understood that the specific working process and functions of each module in the above embodiment can refer to the above Figures 1 to 9 In order to avoid repetition, the description of the method provided in the embodiment will not be repeated here.

[0141] Fig.11 Schematic diagram of the structure of a data processing device provided by an exemplary embodiment of the present application. Fig.11 As shown, the data processing device 1100 includes: a second acquisition module 1110 and an output module 1120 .

[0142] The second acquisition module 1110 is used to obtain a set of model parameters of the deep learning model in a shared parameter pool, the shared parameter pool includes multiple positions storing parameter values, at least some of the positions in the shared parameter pool are respectively associated with model parameters in a subset of the set, so that the model parameters in the same subset share the parameter value of the same position, and the subsets associated with different positions have no intersection.

[0143] The output module 1120 is used to input the data to be processed into a deep learning model using a set of model parameters to obtain an output result.

[0144] An embodiment of the present application provides a data processing device, which achieves flexible compression control of the number of model parameters of the deep learning model by representing all parameters in the deep learning model only through a compact shared parameter pool sampling space, binds an index to each model parameter and samples the parameter from the shared parameter pool during the deep learning model training process, and optimizes the parameters in the shared parameter pool through joint gradient optimization. At the same time, in the process of instantiating the model construction, a fixed pseudo-random seed is used to ensure the consistency of the bound sampling indexes for training and testing through pseudo-random sampling, thereby avoiding the storage of a large number of index parameters.

[0145] It should be understood that the specific working process and functions of the second acquisition module 1110 and the output module 1120 in the above embodiment can refer to the above Figure 1-Figure 9 To avoid repetition, the description of the method provided in the embodiment will not be repeated here.

[0146] Fig.12 It is a block diagram of an electronic device 1200 for training deep learning models and data processing provided by an exemplary embodiment of the present application.

[0147] Reference Fig.12 , the electronic device 1200 includes a processing component 1210, which further includes one or more processors, and a memory resource represented by a memory 1220 for storing instructions executable by the processing component 1210, such as an application. The application stored in the memory 1220 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1210 is configured to execute instructions to perform the above-mentioned method of training a deep learning model, or a data processing method.

[0148] The electronic device 1200 may also include a power supply component configured to perform power management of the electronic device 1200, a wired or wireless network interface configured to connect the electronic device 1200 to a network, and an input / output (I / O) interface. The electronic device 1200 may be operated based on an operating system stored in the memory 1220, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, or the like.

[0149] A non-temporary computer-readable storage medium, when the instructions in the storage medium are executed by the processor of the above-mentioned electronic device 1200, enables the above-mentioned electronic device 1200 to execute a method for training a deep learning model, including: before each training of the deep learning model, obtaining a set of model parameters of the deep learning model in a shared parameter pool, the shared parameter pool including multiple positions storing parameter values, at least some of the positions in the shared parameter pool are respectively associated with model parameters in a subset of the set, so that the model parameters in the same subset share the parameter value of the same position, and the subsets associated with different positions have no intersection; training the deep learning model using the set of model parameters to obtain training results; and updating the parameter values ​​stored in each position in the shared parameter pool according to the training results.

[0150] Alternatively, a data processing method comprises: obtaining a set of model parameters of a deep learning model in a shared parameter pool, the shared parameter pool comprising a plurality of positions storing parameter values, at least some of the positions in the shared parameter pool being respectively associated with model parameters in a subset of the set, so that the model parameters in the same subset share the parameter value of the same position, and subsets associated with different positions have no intersection; and inputting the data to be processed into the deep learning model using the set of model parameters to obtain an output result.

[0151] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present application, and will not be described one by one here.

[0152] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0153] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0154] In the several embodiments provided in the present application, it should be understood that the disclosed methods, devices and electronic devices can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0155] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0156] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0157] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program check codes. It should be noted that in the description of the present application, the terms "first", "second", "third", etc. are only used for descriptive purposes and cannot be understood as indicating or implying relative importance. In addition, in the description of the present application, unless otherwise specified, the meaning of "multiple" is two or more.

[0158] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A data processing method for processing data using a deep learning model, wherein the deep learning model is used for image classification, characterized in that: include: Obtaining a set of model parameters of the deep learning model in a shared parameter pool, wherein the shared parameter pool includes a plurality of positions storing parameter values, and at least some of the positions in the shared parameter pool are respectively associated with model parameters in a subset of the set, so that model parameters in the same subset share the parameter value of the same position, and the subsets associated with different positions have no intersection; as well as The data to be processed is input into the deep learning model using the set of model parameters to obtain an output result.

2. The method according to claim 1, characterized in that The position of each of the model parameters is associated with the shared parameter pool through a pseudo-random function.

3. The method according to claim 1, characterized in that The step of obtaining a set of model parameters of the deep learning model in a shared parameter pool includes: Determine the pseudo-random seed; Determining the associated position of each of the model parameters in the shared parameter pool according to the pseudo-random seed and the pseudo-random function; and The parameter value of each of the model parameters is obtained from the associated position of each of the model parameters.

4. A method for training a deep learning model for image classification, characterized in that: include: Before each training of the deep learning model, a set of model parameters of the deep learning model is obtained from a shared parameter pool, wherein the shared parameter pool includes a plurality of positions storing parameter values, and at least some of the positions in the shared parameter pool are respectively associated with model parameters in a subset of the set, so that model parameters in the same subset share the parameter value of the same position, and the subsets associated with different positions have no intersection; Training the deep learning model using the set of model parameters to obtain a training result; as well as The parameter values ​​stored at each position in the shared parameter pool are updated according to the training results.

5. The method according to claim 4, characterized in that The position of each of the model parameters is associated with the shared parameter pool through a pseudo-random function.

6. The method according to claim 4, characterized in that Also includes: Determining the shared parameter pool; Initializing the deep learning model; Determine the pseudo-random seed; as well as Determine the associated position of each of the model parameters of the deep learning model in the shared parameter pool according to the pseudo-random seed and the pseudo-random function.

7. The method according to claim 5, characterized in that Determining the shared parameter pool includes: Instantiating the shared parameter pool; and Initialize the shared parameter pool.

8. The method according to claim 4, characterized in that Also includes: In response to the deep learning model not converging, obtaining a set of model parameters of the deep learning model in an updated shared parameter pool; Training the deep learning model using the set of model parameters to obtain a training result; as well as The parameter values ​​stored at each position in the shared parameter pool are updated according to the training results.

9. A data processing device for processing data using a deep learning model, wherein the deep learning model is used for image classification, characterized in that: include: A second acquisition module is used to obtain a set of model parameters of the deep learning model in a shared parameter pool, wherein the shared parameter pool includes a plurality of positions storing parameter values, and at least some of the positions in the shared parameter pool are respectively associated with model parameters in a subset of the set, so that model parameters in the same subset share the parameter value of the same position, and the subsets associated with different positions have no intersection; An output module is used to input the data to be processed into the deep learning model using the set of model parameters to obtain an output result.

10. A device for training a deep learning model, wherein the deep learning model is used for image classification, characterized in that: include: A first acquisition module is used to acquire a set of model parameters of the deep learning model from a shared parameter pool before each training of the deep learning model, wherein the shared parameter pool includes a plurality of positions storing parameter values, and at least some of the positions in the shared parameter pool are respectively associated with model parameters in a subset of the set, so that model parameters in the same subset share the parameter value of the same position, and the subsets associated with different positions have no intersection; A training module, used to train the deep learning model using the set of model parameters to obtain a training result; as well as An updating module is used to update the parameter values ​​stored in each position in the shared parameter pool according to the training results.

11. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, which is used to execute the data processing method described in any one of claims 1 to 3, and / or to execute the method for training a deep learning model described in any one of claims 4 to 8.

12. An electronic device, characterized in that: The electronic device comprises: processor; a memory for storing instructions executable by the processor, The processor is used to execute the data processing method described in any one of claims 1 to 3, and / or to execute the method for training a deep learning model described in any one of claims 4 to 8.