Method and apparatus for training neural networks

Abstract

The application relates to the field of artificial intelligence, and provides a neural network training method and device, which can realize fine-grained control of parameter groups of a neural network in the iteration step dimension, improves training precision while accelerating training. The method comprises the following steps: obtaining a neural network to be trained; grouping parameters of the neural network to be trained to obtain M parameter groups, M being a positive integer greater than or equal to 1; obtaining a sampling probability distribution and a training iteration step arrangement mode, the sampling probability distribution being used for representing the probability of each parameter group in the M parameter groups being sampled in each training iteration step, and the training iteration step arrangement mode comprising interval arrangement and period arrangement; freezing or stopping updating of the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement mode; and training the neural network to be trained according to the frozen parameter groups or the stopped-updating parameter groups.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, in particular to a neural network training method and device. BACKGROUND

[0002] Deep learning technology has made great progress in computer vision. Taking image recognition as an example, a deep neural network model has led traditional computer vision methods since 2012 in the ImageNet large scale visual recognition challenge (ILSVRC) with a huge advantage. The ImageNet (ILSVRC2012) dataset has about 1.28 million images, and it takes about 8 hours to train 90 rounds using a ResNet50 neural network on 8 V100 computing cards. The GPT-3 model released by OpenAI has about 175 billion parameters, and it takes 45 TB of data to train, and it costs 13 million US dollars to train once. As the size of the dataset becomes larger and the number of network model parameters becomes larger, while obtaining a model with higher precision, the time and money spent on training the model also increases. Therefore, how to accelerate the training of neural networks has become a problem to be solved. SUMMARY

[0003] The present application provides a neural network training method and device, which can achieve fine-grained control of parameter groups of a neural network in the dimension of iteration steps, improving training accuracy while accelerating training.

[0004] In a first aspect, a neural network training method is provided, which includes: obtaining a neural network to be trained; grouping parameters of the neural network to be trained to obtain M groups of parameters, M being a positive integer greater than or equal to 1; obtaining a sampling probability distribution and a training iteration step arrangement, the sampling probability distribution being used to represent the probability of each group of parameters in the M groups of parameters being sampled in each training iteration step, and the training iteration step arrangement including interval arrangement and periodic arrangement; freezing or stopping updating the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement; and training the neural network to be trained according to the frozen parameter groups or the stopped-updating parameter groups.

[0005] The neural network training method of the present application embodiment processes the parameter groups of the neural network in the dimension of iteration steps, realizes fine-grained control of the acceleration process, improves training accuracy while accelerating training. By using the training iteration step arrangement and the sampling probability distribution to sample and process the parameter groups, the training cost and the training accuracy can be more flexibly selected, for example, the sampling probability can be determined according to the specific cost proportion of each group of parameters.

[0006] In some possible implementation manners, according to the sampling probability distribution and the training iteration step arrangement manner, the sampled parameter group is frozen or stopped updating, including: determining a first iteration step according to the training iteration step arrangement manner, the first iteration step being an iteration step to be sampled; determining an mth parameter group to be sampled in the first iteration step according to the sampling probability distribution, m being less than or equal to M-1; freezing the mth parameter group in the first iteration step to the first parameter group, freezing the mth parameter group in the first iteration step to the first parameter group indicating that gradient calculation is not performed on the mth parameter group to the first parameter group, and parameter updating is not performed.

[0007] According to the sampling probability distribution, some parameter groups are determined to be frozen, gradient calculation is not performed, and parameter updating is not performed, so that the acceleration of neural network training can be realized. For the frozen parameter groups, in one period, the parameter groups of subsequent iteration steps do not need to use the parameters of the previously frozen parameter groups, so that the problem of momentum deviation can be avoided.

[0008] In some possible implementation manners, according to the sampling probability distribution and the training iteration step arrangement manner, the sampled parameter group is frozen or stopped updating, including: determining a first iteration step according to the training iteration step arrangement manner, the first iteration step being an iteration step to be sampled; determining an mth parameter group to be sampled in the first iteration step according to the sampling probability distribution, m being less than or equal to M-1; freezing the mth parameter group in the first iteration step to the first parameter group, freezing the mth parameter group in the first iteration step to the first parameter group indicating that gradient calculation is not performed on the mth parameter group to the first parameter group, and parameter updating is not performed.

[0009] According to the sampling probability distribution, some parameter groups are determined to be frozen, gradient calculation is performed, but parameter updating is not performed, so that the precision of neural network training can be improved. For the stopped updating parameter groups, gradient calculation is still performed, so that the parameters of the corresponding parameter groups of subsequent iteration steps can be kept updated, and the problem of momentum deviation can be avoided.

[0010] In some possible implementation manners, when the training iteration step arrangement manner is interval arrangement, the first iteration step is determined according to the training iteration step arrangement manner, including: determining a first interval; and determining one or more first iteration steps in the plurality of training iteration steps every first interval.

[0011] In some possible implementation manners, when the training iteration step arrangement manner is cycle arrangement, the first iteration step is determined according to the training iteration step arrangement manner, including: determining that the number of the first iteration steps is M-1; determining a first cycle according to the number of the first iteration steps and a first proportion, the first cycle including the first iteration steps and iteration steps of full-network training, the first proportion being a proportion of the first iteration steps in the first cycle, and the first iteration steps being the last M-1 iteration steps in the first cycle.

[0012] The training method of the neural network can determine the iteration steps to be sampled in the two manners, the periodic arrangement manner can effectively improve the speed of neural network training, and the interval arrangement manner can effectively improve the accuracy of neural network training.

[0013] In a second aspect, a data processing method is provided, which includes: obtaining to-be-processed data; processing the to-be-processed data according to a target neural network, the target neural network being obtained by training, and the training of the target neural network including: obtaining a to-be-trained neural network; grouping parameters of the to-be-trained neural network to obtain M groups of parameters, M being a positive integer greater than or equal to 1; obtaining a sampling probability distribution and a training iteration step arrangement manner, the sampling probability distribution being used to represent a probability of sampling each group of parameters in the M groups of parameters in each training iteration step, and the training iteration step arrangement manner including an interval arrangement and a periodic arrangement; freezing or stopping updating the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement manner; and training the to-be-trained neural network according to the frozen parameter groups or the stopped-updating parameter groups.

[0014] The data processing method provided in the present application uses the neural network trained by the neural network training method of the first aspect and any one of the implementation manners of the first aspect to process data, which can effectively improve the ability of neural network data processing.

[0015] In a third aspect, a neural network training apparatus is provided, which includes: an obtaining module, configured to obtain a to-be-trained neural network; a processing module, configured to group parameters of the to-be-trained neural network to obtain M groups of parameters, M being a positive integer greater than or equal to 1; the obtaining module is further configured to obtain a sampling probability distribution and a training iteration step arrangement manner, the sampling probability distribution being used to represent a probability of sampling each group of parameters in the M groups of parameters in each training iteration step, and the training iteration step arrangement manner including an interval arrangement and a periodic arrangement; the processing module is further configured to freeze or stop updating the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement manner; and train the to-be-trained neural network according to the frozen parameter groups or the stopped-updating parameter groups.

[0016] The embodiments of the present application further provide a neural network training apparatus, which can be used to implement the method in any one of the implementation manners of the first aspect.

[0017] In some possible implementation manners, in combination with the third aspect, the processing module freezes or stops updating the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement manner, including: determining a first iteration step according to the training iteration step arrangement manner, the first iteration step being a to-be-sampled iteration step; determining an m-th parameter group sampled in the first iteration step according to the sampling probability distribution, m being less than or equal to M-1; freezing the m-th parameter group in the first iteration step to the first parameter group, freezing the m-th parameter group in the first iteration step to the first parameter group indicating that gradient calculation is not performed on the m-th parameter group to the first parameter group, and parameter updating is not performed.

[0018] In some possible implementation manners, in combination with the third aspect, the processing module freezes or stops updating the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement manner, including: determining a first iteration step according to the training iteration step arrangement manner, the first iteration step being a to-be-sampled iteration step; determining an m-th parameter group sampled in the first iteration step according to the sampling probability distribution, m being less than or equal to M-1; stopping updating the m-th parameter group in the first iteration step to the first parameter group, stopping updating the m-th parameter group in the first iteration step to the first parameter group indicating that gradient calculation is performed on the m-th parameter group to the first parameter group, and parameter updating is not performed.

[0019] In some possible implementation manners, in combination with the third aspect, when the training iteration step arrangement manner is interval arrangement, the processing module determines the first iteration step sampled according to the training iteration step arrangement manner, including: determining a first interval; and determining one or more first iteration steps in the plurality of training iteration steps at every first interval.

[0020] In some possible implementation manners, in combination with the third aspect, when the training iteration step arrangement manner is periodic arrangement, the processing module determines the first iteration step sampled according to the training iteration step arrangement manner, including: determining that the number of the first iteration steps is M-1; determining a first period according to the number of the first iteration steps and a first proportion, the first period including the first iteration steps and iteration steps of full-network training, the first proportion being a proportion of the first iteration steps in the first period, and the first iteration steps being the last M-1 iteration steps in the first period.

[0021] In a fourth aspect, a data processing apparatus is provided, and the apparatus includes an obtaining module configured to obtain data to be processed, and a processing module configured to process the data to be processed according to a target neural network, the target neural network being obtained by training, and the training of the target neural network includes obtaining a neural network to be trained, grouping parameters of the neural network to be trained to obtain M groups of parameters, M being a positive integer greater than or equal to 1, obtaining a sampling probability distribution and a training iteration step arrangement, the sampling probability distribution being used to represent a probability of each group of parameters in the M groups of parameters being sampled in each training iteration step, and the training iteration step arrangement including interval arrangement and periodic arrangement, freezing or stopping updating of the sampled groups of parameters according to the sampling probability distribution and the training iteration step arrangement, and training the neural network to be trained according to the frozen groups of parameters or the stopped-updating groups of parameters.

[0022] In a fifth aspect, an electronic device is provided, and the electronic device includes a memory and a processor, the memory being configured to store program instructions, and when the program instructions are executed in the processor, the processor is configured to perform the method in any one of the implementation manners of the first aspect and the second aspect.

[0023] The processor in the fifth aspect described above can be a central processing unit (CPU) or a combination of a CPU and a neural network operation processor.

[0024] In a sixth aspect, a computer readable medium is provided, and the computer readable medium stores program codes for execution by a device, and the program codes include codes for performing the method in any one of the implementation manners of the first aspect and the second aspect.

[0025] In a seventh aspect, a computer program product including instructions is provided, and when the computer program product is run on a computer, the computer is caused to perform the method in any one of the implementation manners of the first aspect and the second aspect.

[0026] In an eighth aspect, a chip is provided, and the chip includes a processor and a data interface, the processor reads instructions stored on a memory through the data interface, and performs the method in any one of the implementation manners of the first aspect and the second aspect.

[0027] Optionally, as an implementation manner, the chip can further include a memory, and the memory stores instructions, and the processor is configured to execute the instructions stored on the memory, and when the instructions are executed, the processor is configured to perform the method in any one of the implementation manners of the first aspect and the second aspect.

[0028] The chip can be a field-programmable gate array (FPGA) or an application-specific integrated circuit (ASIC). BRIEF DESCRIPTION OF DRAWINGS

[0029] Figure 1 is a schematic structural diagram of a convolutional neural network according to an embodiment of the present application;

[0030] Figure 2 is a schematic block diagram of a system architecture to which a neural network training method according to an embodiment of the present application is applied;

[0031] Figure 3 is a schematic diagram of an interval arrangement of training iteration steps according to an embodiment of the present application;

[0032] Figure 4 is a schematic diagram of momentum offset according to an embodiment of the present application;

[0033] Figure 5 is a schematic diagram of a period arrangement of training iteration steps according to an embodiment of the present application;

[0034] Figure 6 is a schematic flowchart of a neural network training method according to an embodiment of the present application;

[0035] Figure 7 is a schematic block diagram of neural network parameter grouping according to an embodiment of the present application;

[0036] Figure 8 is a schematic block diagram of a neural network training method according to an embodiment of the present application;

[0037] Figure 9 is a schematic diagram of training iteration step sampling in a period arrangement according to an embodiment of the present application;

[0038] Figure 10 is a schematic block diagram of a static graph deep learning framework computation graph according to an embodiment of the present application;

[0039] Figure 11 is a schematic diagram of training iteration step sampling in an interval arrangement according to an embodiment of the present application;

[0040] Figure 12 is a schematic flowchart of a data processing method according to an embodiment of the present application;

[0041] Figure 13 is a schematic block diagram of a neural network training device according to an embodiment of the present application;

[0042] Figure 14is a schematic block diagram of a data processing apparatus of an embodiment of the present application;

[0043] Figure 15 is a hardware structure schematic diagram of a training apparatus of a neural network of an embodiment of the present application;

[0044] Figure 16 is a hardware structure schematic diagram of a data processing apparatus of an embodiment of the present application. DETAILED DESCRIPTION

[0045] The terms used in the following embodiments are only for the purpose of describing particular embodiments and are not intended to be limiting of the present application. As used in the specification and the appended claims, the singular forms "a," "an" and "the" are intended to include both singular and plural forms, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. The term "one or more" means one, two, three, four, five, six, seven, eight, nine, ten or more, as are appropriate, unless otherwise indicated by context.

[0046] Reference throughout this specification to "one embodiment" or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present application. Thus, the appearances of the phrases "in one embodiment" or "in some embodiments" or "in other embodiments" or "in still other embodiments" or similar phrases in various places throughout this specification are not necessarily all referring to the same embodiment, unless otherwise indicated by context. The term "comprising," "including," "containing," and "having" and their variations, as used herein, mean "including but not limited to," unless otherwise indicated by context.

[0047] In order to facilitate understanding of the technical solutions of the present application, first, the concepts involved in the present application are briefly introduced.

[0048] Deep learning: a machine learning technique based on deep neural network algorithm, its main feature is to use multiple nonlinear transformation to process and analyze data. Mainly applied to artificial intelligence field of perception, decision-making and other scenes, such as image recognition, speech recognition, natural language translation, computer game and so on.

[0049] Training: In this application embodiment, training specifically refers to the training of a neural network, which generally includes forward computation of the model output, calculation of loss based on the model output and labels, backpropagation to calculate gradients, and parameter updates. Using existing datasets and their corresponding labels, the model is optimized using the backpropagation algorithm and some parameter update method to minimize the loss value.

[0050] Freeze: In the backpropagation step of neural network training, the gradients of certain parameters are not calculated and these parameters are not updated, which is called back-freezing these parameters.

[0051] Stop updating: In the reverse step of the neural network training process, the gradient of the parameters is continued to be calculated, but the updating of these parameters is stopped.

[0052] Cost: In this embodiment of the application, the cost refers to the resources consumed during the training of the neural network, which can generally be estimated based on the computational load of the neural network.

[0053] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0054] Currently, acceleration of neural network training mainly focuses on hardware upgrades and algorithm optimization. On the hardware side, higher GPU performance comes at a higher cost; multi-GPU parallelism, multi-machine parallelism, and large-scale clusters are also common training acceleration methods. On the algorithm side, mixed-precision training can reduce the computational load of neural networks, effectively accelerating training in certain scenarios. The neural network training method in this application primarily involves algorithmic improvements, accelerating training and reducing actual overhead while maintaining unchanged hardware conditions.

[0055] The neural network training method of this application embodiment can be applied to objects that can be Figure 1 The convolutional neural network structure shown is illustrated. Figure 1 In this example, the Convolutional Neural Network (CNN) 100 may include an input layer 110, convolutional / pooling layers 120 (where pooling layers are optional), and a neural network layer 130. The input layer 110 acquires the data to be processed and passes it to the convolutional / pooling layers 120 and the subsequent neural network layer 130 for processing, thereby obtaining the processed data result. The following section... Figure 1 This section provides a detailed introduction to the internal layer structure of CNN 100.

[0056] Convolutional / pooling layers 120:

[0057] Convolutional layers:

[0058] likeFigure 1 The convolutional / pooling layer 120 shown may include layers as in Examples 121-126. For instance, in one implementation, layer 121 is a convolutional layer, layer 122 is a pooling layer, layer 123 is a convolutional layer, layer 124 is a pooling layer, layer 125 is a convolutional layer, and layer 126 is a pooling layer; in another implementation, layers 121 and 122 are convolutional layers, layer 123 is a pooling layer, layers 124 and 125 are convolutional layers, and layer 126 is a pooling layer. That is, the output of the convolutional layer can be used as the input to a subsequent pooling layer, or as the input to another convolutional layer to continue the convolution operation.

[0059] The following section will use convolutional layer 121 as an example to introduce the internal working principle of a convolutional layer.

[0060] Convolutional layer 121 can include many convolution operators, also known as kernels. In data processing, a convolution operator acts as a filter that extracts specific information from the input data matrix. A convolution operator can essentially be a weight matrix, which is usually predefined.

[0061] The weight values ​​in these weight matrices need to be obtained through extensive training in practical applications. The weight matrices formed by the weight values ​​obtained through training can be used to extract information from the input data, thereby enabling the convolutional neural network 100 to make correct predictions.

[0062] When a convolutional neural network 100 has multiple convolutional layers, the initial convolutional layers (e.g., 121) tend to extract more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by later convolutional layers (e.g., 126) become more and more complex, such as high-level semantic features. Features with higher semantic levels are more suitable for the problem to be solved.

[0063] Pooling layer:

[0064] Because it is often necessary to reduce the number of training parameters, pooling layers are often introduced periodically after convolutional layers, such as... Figure 1 In the example of 120, layers 121-126 can be a convolutional layer followed by a pooling layer, or multiple convolutional layers followed by one or more pooling layers. The sole purpose of pooling layers in data processing is to reduce the spatial size of the data.

[0065] Neural network layer 130:

[0066] After the processing of the convolutional / pooling layer 120, the convolutional neural network 100 is still not sufficient to output the required output information. As mentioned previously, the convolutional / pooling layer 120 only extracts features and reduces the parameters brought by the input data. However, in order to generate the final output information (the required class information or other related information), the convolutional neural network 100 needs to utilize the neural network layer 130 to generate an output of the number of required classes. Therefore, the neural network layer 130 can include multiple hidden layers (such as 131, 132, and 13n as shown) and an output layer 140, and the parameters contained in the multiple hidden layers can be pre-trained according to the relevant training data of a specific task type, which can include recognition, classification, and the like. Figure 1

[0067] The neural network can use a back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward propagation of the input signal will produce an error loss, and the initial neural network model parameters are updated by back propagating the error loss information, so as to make the error loss converge. The back propagation algorithm is a back propagation movement dominated by error loss, aiming to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0068] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140, which has a loss function similar to the classification cross-entropy, and is specifically used to calculate the prediction error. Once the forward propagation of the entire convolutional neural network 100 is completed, the back propagation will start to update the weight values and biases of the layers mentioned above, in order to reduce the loss of the convolutional neural network 100, and the error between the result output by the convolutional neural network 100 through the output layer and the ideal result. Figure 1 Figure 1

[0069] ​​​A neural network training method in the prior art freezes each layer of the network from front to back during the training process. When a layer is frozen, the layer is not trained until the end of the network training. For example, the parameters of a neural network are divided into six parts, and a total of 90 epochs are trained. The first part is epoch 1-46, the second part is epoch 47-53, the third part is 54-61, the fourth part is epoch 62-70, the fifth part is epoch 71-79, and the sixth part is epoch 80-90. The neural network training method uses certain calculation rules to train each group of parameters in the first part, that is, each parameter in the network is calculated and updated in the gradient calculation process. In the second part, the first group of parameters of the neural network is frozen, that is, only the forward calculation is involved, and the gradient is not calculated in the back propagation process, and the parameter is not updated. In the third part, the first two groups of parameters of the neural network are frozen. In the fourth part, the first three groups of parameters of the neural network are frozen. In the fifth part, the first four groups of parameters of the neural network are frozen. In the sixth part, the first five groups of parameters of the neural network are frozen. The learning rate of each group of parameters is scaled according to the proportion of the training time length of the group of parameters to the total training time length. In the training of the above neural network, the learning rate of the last group of parameters to the first group of parameters is increased. This method is only suitable for the training of part of the neural network, freezes a group of parameters every several epochs, coarsely controls the parameter freezing in the training process, has a large precision loss, and needs to scale the learning rate according to the proportion of the freezing time length of each group of parameters, which increases the additional workload.

[0070] Another existing neural network training method calculates a certain standard according to the state of the network gradient during the training of the neural network, and then determines whether freezing is needed according to the standard. This method introduces additional calculation, and in some scenarios, the training speed does not increase but decreases. The existing neural network training method also adds a random factor during the training of the network to randomly skip the forward and reverse calculations of part of the residual branches, to achieve the purpose of accelerating the training. However, this method can only be used on some specific network structures, and has great limitations and a narrow range of use.

[0071] The neural network training method of the embodiments of the present application automatically groups the parameters of the user network, introduces a sampling probability distribution to finely control the freezing during the training process, finely arranges the training steps at intervals or periodically, corrects the momentum offset, so that each group of parameters of the network has a certain probability of updating during the entire training process, and there is no situation that a group of parameters is trained in the early stage and completely frozen without updating in the later stage, thereby ensuring the training accuracy.

[0072] The training method of the neural network of the embodiments of the present application is applicable to deep learning frameworks such as mindspore, tensorflow, and pytorch, and to scenarios where various computer vision tasks need to be accelerated in neural network training with hardware platforms such as Ascend chips and GPUs, where the computer vision tasks can be target recognition, target detection, semantic segmentation, etc.

[0073] Figure 2 A schematic block diagram of a system architecture to which the training method of the neural network of the embodiments of the present application is applied is shown, which can implement the acceleration of the training of the neural network as shown. Figure 1 Figure 2 As shown, the system architecture includes a probability distribution module and a training iteration step arrangement module, which are introduced below respectively.

[0074] The probability distribution module is used to introduce a sampling probability distribution for fine-grained freeze control of the training process, where the probability distribution includes a sampling probability and a freeze probability. The sampling probability controls the probability of each group of parameters being sampled in the neural network, for example, if a certain group of parameters is sampled at a certain training iteration step, then the first group of parameters from the group of parameters will be reversed and frozen. Once the sampling probability distribution is determined, the freeze probability distribution of the parameters is also determined.

[0075] The sampling probability distribution formula is:

[0076] p s (i)=f(i)

[0077] p(n-1)=0,

[0078] The freeze probability distribution formula is:

[0079] p freeze (0)=1-p0

[0080] Where p0 represents the probability of not freezing any parameters in the neural network training process, i.e., the probability of whole-network training; n represents the number of parameter groups of the network, and i represents the index of the parameter group, ranging from 0 to n-1; the formula f(x) can be a continuous function or a discrete function, for example, the sampling probability of each group of parameters is determined according to the actual test overhead ratio, the probability of the i-th group of parameters being frozen is the sum of the sampling probabilities of the i-th group to the n-1-th group of parameters, and the sum of the sampling probabilities of all groups of parameters is 1. The greater the area included in the freeze probability distribution curve, the lower the training overhead.

[0081] ​The training iteration step arrangement module is configured to arrange the training iteration steps in a fine-grained manner according to the probability distribution formula selected by the probability distribution module, so as to determine which groups of parameters are frozen at each iteration in the training process. Figure 3 An arrangement diagram of training iteration steps with a whole-network training probability p0 of 0.5 is shown. Since p0 is 0.5, whole-network training is performed every other step, that is, at iteration step 0, iteration step 2, and iteration step 4, while at iteration step 1, iteration step 3, and iteration step 5, the parameters of the third, fifth, and first groups are sampled according to the sampling probability distribution function, and thus the parameters of the previous three, five, and one groups are frozen. This arrangement is referred to as interval arrangement.

[0082] Figure 3 In the figure, the active layers represent that both forward calculation and backward calculation are performed, and the frozen layers represent that only forward calculation is performed and backward calculation is frozen. One active layer or one frozen layer represents one group of parameters. Figure 3 The arrangement of equal-interval whole-network training in the figure has the case of unfreezing the frozen layers at each iteration step, which causes momentum deviation. For example, in the figure, the parameters of the first three groups at iteration step 2 are not updated because the gradient is missing at iteration step 1, and thus the momentum used at iteration step 2 is still based on the momentum at iteration step 0. Figure 4

[0083] Therefore, the embodiment of the present application designs an arrangement scheme of periodic step-by-step freezing (periodic mode) as shown in the figure. Figure 5 In the figure, the index 0 represents whole-network training, and the indexes 1, 2, 3, 4, and 5 respectively represent freezing the parameters of the previous one, two, three, four, and five groups during backward calculation. The last group of parameters of the network does not participate in freezing. Figure 5 The iteration step arrangement in the figure and the exponential curve are both periodic, and the sampling probability is linearly decreasing, which means that the probability of sampling the earlier groups of parameters is higher. Thus, the following can be obtained. Figure 5 The left side of the figure is an arrangement diagram of training iteration steps.

[0084] For any sampling probability distribution function, according to the probability of sampling at each step and the probability of whole-network training, the above-mentioned interval arrangement and periodic arrangement of training iteration steps can be achieved. For interval arrangement, one or more iteration steps are selected to be frozen for sampling every n iteration steps in 10 consecutive training iterations. n can be a preset value or determined according to the whole-network training probability. For example, Figure 3 In the figure, when p0 is 0.5, n is 1, and thus frozen sampling is performed every 1 iteration step. For periodic arrangement, the iteration steps of whole-network training can be adjusted according to the size of p0 and the ratio of p0 to 1-p0. For example, Figure 5 ​When p0 is 0.5, it means that the number of iteration steps of the whole network training and the number of iteration steps of the frozen sampling are in a ratio of 1:1, iteration steps 0 to 14 are whole network training, and iteration steps 15 to 29 are frozen sampling. Then, according to the sampling probability curve, the probability of the earlier parameter group being sampled is greater.

[0085] Figure 6 A schematic flowchart of the method for training the neural network is shown in FIG. 1, which includes steps 601 to 604, which are described below. Figure 6

[0086] S601, obtaining a neural network to be trained.

[0087] The method for training the neural network can be applied to tasks such as target detection, image segmentation, natural language processing, and speech recognition. The neural network to be trained can be a convolutional neural network as shown in FIG. 2, which can be a neural network in the ResNet, MobileNet, or other series, and the present embodiment does not make specific limitations. Figure 1

[0088] S602, grouping parameters of the neural network to be trained to obtain M groups of parameters, M being a positive integer greater than or equal to 1.

[0089] The neural network parameters of the present embodiment include operators of each layer of the neural network, and the training of the neural network is the determination of the weights of the operators in each layer of the neural network. The method for training the neural network of the present embodiment can automatically group the parameters of the neural network to be trained, and the grouping standard can be set in advance. The parameter grouping follows the principle of input to output, as shown in FIG. 3, the group of parameters closest to the input is determined as the first group of parameters, i.e., group 0 in FIG. 4, and so on, and the group of parameters farthest from the input is determined as the last group of parameters, i.e., group 5 in FIG. 5. When grouping parameters, a single operator can be grouped as a group of parameters, or a plurality of consecutive operators can be grouped as a group of parameters. Generally, a convolutional operator and a batch normalization (BN) operator after it are grouped as a group of parameters. For example, for the neural network in the ResNet, MobileNet, or other series in S601, each convolutional operator and BN operator can be pre-set as a group of parameters. After obtaining the neural network in the ResNet, MobileNet, or other series, the method for training the neural network can automatically group the parameters of these neural networks according to the pre-set parameters. Figure 7 Figure 7 Figure 7 Figure 8 A schematic block diagram of the method for training the neural network of the present embodiment is shown in FIG. 6, which includes steps 601 to 604, which are described below. Figure 8 ​​​​​As shown, in the parameter automatic grouping step, the input data can be automatically grouped into group 0 to group 5.

[0090] S603, obtain a sampling probability distribution and a training iteration step arrangement mode, the sampling probability distribution is used to represent a probability that each group of parameters in the M groups of parameters is sampled in each training iteration step, and the training iteration step arrangement mode includes interval arrangement and periodic arrangement.

[0091] Specifically, first, the training iteration step arrangement mode is determined. In the neural network training method of the embodiment of the present application, partial iteration steps are subjected to whole network training, and the whole network training means that gradient calculation and parameter updating are performed on each group of parameters in the iteration steps. In the neural network training, training data needs to be used to minimize the loss function, so as to determine the value of the neural network parameter. To minimize the loss function, the extreme value of the loss function needs to be obtained. The direction of the gradient of the vector field is the direction in which the function value rises fastest. That is, the opposite direction is the direction in which the function value falls fastest. Therefore, by calculating the gradient of the loss function (i.e., calculating the partial derivative of all parameters) and updating the parameters in the opposite direction, the loss function can quickly reach a minimum value after iteration. Partial iteration steps are sampled, so it is necessary to determine which iteration steps are sampled. The training iteration step arrangement mode includes interval arrangement and periodic arrangement. The interval arrangement means that a plurality of training iteration steps are determined, and one or more sampled training iteration steps are determined at a certain interval. For example, the training iteration steps can be determined according to the following manner: determine a whole network training probability p0, multiply the number of the plurality of iteration steps to be trained by p0, and the number of iteration steps subjected to the whole network training can be obtained. Then, the iteration steps subjected to the whole network training are evenly distributed in the plurality of iteration steps to be trained. The value of p0 is artificially preset and ranges from 0 to 1. Table 1 shows some examples of determining the training iteration step arrangement mode according to the whole network training probability:

[0092] Table 1

[0093] [P0 = 0] [P0 = 0.1] [P0 = 0.2] [P0 = 0.3] [P0 = 0.4] [P0 = 0.5] [P0 = 0.6] [P0 = 0.7] [P0 = 0.8] [P0 = 0.9] [P0 = 1] Step 0 ◆ ◆ ◆ ◆ ◆ ◆ ◇ ◇ ◇ ◇ ◇ Step 1 ◆ ◆ ◆ ◆ ◇ ◇ ◆ ◇ ◇ ◇ ◇ Step 2 ◆ ◆ ◆ ◇ ◆ ◆ ◇ ◆ ◇ ◇ ◇ Step 3 ◆ ◆ ◆ ◆ ◇ ◇ ◆ ◇ ◇ ◇ ◇ Step 4 ◆ ◆ ◇ ◆ ◆ ◆ ◇ ◇ ◆ ◇ ◇ Step 5 ◆ ◆ ◆ ◇ ◇ ◇ ◆ ◆ ◇ ◇ ◇ Step 6 ◆ ◆ ◆ ◆ ◆ ◆ ◇ ◇ ◇ ◇ ◇ Step 7 ◆ ◆ ◆ ◆ ◇ ◇ ◆ ◇ ◇ ◇ ◇ Step 8 ◆ ◆ ◆ ◇ ◆ ◆ ◇ ◆ ◇ ◇ ◇ Step 9 ◆ ◇ ◇ ◆ ◆ ◇ ◇ ◇ ◆ ◆ ◇

[0094] In Table 1, 10 iteration steps are taken as an example, i.e., step0 to step9, and represents that the iteration step will be sampled, represents that the iteration step will not be sampled, and the iteration step is trained in the whole network. For example, when p0=0, the number of iteration steps trained in the whole network is 0, and step0 to step9 will be sampled; for example, when p0=0.3, the number of iteration steps trained in the whole network is 3, and the 3 iteration steps trained in the whole network are evenly distributed in the 10 iteration steps, and step2, step5 and step8 are the iteration steps trained in the whole network, and step0, step1, step3, step4, step6, step7 and step9 are the iteration steps to be sampled. It should be understood that the arrangement of the training iteration steps determined according to the whole network training probability in Table 1 is only an example of the interval arrangement in the embodiments of the present application, and does not constitute a limitation on the embodiments of the present application.

[0095] The periodic arrangement means that a plurality of training iteration steps are taken as one period, and first, the number of iteration steps to be sampled is determined as M-1; the period is determined according to the number of iteration steps to be sampled and a certain ratio, wherein the period includes the iteration steps to be sampled and the iteration steps trained in the whole network, the certain ratio is the ratio of the iteration steps to be sampled in the period, and the iteration steps to be sampled are the last M-1 iteration steps in the period. For example, the whole network training probability p0 can be determined, and the whole network training probability p0 can be a value artificially preset, and the certain ratio is 1-p0. Corresponding to Figure 8 In the interval arrangement, iteration steps 0, 2 and 4 are trained in the whole network, and iteration steps 1, 3 and 5 are sampled.

[0096] After the iteration steps to be sampled are determined according to the arrangement of the training iteration steps, for each iteration step to be sampled, the parameter group to be sampled in the iteration step is further determined. The training method of the neural network in the embodiments of the present application determines the probability of each parameter group in M parameter groups being sampled in each training iteration step according to the sampling probability distribution, i.e., the mth parameter group to be sampled in a certain iteration step is determined according to the sampling probability distribution, and m is less than or equal to M-1. After the mth parameter group to be sampled in a certain iteration step is determined, the same processing is performed on the mth parameter group to the first parameter group.

[0097] In S604, the parameter group to be sampled is frozen or stopped according to the sampling probability distribution and the arrangement of the training iteration steps.

[0098] In S603, the processing includes freezing and stopping, wherein freezing means that gradient calculation is not performed on the mth parameter group to the first parameter group, and parameter updating is not performed; and stopping means that gradient calculation is performed on the mth parameter group to the first parameter group, and parameter updating is not performed. The corresponding sampling probability distribution formula and freezing / stopping probability distribution formula can be referred to the above description of the interval arrangement. Figure 2The description. For example. Figure 8 In the above, when p0 is 0.5, multiple sampling probability distribution curves can be obtained according to the sampling probability distribution formula. The horizontal axis represents the parameter set, and the vertical axis represents the probability of being sampled. Selecting one of the curves to sample iterations 1, 3, and 5 yields the following results: Figure 8 The training iteration step sampling distribution diagram is shown. In iteration step 1, the parameter group sampled is group 2, so group 0 to group 1 are frozen; in iteration step 3, the parameter group sampled is group 0, so group 0 is frozen; in iteration step 5, the parameter group sampled is group 4, so group 0 to group 4 are frozen.

[0099] S605 trains the neural network to be trained based on the frozen parameter set or the stopped parameter set.

[0100] Based on the above sampling probability distribution and the training iteration step arrangement, the training iteration step sampling distribution can be obtained. Gradient calculation and parameter updates are performed on parameter groups that are not frozen or stopped; gradient calculation and parameter updates are not performed on frozen parameter groups; and gradient calculation is performed on stopped parameter groups, but parameter updates are not performed. The neural network to be trained is then iteratively trained in this way.

[0101] It should be understood that the neural network training method of this application embodiment can be used for training corresponding neural networks, which can be as follows: Figure 1 The neural network shown is illustrated. The training method for the neural network in this application embodiment can be applied to visual tasks such as object detection and image segmentation, as well as non-visual tasks such as natural language processing and speech recognition.

[0102] Since an epoch represents a complete training of a neural network model using all the data in the training set, while a training iteration step represents updating the parameters of the neural network model once, and in some cases an epoch may include ten thousand training iteration steps, controlling the training of the neural network in the dimension of training iteration steps has higher precision than in the dimension of epochs.

[0103] The neural network training method of this application processes the parameter set of the neural network at the iteration step dimension, realizing fine-grained control over the acceleration process and improving training accuracy while accelerating training. By sampling and processing the parameter set through the arrangement of training iteration steps and the sampling probability distribution, a more flexible choice can be made between training overhead and training accuracy. For example, the corresponding sampling probability can be determined according to the specific overhead ratio of each parameter set. The momentum shift problem is corrected. For frozen parameter sets, within one cycle, the parameter sets of subsequent iteration steps do not need to use the parameters of the previously frozen parameter sets. For parameter sets that are not updated, gradient calculation is still performed, so that the parameters of the parameter sets of subsequent iteration steps can be kept updated.

[0104] The training method of the neural network of the embodiment of the application is described in detail below in combination with specific examples.

[0105] The training method of the neural network of the embodiment of the application can be applied to precision verification of target recognition tasks on multiple networks respectively according to the ImageNet dataset. In the scene of using a static graph deep learning framework, a calculation graph can construct multiple reverse paths to cooperate with the training method of the neural network of the embodiment of the application for training acceleration, such as tensorflow, mindspore and other static graph deep learning frameworks; in the scene of using a dynamic graph deep learning framework, reverse truncation can be performed in each reverse process, such as pytorch and other dynamic graph deep learning frameworks.

[0106] Step one: input the neural network of ResNet50, ResNet18, MobileNetV2, etc.

[0107] Step two: automatically group parameters, and group each convolution operator and batch normalization (BN) operator into a group.

[0108] Step three: select the uniform sampling probability distribution with the most reduced overhead, which indicates that the probability of each group of parameters being sampled is the same; and select a periodic arrangement mode. Figure 9 A training iteration step sampling schematic diagram using the periodic arrangement mode of the embodiment of the application is shown. In one period, iteration step 0 to iteration step 4 perform whole network training, iteration step 5 freezes the first group of parameters, iteration step 6 freezes the first two groups of parameters, iteration step 7 freezes the first three groups of parameters, iteration step 8 freezes the first four groups of parameters, and iteration step 9 freezes the first five groups of parameters.

[0109] Step four: use the tensorflow deep learning framework, and the calculation graph constructs multiple paths to start iteration training. The calculation graph is shown in Figure 10 .

[0110] The training method of the neural network described above is used to train and evaluate different networks on the ImageNet dataset, and the precision test results obtained are shown in the following table:

[0111] Table 2

[0112] Baseline (%) Accuracy (%) ResNet50 76.81 76.92(+0.11) ResNet34 74.43 74.38(-0.05) ResNet18 70.7 70.98(+0.28) ResNet101 78.84 78.85(+0.01) MobileNetV2 71.96 72.04(+0.08) Vgg16_bn 73.82 73.55(-0.27) ResNeXt50 77.68 77.64(-0.04) DenseNet121 75.84 75.82(-0.02) AlexNet 57.02 56.98(-0.04) InceptionV3 76.20 76.15(-0.05)

[0113] As shown in Table 2, the precision obtained by using the training method of the neural network described above to train different networks on the ImageNet dataset is basically the same as the baseline, but the speed of neural network training is improved by 20%.

[0114] The training method of the neural network of the embodiments of the present application can have a certain regularization effect in a recognition task, and can improve the model accuracy to a certain extent while reducing the overhead by a small margin. Another process of network training using the training method of the neural network of the embodiments of the present application is introduced below.

[0115] Step one: input ResNet50, ResNet18, MobileNetV2, etc. neural network;

[0116] Step two: automatically group parameters, and group each convolution operator and batch normalization (BN) operator into a group;

[0117] Step three: select a linearly decreasing sampling probability distribution, and the probability of sampling the earlier parameter group is higher; the training iteration step arrangement mode selects interval arrangement. The sampled parameters still calculate the gradient and momentum, so that the momentum of the corresponding parameter group in the next iteration step can be avoided, but the parameter is not updated. Due to the randomness of stochastic gradient descent (SGD), a certain noise is introduced, and in the normal training process of the network, the useful signal brought by the gradient transmitted from the loss to the front layer is already very weak, resulting in a high signal-to-noise ratio of the front layer of the network. The interval arrangement mode randomly stops the gradient update of the front layer, which is beneficial to reduce the negative effect of the high signal-to-noise ratio, and plays a certain optimization role, thereby improving the accuracy of the trained neural network. Figure 11 A training iteration step sampling schematic diagram using the interval arrangement mode of the embodiments of the present application is shown.

[0118] The above neural network training method is used to train and test different networks on the ImageNet dataset, and the accuracy test results are as shown in the following table:

[0119] Table 3

[0120]

[0121]

[0122] As can be seen from Table 3, the accuracy of the different networks obtained by using the above neural network training method to train the ImageNet dataset has been improved compared with the baseline.

[0123] Unlike existing regularization methods, the training method of the neural network of the embodiments of the present application can reduce the overhead by a small margin while improving the accuracy through a certain sampling probability distribution and parameter stop without changing the user network structure.

[0124] Figure 12A schematic flowchart illustrating an embodiment of a data processing method is shown, including steps 1201 to 1202.

[0125] S1201, Obtain the data to be processed.

[0126] S1202, the target neural network processes the data to be processed. The target neural network is obtained through training. The training of the target neural network includes: obtaining the neural network to be trained; grouping the parameters of the neural network to be trained to obtain M groups of parameters, where M is a positive integer greater than or equal to 1; obtaining the sampling probability distribution and the training iteration step arrangement, where the sampling probability distribution is used to characterize the probability of each parameter in the M groups of parameters being sampled in each training iteration step, and the training iteration step arrangement includes interval arrangement and periodic arrangement; freezing or stopping the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement; and training the neural network to be trained according to the frozen or stopped parameter groups.

[0127] Figure 12 The neural network used in the data processing is based on Figure 6 The neural network is trained using the training method described above. Figure 6 For the sake of brevity, the embodiments of this application will not be described in detail here.

[0128] The training method of the neural network according to the embodiments of this application has been described in detail above. The following section will combine... Figure 13 to Figure 16 The relevant apparatus of the embodiments of this application will be described.

[0129] Figure 13 A schematic block diagram of a neural network training apparatus according to an embodiment of this application is shown, including a storage module 1310, an acquisition module 1320, and a processing module 1330, which will be described below.

[0130] Storage module 1310 is used to store programs.

[0131] The acquisition module 1320 is used to acquire the neural network to be trained.

[0132] The processing module 1330 is used to group the parameters of the neural network to be trained to obtain M groups of parameters, where M is a positive integer greater than or equal to 1.

[0133] The acquisition module 1320 is also used to acquire the sampling probability distribution and the training iteration step arrangement. The sampling probability distribution is used to characterize the probability that each parameter in the M groups of parameters is sampled in each training iteration step. The training iteration step arrangement includes interval arrangement and periodic arrangement.

[0134] The processing module 1330 is further configured to freeze or stop updating the sampled parameter set according to the sampling probability distribution and the training iteration step arrangement manner, and train the neural network to be trained according to the frozen parameter set or the stopped-updating parameter set.

[0135] Optionally, the processing module 1330 freezes or stops updating the sampled parameter set according to the sampling probability distribution and the training iteration step arrangement manner, and specifically configured to: determine a first iteration step according to the training iteration step arrangement manner; determine the mth parameter set sampled in the first iteration step according to the sampling probability distribution, where m is less than or equal to M-1; freeze the mth parameter set to the first parameter set in the first iteration step, which means that gradient calculation is not performed on the mth parameter set to the first parameter set, and parameter updating is not performed.

[0136] Optionally, the processing module 1330 freezes or stops updating the sampled parameter set according to the sampling probability distribution and the training iteration step arrangement manner, and specifically configured to: determine a first iteration step according to the training iteration step arrangement manner; determine the mth parameter set sampled in the first iteration step according to the sampling probability distribution, where m is less than or equal to M-1; freeze the mth parameter set to the first parameter set in the first iteration step, which means that gradient calculation is not performed on the mth parameter set to the first parameter set, and parameter updating is not performed.

[0137] Optionally, when the training iteration step arrangement manner is interval arrangement, the processing module 1130 determines the first iteration step according to the training iteration step arrangement manner, and specifically configured to: determine a first interval; and determine one or more first iteration steps in the plurality of training iteration steps at every first interval. Optionally, when the training iteration step arrangement manner is periodic arrangement, the processing module 1130 determines the first iteration step according to the training iteration step arrangement manner, and specifically configured to: determine that the number of first iteration steps is M-1; determine a first period according to the number of first iteration steps and a first proportion, the first period includes the first iteration steps and the iteration steps of full-network training, and the first proportion is the proportion of the first iteration steps in the first period, and the first iteration steps are the last M-1 iteration steps in the first period.

[0138] It should be understood that the training device 1300 of the neural network of the embodiments of the present application can be used to implement each step in the method Figure 6 , and the specific implementation can refer to the description of the method Figure 6 above. For the sake of brevity, the embodiments of the present application will not be described here.

[0139] Figure 14 A schematic block diagram of a data processing device according to an embodiment of the present application is shown, which includes a storage module 1410, an acquisition module 1420, and a processing module 1430, which will be introduced respectively as follows.

[0140] The storage module 1410 is configured to store programs.

[0141] The acquisition module 1420 is configured to acquire to-be-processed data.

[0142] The processing module 1430 is configured to process the to-be-processed data according to a target neural network, the target neural network being obtained by training, and the training of the target neural network including: acquiring a neural network to be trained; grouping parameters of the neural network to be trained to obtain M groups of parameters, M being a positive integer greater than or equal to 1; acquiring a sampling probability distribution and a training iteration step arrangement, the sampling probability distribution being used to represent a probability of each group of parameters in the M groups of parameters being sampled in each training iteration step, and the training iteration step arrangement including interval arrangement and periodic arrangement; freezing or stopping updating of the sampled groups of parameters according to the sampling probability distribution and the training iteration step arrangement; and training the neural network to be trained according to the frozen groups of parameters or the stopped-updating groups of parameters.

[0143] It should be understood that the neural network training apparatus 1400 of the embodiments of the present application can be used to implement each step in the method of Figure 12 , and the specific implementation can refer to the description of the method of Figure 12 above. For brevity, the embodiments of the present application will not be described here.

[0144] Figure 15 is a hardware structure schematic diagram of the neural network training apparatus 1500 of the embodiments of the present application, as shown in Figure 15 , including a memory 1501, a processor 1502, a communication interface 1503 and a bus 1504. Wherein, the memory 1501, the processor 1502 and the communication interface 1503 are in communication connection with each other through the bus 1504.

[0145] The memory 1501 can be a ROM, a static storage device and a RAM. The memory 1501 can store programs, and when the programs stored in the memory 1501 are executed by the processor 1502, the processor 1502 and the communication interface 1503 are used to execute each step of the neural network training method of the embodiments of the present application.

[0146] The processor 1502 can adopt a general-purpose CPU, a microprocessor, an ASIC, a GPU or one or more integrated circuits, and is used to execute related programs to implement the functions required to be executed by the units in the neural network training apparatus of the embodiments of the present application, or execute the neural network training method of the method embodiments of the present application.

[0147] The processor 1502 can also be an integrated circuit chip with signal processing capability, for example, can be Figure 4The chip is shown. In the implementation process, each step of the neural network training method of the embodiment of the application can be completed by the integrated logic circuit of the hardware in the processor 1502 or the instructions in the form of software.

[0148] The above-mentioned processor 1502 can also be a general processor, a DSP, an ASIC, an FPGA or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The disclosed methods, steps and logic block diagrams in the embodiments of the application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the application can be directly embodied as hardware decoding processor execution or executed by hardware and software module combination in the decoding processor. The software module can be located in the random storage, flash memory, read-only memory, programmable read-only memory or electrically erasable programmable memory, register and other mature storage media in the art. The storage medium is located in the memory 1501, and the processor 1502 reads the information in the memory 1501, and combines the hardware to complete the function required by the unit included in the neural network training device of the embodiment of the application, or executes the neural network training method of the method embodiment of the application.

[0149] The communication interface 1503 uses a transceiver such as but not limited to a transceiver to realize the communication between the device 1500 and other devices or communication networks. For example, the neural network to be trained can be obtained through the communication interface 1503.

[0150] The bus 1504 can include a path for transmitting information between various components (e.g., memory 1501, processor 1502, communication interface 1503) of the device 1500.

[0151] Figure 16 The hardware structure schematic diagram of the data processing device 1600 of the embodiment of the application is shown, including memory 1601, processor 1602, communication interface 1603 and bus 1604. Among them, the memory 1601, the processor 1602, the communication interface 1603 realize the communication connection between each other through the bus 1604.

[0152] The memory 1601 can be a ROM, a static storage device and a RAM. The memory 1601 can store programs, and when the programs stored in the memory 1601 are executed by the processor 1602, the processor 1602 and the communication interface 1603 are used to execute each step of the data processing method of the embodiment of the application.

[0153] The processor 1602 can be a general purpose, CPU, microprocessor, ASIC, GPU, or one or more integrated circuits executing instructions to perform the functions of the units in the data processing apparatus of the embodiments of the present application, or to perform the data processing methods of the method embodiments of the present application.

[0154] The processor 1602 can also be an integrated circuit chip that has a processing capability of signals. In the implementation process, each step of the data processing method of the embodiments of the present application can be completed by the integrated logic circuit of hardware in the processor 1602 or the instructions in the form of software.

[0155] The processor 1602 described above can also be a general purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general purpose processor can be a microprocessor or the processor can be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, or other mature storage medium in the art. The storage medium is located in the storage 1601, and the processor 1602 reads the information in the storage 1601, and combines the hardware to complete the functions required to be executed by the units included in the data processing apparatus of the embodiments of the present application, or to execute the data processing method of the method embodiments of the present application.

[0156] The communication interface 1603 uses a transceiver device such as but not limited to a transceiver to realize the communication between the device 1600 and other devices or communication networks. For example, the data to be processed can be obtained through the communication interface 1603.

[0157] The bus 1604 can include a path for transmitting information between the various components (e.g., the storage 1601, the processor 1602, the communication interface 1603) of the device 1600.

[0158] It should be noted that although the above 1500 and 1600 only show the memory, the processor, the communication interface, in the specific implementation process, those skilled in the art should understand that the device 1500 and 1600 can also include other devices necessary for normal operation. At the same time, according to the specific needs, those skilled in the art should understand that the device 1500 and 1600 can also include hardware devices for realizing other additional functions. In addition, those skilled in the art should understand that the device 1500 and 1600 can also include only the devices necessary for the embodiments of the present application, and do not have to include Figure 15and Figure 16 all of the devices shown in FIG. 6.

[0159] It should be appreciated that the processor in the embodiments of the present application can be a central processing unit (CPU). The processor can also be other general purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic components, discrete hardware components, etc. The general purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0160] It should also be understood that the memory in the embodiments of the present application can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. The nonvolatile memory can be a read-only memory (ROM), programmable ROM (PROM), erasable PROM (EPROM), electrically EPROM (EEPROM) or flash memory. The volatile memory can be a random access memory (RAM) used as an external cache. By way of example, and not limitation, many forms of random access memory (RAM) are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM).

[0161] The embodiments of the present application further provide a computer program product, which, when executed by the processors 1502 and 1602, implements the method of any of the method embodiments of the present application. The computer program product can be stored in the memories 1501 and 1601, and the program is finally converted into an executable target file that can be executed by the processors 1502 and 1602 through processes such as preprocessing, compiling, assembling, and linking.

[0162] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program, when executed by a computer, implements the method of any of the method embodiments of the present application. The computer program can be a high-level language program or an executable target program. The computer readable storage medium is, for example, the memories 1501 and 1601.

[0163] The embodiments of the present application further provide a chip. The chip includes a processor and a data interface. The processor reads instructions stored on a memory through the data interface and executes the method of any of the method embodiments of the present application.

[0164] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any combination thereof. When implemented by software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the above-described processes or functions are generated, in whole or in part, according to the embodiments of the present application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer readable storage medium or transferred from one computer readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server, or data center to another by wired (for example, infrared, wireless, microwave, etc.) or wireless means. The computer readable storage medium can be any available medium accessible by a computer or a data storage device such as a server, data center, etc. containing a set of one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium (for example, a DVD), or a semiconductor medium. The semiconductor medium can be a solid state disk.

[0165] It should be understood that the term "and / or" in this document is merely used to describe associated relationship, and it can mean three relationships, for example, A and / or B can mean that A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. In addition, the character " / " in this document generally means that the associated objects before and after the " / " are in an "or" relationship, but can also mean an "and / or" relationship, which can be understood according to the context before and after.

[0166] In this application, "at least one" means one or more, and "multiple" means two or more. "At least one of the following" or the like means any combination of the items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean a, b, c, a-b, a-c, b-c, or a-b-c, where a, b, and c can be single or multiple.

[0167] It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0168] Those of ordinary skill in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0169] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0170] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are only schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed objects can be indirect coupling or communication connection through some interface, device or unit, and can be electrical, mechanical or other forms.

[0171] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, i.e. may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0172] In addition, the functional units in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit.

[0173] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of software products. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0174] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of training a neural network, the method comprising: The method comprises: obtaining a neural network to be trained, the neural network to be trained being used for performing at least one of object detection, image segmentation, natural language processing or speech recognition; grouping parameters of the neural network to be trained to obtain M groups of parameters, M being a positive integer greater than or equal to 1; obtaining a sampling probability distribution and a training iteration step arrangement, the sampling probability distribution being used for representing a probability of each group of parameters in the M groups of parameters being sampled in each training iteration step, and the training iteration step arrangement comprising interval arrangement and period arrangement; freezing or stopping updating of the sampled group of parameters according to the sampling probability distribution and the training iteration step arrangement; training the neural network to be trained according to the frozen group of parameters or the stopped-updating group of parameters.

2. The method of claim 1, wherein, The freezing or stopping updating of the sampled group of parameters according to the sampling probability distribution and the training iteration step arrangement comprises: determining a first iteration step to be sampled according to the training iteration step arrangement; determining an mth group of parameters to be sampled in the first iteration step according to the sampling probability distribution, m being less than or equal to M-1; freezing the mth group of parameters in the first iteration step to a first group of parameters, the freezing of the mth group of parameters in the first iteration step to the first group of parameters indicating that gradient calculation is not performed on the mth group of parameters to the first group of parameters, and parameter updating is not performed.

3. The method of claim 1, wherein, The freezing or stopping updating of the sampled group of parameters according to the sampling probability distribution and the training iteration step arrangement comprises: determining a first iteration step to be sampled according to the training iteration step arrangement; determining an mth group of parameters to be sampled in the first iteration step according to the sampling probability distribution, m being less than or equal to M-1; stopping updating of the mth group of parameters in the first iteration step to a first group of parameters, the stopping updating of the mth group of parameters in the first iteration step to the first group of parameters indicating that gradient calculation is performed on the mth group of parameters to the first group of parameters, and parameter updating is not performed.

4. The method according to claim 2 or 3, characterized in that, When the training iteration step arrangement is interval arrangement, the determining of the first iteration step according to the training iteration step arrangement comprises: determining a first interval; in a plurality of training iteration steps, determining one or more first iteration steps every first interval.

5. The method according to claim 2 or 3, characterized in that, When the training iteration step arrangement is period arrangement, the determining of the first iteration step according to the training iteration step arrangement comprises: determining a number of first iteration steps to be M-1; determining a first period according to the number of first iteration steps and a first proportion, the first period comprising the first iteration steps and iteration steps of full-network training, the first proportion being a proportion of the first iteration steps in the first period, and the first iteration steps being last M-1 iteration steps in the first period.

6. A method of data processing, characterized by, The method comprises: obtaining to-be-processed data; According to the target neural network, the target neural network is used to perform at least one of target detection, image segmentation, natural language processing or speech recognition, the target neural network is obtained by training, and the training of the target neural network comprises: obtaining a neural network to be trained; grouping parameters of the neural network to be trained to obtain M groups of parameters, M being a positive integer greater than or equal to 1; obtaining a sampling probability distribution and a training iteration step arrangement mode, the sampling probability distribution being used to represent the probability of each group of parameters in the M groups of parameters being sampled in each training iteration step, and the training iteration step arrangement mode comprising interval arrangement and periodic arrangement; freezing or stopping updating of the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement mode; training the neural network to be trained according to the frozen parameter groups or the stopped updating parameter groups.

7. A device for training a neural network, characterized by comprises: an obtaining module, configured to obtain a neural network to be trained, the neural network to be trained being used to perform at least one of target detection, image segmentation, natural language processing or speech recognition; a processing module, configured to group parameters of the neural network to be trained to obtain M groups of parameters, M being a positive integer greater than or equal to 1; the obtaining module is further configured to obtain a sampling probability distribution and a training iteration step arrangement mode, the sampling probability distribution being used to represent the probability of each group of parameters in the M groups of parameters being sampled in each training iteration step, and the training iteration step arrangement mode comprising interval arrangement and periodic arrangement; the processing module is further configured to freeze or stop updating of the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement mode; training the neural network to be trained according to the frozen parameter groups or the stopped updating parameter groups.

8. The apparatus of claim 7, wherein, The processing module freezes or stops updating of the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement mode, comprising: determining a first iteration step according to the training iteration step arrangement mode, the first iteration step being an iteration step to be sampled; determining an mth group of parameters to be sampled in the first iteration step according to the sampling probability distribution, m being less than or equal to M-1; freezing the mth group of parameters in the first iteration step to a first group of parameters, the freezing of the mth group of parameters in the first iteration step to the first group of parameters indicating that gradient calculation is not performed on the mth group of parameters to the first group of parameters, and parameter updating is not performed.

9. The apparatus of claim 7, wherein, The processing module freezes or stops updating of the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement mode, comprising: determining a first iteration step according to the training iteration step arrangement mode, the first iteration step being an iteration step to be sampled; determining an mth group of parameters to be sampled in the first iteration step according to the sampling probability distribution, m being less than or equal to M-1; stopping updating of the mth group of parameters in the first iteration step to a first group of parameters, the stopping updating of the mth group of parameters in the first iteration step to the first group of parameters indicating that gradient calculation is performed on the mth group of parameters to the first group of parameters, and parameter updating is not performed.

10. The apparatus of claim 8 or 9, wherein, When the training iteration step arrangement mode is interval arrangement, the processing module determines the first iteration step according to the training iteration step arrangement mode, including: determining a first interval; in a plurality of training iteration steps, one or more first iteration steps are determined every first interval.

11. The apparatus of claim 8 or 9, wherein, When the training iteration step arrangement mode is periodic arrangement, the processing module determines the first iteration step according to the training iteration step arrangement mode, including: determining that the number of first iteration steps is M-1; determining a first period according to the number of first iteration steps and a first proportion, the first period including the first iteration step and the iteration step of whole network training, the first proportion being the proportion of the first iteration step in the first period, the first iteration step being the last M-1 iteration steps of the first period.

12. An apparatus for data processing, characterized by including: an acquisition module, configured to acquire to-be-processed data; a processing module, configured to process the to-be-processed data according to a target neural network, the target neural network being used for performing at least one of target detection, image segmentation, natural language processing or speech recognition, the target neural network being obtained through training, and the training of the target neural network including: acquiring a neural network to be trained; grouping parameters of the neural network to be trained to obtain M groups of parameters, M being a positive integer greater than or equal to 1; acquiring a sampling probability distribution and a training iteration step arrangement mode, the sampling probability distribution being used for representing the probability of sampling each group of parameters in the M groups of parameters in each training iteration step, and the training iteration step arrangement mode including interval arrangement and periodic arrangement; freezing or stopping updating the sampled parameter groups according to the sampling probability distribution and the training iteration step arrangement mode; training the neural network to be trained according to the frozen parameter groups or the stopped-updating parameter groups.

13. A chip, characterized by The chip includes a processor and a memory, the processor and the memory are coupled, the memory stores instructions, and the processor is used for executing the instructions stored on the memory, when the instructions are executed, the processor is used for executing the method in any one of claims 1 to 5 or claim 6.

14. A computer-readable storage medium, characterized in that, The computer readable medium stores program code for execution by a device, which when executed by the device, causes the device to perform the method in any one of claims 1 to 5 or claim 6.

Images