Network pruning method, data processing method and device, processing core and electronic device

Abstract

The present disclosure provides a network pruning method, which comprises: in a neural network training process, determining importance weights of convolution kernels of a neural network relative to a sample data set according to N batches of sample data in the sample data set, the neural network comprising at least one convolution layer, each convolution layer comprising a plurality of convolution kernels, and N being an integer greater than 1; and performing pruning processing on the convolution kernels of the trained neural network according to the importance weights to obtain a target network after pruning, the trained neural network being obtained by training according to the N batches of sample data. The present disclosure also provides a data processing method and device, a processing core and an electronic device.

Description

Technical Field

[0001] This disclosure relates to the field of processor technology, and in particular to a network pruning method, a data processing method and apparatus, a processing core device, and a medium. Background Technology

[0002] Convolutional Neural Networks (CNNs), as one of the representative network structures in deep learning technology, have made breakthrough progress and been widely used in many fields such as image processing, speech recognition, and natural language processing.

[0003] As deep learning tasks become increasingly complex, the redundancy of convolutional neural networks (such as parameter complexity and computational complexity) also gradually increases. To reduce network redundancy, it is necessary to compress convolutional neural networks through network pruning, thereby simplifying the network structure and speeding up network operation. Summary of the Invention

[0004] This disclosure provides a network pruning method, a data processing method and apparatus, a processing core device, and a medium.

[0005] In a first aspect, this disclosure provides a network pruning method, which includes: during the training of a neural network, determining the importance weights of the convolutional kernels of the neural network relative to the sample data set based on N batches of sample data in the sample data set, wherein the neural network includes at least one convolutional layer, each convolutional layer includes multiple convolutional kernels, and N is an integer greater than 1; and pruning the convolutional kernels of the trained neural network according to the importance weights to obtain the pruned target network, wherein the trained neural network is trained based on N batches of sample data.

[0006] Secondly, this disclosure provides a data processing method, which includes: processing data to be processed through a target network to obtain a processing result of the data to be processed, wherein the target network is obtained by processing according to the network pruning method of the first aspect above.

[0007] Thirdly, this disclosure provides a network pruning device, which includes: a computing module, used to determine the importance weights of the convolutional kernels of the neural network relative to the sample data set based on N batches of sample data in the sample data set during the training process of the neural network, wherein the neural network includes at least one convolutional layer, each convolutional layer includes multiple convolutional kernels, and N is an integer greater than 1; and a pruning module, used to prune the convolutional kernels of the trained neural network according to the importance weights to obtain the pruned target network, wherein the trained neural network is trained based on N batches of sample data.

[0008] Fourthly, this disclosure provides a data processing apparatus for processing data to be processed through a target network to obtain a processing result of the data to be processed, wherein the target network is obtained by processing according to the network pruning apparatus of the third aspect described above.

[0009] Fifthly, this disclosure provides a processing core that includes the network pruning device of the third aspect and the data processing device of the fourth aspect described above.

[0010] In a sixth aspect, this disclosure provides a processing kernel for loading a neural network model to complete deep learning processing, wherein the convolutional kernel in the neural network model is a convolutional kernel obtained according to the network pruning method of the first aspect described above.

[0011] In a seventh aspect, this disclosure provides an electronic device comprising: a plurality of processing cores; and an on-chip network configured to interact with data between the plurality of processing cores and external data; wherein one or more processing cores store one or more instructions, and the one or more instructions are executed by the one or more processing cores to enable the one or more processing cores to perform the network pruning method of the first aspect and the data processing method of the second aspect described above.

[0012] Eighthly, this disclosure provides a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processing core, implements the network pruning method of the first aspect and the data processing method of the second aspect described above.

[0013] The network pruning method, data processing method and apparatus, processing kernel and electronic device provided in this disclosure can prune the convolution kernel of the trained neural network according to the importance weight of the convolution kernel relative to the sample data set during the neural network training process. This enables an overall evaluation of the importance of the convolution kernel based on the importance weight of the convolution kernel on the global sample data set, thereby improving the universality of the importance evaluation results of the neural network convolution kernel on the global sample dataset.

[0014] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0015] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the embodiments of the present disclosure to explain the disclosure and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0016] Figure 1 A flowchart of the network pruning method provided in this embodiment of the disclosure;

[0017] Figure 2 A flowchart illustrating the specific process for determining the importance weight of a convolutional kernel relative to a sample dataset, provided in embodiments of this disclosure;

[0018] Figure 3 A detailed flowchart of the extrusion excitation transformation provided in the embodiments of this disclosure;

[0019] Figure 4 This is a schematic diagram of the extrusion excitation transformation module provided in an embodiment of the present disclosure;

[0020] Figure 5 A flowchart illustrating the process of updating importance weights provided in this embodiment of the disclosure;

[0021] Figure 6 This is a schematic diagram illustrating the specific process of pruning in an embodiment of this disclosure;

[0022] Figure 7 The processing flow of the network pruning method as an exemplary embodiment of this disclosure;

[0023] Figure 8 A flowchart of a data processing method according to an embodiment of the invention;

[0024] Figure 9 A block diagram of the network pruning device provided in the embodiments of this disclosure;

[0025] Figure 10 This is a block diagram of the data processing apparatus provided in the embodiments of this disclosure;

[0026] Figure 11 This is a block diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0027] To enable those skilled in the art to better understand the technical solutions of this disclosure, exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments of this disclosure to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0028] Where there is no conflict, the various embodiments of this disclosure and the features thereof in the embodiments may be combined with each other.

[0029] As used herein, the term “and / or” includes any and all combinations of one or more related enumerated entries.

[0030] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. As used herein, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will also be understood that when the terms “comprising” and / or “made of” are used in this specification, they specify the presence of features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or groups thereof. Words such as “connected” or “linked” are not limited to physical or mechanical connections but can include electrical connections, whether direct or indirect.

[0031] Unless otherwise specified, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in commonly used dictionaries should be interpreted as having a meaning consistent with their meaning in the context of the relevant art and this disclosure, and will not be interpreted as having an idealized or overly formal meaning, unless expressly so defined herein.

[0032] In this embodiment of the disclosure, network pruning is a neural network compression method. By deleting relatively redundant and unimportant connections, nodes, and convolution kernels in the neural network that have little impact on the output results, it achieves the effects of reducing the computational and storage requirements of the network, simplifying the network structure, and speeding up the network operation.

[0033] In some scenarios, pruning of convolutional neural networks can be divided into pruning based on weights and pruning based on features, depending on the pruning criteria.

[0034] Weight-based pruning methods, such as resetting weights with a certain value (e.g., norm) less than a threshold to 0, are often applied after model training. Another example is adding regularization based on certain weight values ​​to learn sparse weights during training. However, weight-based pruning methods often require additional techniques to ensure the topology of the pruned weights is structured; otherwise, it can hinder model deployment and impact model acceleration on GPUs.

[0035] For feature-based pruning methods, such as calculating a certain value (e.g., norm) for a feature of a specific channel across the entire dataset, the corresponding weight / kernel is set to 0 if it is less than a threshold. This type of pruning method can directly achieve structured channel pruning. Furthermore, this pruning method is often applied after model training.

[0036] This disclosure provides a network pruning method, data processing method and apparatus, processing core and electronic device, which can prune the network during training based on the input sample data set to obtain the pruned target network.

[0037] In some embodiments, the target network is used to perform data processing tasks, including any one of image processing, speech processing, text processing, and video processing tasks. That is, the target network of this disclosure can be widely applied in various fields. The execution of the network pruning method in this disclosure is not limited to any specific application scenario. When processing data to be processed for any data processing task, as long as the deep learning algorithm used employs convolutional layers, the network pruning method provided in this disclosure can be used for network pruning, without limitation on the specific type of data processing task.

[0038] Figure 1 A flowchart illustrating a network pruning method provided in an embodiment of this disclosure.

[0039] Reference Figure 1 This disclosure provides a network pruning method, which includes the following steps.

[0040] S110, During the training of the neural network, the importance weight of the convolutional kernel of the neural network relative to the sample data set is determined based on the N batches of sample data in the sample data set. The neural network includes at least one convolutional layer, and each convolutional layer includes multiple convolutional kernels, where N is an integer greater than 1.

[0041] S120, according to the importance weight, the convolution kernels of the trained neural network are pruned to obtain the pruned target network. The trained neural network is trained based on N batches of sample data.

[0042] According to the network pruning method of this disclosure, during the training process of a neural network, the convolutional kernels of the neural network are pruned based on their importance weights relative to the sample dataset. In this disclosure, the importance of the convolutional kernels can be comprehensively evaluated based on the global sample dataset according to their importance weights. The value of the importance weights depends not on a single sample, but on the entire sample dataset. This improves the universality of the importance evaluation results of the neural network's convolutional kernels on the global sample dataset, resulting in a more lightweight, efficient, and accurate network after pruning.

[0043] Figure 2 This diagram illustrates a specific flowchart of an embodiment of the present disclosure for determining the importance weight of a convolutional kernel relative to a sample dataset. (Refer to...) Figure 2 In some embodiments, the step of determining the importance weight of the convolution kernel of the neural network relative to the sample data set based on N batches of sample data in the sample data set in S110 may specifically include the following sub-steps.

[0044] S11, for the i-th batch of neural network training, perform squeeze excitation transformation on the output data of the m-th convolutional layer to obtain the importance sub-weights of multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch. The input data of the first convolutional layer is the sample data of the i-th batch, 1≤i≤N, m≥1 and i and m are integers.

[0045] In this step, convolutional layers can be used to perform convolution operations on the input data. The input data of the first convolutional layer is the sample data of the i-th batch, and the input of other convolutional layers besides the first convolutional layer is the output of the previous convolutional layer or other network layers.

[0046] In this embodiment of the disclosure, the output data of the m-th convolutional layer can be subjected to squeeze excitation transformation through the squeeze and excite (SE) module.

[0047] The squeeze excitation transformation is used to explicitly model the interdependencies between feature channels, enabling adaptive recalibration of feature responses across channels. This allows the network to automatically learn the importance of each feature channel and then prioritize features useful for the current task while suppressing those less relevant. Through this mechanism, the network can learn to selectively emphasize informative features using global information, thereby enhancing its representational capabilities.

[0048] S12, based on the importance sub-weights of the m-th convolutional layer relative to the i-th batch, update the importance weights of the m-th convolutional layer relative to the first i-1 batches to obtain the importance weights of the m-th convolutional layer relative to the first i batches, where the importance weights of the m-th convolutional layer relative to the first 0 batches are the initial importance weights of the multiple convolutional kernels of the m-th convolutional layer.

[0049] S13, based on the importance weights of the m-th convolutional layer relative to the first N batches, determine the importance weights of the multiple convolutional kernels of the m-th convolutional layer relative to the sample data set.

[0050] Through the above steps S11-S13, the importance sub-weights of multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch can be calculated by squeezing excitation transformation, and the importance sub-weights of the i-th batch are updated to the importance weights of the first i-1 batches. The importance weights of the m-th convolutional layer relative to the first N batches are calculated iteratively, and the importance weights of multiple convolutional kernels of the m-th convolutional layer relative to the sample data set are finally obtained, thus obtaining the importance features of multiple convolutional kernels of the m-th convolutional layer relative to the global samples.

[0051] Figure 3 A detailed flowchart illustrating the extrusion excitation transformation provided in an embodiment of this disclosure is shown. For example... Figure 3 As shown, in some embodiments, the squeezing excitation transformation mainly includes global average pooling processing and fully connected processing; the above step S11 may specifically include the following sub-steps.

[0052] S31, determine the feature vector of the input data of the m-th convolutional layer.

[0053] S32 performs global average pooling on the feature vector of the input data of the m-th convolutional layer.

[0054] S33, perform at least one fully connected operation on the feature vector after global average pooling to obtain the importance sub-weights of multiple convolution kernels of the m-th convolutional layer relative to the i-th batch.

[0055] Through the above steps S31-S32, the importance features of the multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch can be characterized based on the importance sub-weights of the multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch. This provides a data calculation basis for updating the importance weights of the first i-1 using the importance sub-weights of the i-th batch during the subsequent model training process, thereby finally obtaining the importance features of the multiple convolutional kernels of the m-th convolutional layer relative to the global samples.

[0056] To better understand the method for processing importance sub-weights in the embodiments of this disclosure, the following is combined with... Figure 4 This document details the specific process of performing a squeezing excitation transformation on the output data of the m-th convolutional layer to obtain the importance sub-weights of multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch in this embodiment.

[0057] Figure 4 A schematic diagram of the extrusion excitation transformation module provided in an embodiment of this disclosure is shown. Figure 4 In the process, the squeezing excitation transformation module includes a global average pooling layer and two fully connected layers.

[0058] like Figure 4 As shown, the input to the SE module is a four-dimensional tensor data a(b, ci, hi, wi). This four-dimensional tensor data includes four factors: factor b represents the output data of the i-th batch in the m-th convolutional layer during neural network training; if m=1, then the input data of the first convolutional layer is the sample data of the i-th batch; factor ci represents the number of channels in each sample data; factor hi represents the height data; and factor wi represents the height data; where i represents the batch number.

[0059] exist Figure 4 In this context, the processing flow for four-dimensional tensor data a(b, ci, hi, wi) includes squeezing operations and excitation operations.

[0060] S401, Squeeze operation.

[0061] In this step, global average pooling can be used as the squeeze operation. Specifically, global average pooling can be performed on the input four-dimensional tensor data to squeeze the feature channels. Through global average pooling, the four-dimensional tensor data a(b, c) is spatially compressed. i h i w i The data is compressed to obtain the compressed four-dimensional tensor data (b, c). i ,1,1).

[0062] In the Squeeze operation, the output dimension matches the number of feature channels in the input, representing the global distribution of the response on the feature channels.

[0063] S402, Excite operation.

[0064] In this step, the Excite operation can be implemented using a two fully connected (FC) layer structure. The two fully connected layers model the correlation between channels and output the same number of weights as the input features.

[0065] For example, the compressed four-dimensional tensor data (b, c) can be processed first through a fully connected layer. i From b, c, and 1, we obtain the dimensionality-reduced four-dimensional tensor data (b, c). m ,1,1); where c m Less than c i Then, a second fully connected layer is used to process the dimensionality-reduced four-dimensional tensor data (b, c). m We perform dimensionality upscaling on (b, c) to obtain the upscaled four-dimensional tensor data a1(b, c). o,1,1); where c o It can be equal to c i .

[0066] In some embodiments, the above-described extrusion excitation transformation process can be expressed as the following expression (1).

[0067] a1 = Squeeze_Excite(Xi)(1)

[0068] In the above expression (1), X i Let represent the input data of the i-th batch in the m-th convolutional layer, Squeeze_Excite represent the squeezing transformation operation, and a1 represent the importance sub-weights of the multiple convolutional kernels of the m-th convolutional layer to the i-th batch of input data, for example, a1(b, c o (1, 1). Where X is the input data of the first convolutional layer. i This represents the sample data for the i-th batch.

[0069] In some embodiments, the importance sub-weights a1 of the input data of the i-th batch can be subjected to a first data transformation process on the multiple convolutional kernels of the m-th convolutional layer obtained by the compression transformation, and the value of each element in a1 can be restricted to the range [-1, 1]. This is to limit the value obtained by updating the importance sub-weights of the first i-1 batches according to the importance sub-weights of the i-th batch.

[0070] For example, in the first data conversion process, the elements in a1 that are greater than 1 are set to 1, the elements in a1 that are less than -1 are set to -1, and the values ​​that are greater than or equal to -1 and less than or equal to 1 are retained and the first data conversion process is not performed.

[0071] It should be understood that the specific value ranges mentioned above are merely illustrative and can be customized as needed in specific application scenarios.

[0072] During the processing of the two fully connected layers mentioned above, weights can be generated for each feature channel using specific parameters, and these specific parameters are used to explicitly construct the correlation between feature channels; the weights output after the squeezing and excitation operations can be regarded as the importance of each feature channel after feature selection.

[0073] It should be understood that when the number of fully connected layers is 1, the processing steps of this 1-layer fully connected layer are equivalent to the processing steps of the two fully connected layers.

[0074] In this embodiment of the present disclosure, through the above steps S302 and S303, for the i-th batch of neural network training, the output data of the m-th convolutional layer can be subjected to a squeeze excitation transformation to obtain the importance sub-weights of multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch, thereby characterizing the importance of multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch through the importance sub-weights.

[0075] Figure 5 This diagram illustrates a process flowchart for updating importance weights according to an embodiment of this disclosure. Figure 5 As shown, in some embodiments, the training of each batch of sample data in the neural network includes at least one iterative step; step S12 above may specifically include the following sub-steps.

[0076] S51, calculate the average importance sub-weights of the m-th convolutional layer in the i-th batch over the number of samples in the i-th batch.

[0077] S52, at each current iteration step in the neural network training process, the ratio of the average value to the step order of the current iteration step is used as the importance sub-weight of the m-th convolutional layer relative to the i-th batch in the current iteration step.

[0078] S53, accumulate the importance sub-weights of the m-th convolutional layer relative to the i-th batch in each iteration step to obtain the importance sub-weights of the m-th convolutional layer relative to the i-th batch;

[0079] S54, the importance sub-weights of the m-th convolutional layer relative to the i-th batch are summed with the importance weights of the (i-1)-th batch to obtain the importance weights of the m-th convolutional layer relative to the first i batches.

[0080] For example, the process of processing the importance weight of the m-th convolutional layer relative to the first i batches obtained by accumulating the above S51-S54 can be expressed as the following expression (2).

[0081] a=a+a1.mean(0)[...,None] / (step_cnt+1)(2)

[0082] In the above expression (2), step_cnt represents the step order (e.g., which step) in the entire training iteration process. step_cnt usually starts counting the step order from 0, i.e., step_cnt = i - 1. When step_cnt = 0, it is the first step of the network iteration training. a1 is the importance sub-weight of the m-th convolutional layer in the i-th batch. a1.mean(0)[...,None] means that a1 is averaged along the dimension of sample b, while other dimensions remain unchanged. In addition, i = step_cnt + 1 represents the i-th step in the entire training iteration process. The step number starts from 0, so 1 is added to prevent the division by zero in expression (2).

[0083] It should be understood that when step_cnt starts counting from 1, (step_cnt+1) in the above expression can be replaced with step_cnt.

[0084] In this embodiment of the disclosure, utilizing the squeezing excitation transformation process described above in combination with steps S11, S31-S33, and S401 and S402, for data features (e.g., four-dimensional tensor data a(b, ci, hi, wi)) of different batches of sample data in the current layer (e.g., the m-th convolutional layer), after the squeezing excitation transformation, importance sub-weights are usually obtained with a high probability (e.g., after squeezing excitation transformation of four-dimensional tensor data a(b, ci, hi, wi), four-dimensional tensor data a1(b, ci, hi, wi) is obtained). o ,1,1)).

[0085] Since the features a of different input samples in the current layer will likely result in different a' after the squeezing excitation transformation, in order to remove redundant convolution kernels on the entire dataset, we can refer to the basic principle of moving mean in the BatchNorm algorithm, which accelerates the convergence speed of deep networks.

[0086] In other words, to maintain stability during network training, the moving average can be used to update the mean. The meaning of moving average is: when updating the current value, a certain proportion of previous values ​​are preserved. Taking the mean as an example, when updating the current value, a certain proportion of the previous mean is preserved. This ensures that the previously learned feature distribution is retained after each data normalization, while simultaneously completing the normalization operation. This accelerates network training and ensures that the importance weights of the calculated convolutional kernels depend on the entire dataset rather than a single sample.

[0087] In the scheme of this embodiment, there is usually little correlation between the first sample and the nth sample in a neural network. In order to measure the importance features of the convolution kernel in the m-th convolutional layer on the entire sample set and finally remove redundant convolution kernels on the entire dataset, the solution of the importance sub-weights of the m-th convolutional layer relative to a batch in this scheme requires averaging the number of samples along the sample dimension, that is, taking all samples in the entire batch into account. In the entire training process, the sum of the sample iterations of all different batches used in the training is the total number of samples. That is, the importance weights of multiple convolution kernels in a convolutional layer that are finally calculated are an overall importance evaluation relative to all samples in the world.

[0088] Figure 6 A schematic diagram illustrating the specific process of pruning in an embodiment of this disclosure is shown.

[0089] like Figure 6 As shown, in some embodiments, step S120, which involves pruning the convolutional kernels of the trained neural network according to importance weights, may specifically include the following sub-steps.

[0090] S61, perform a dot product between the importance weights and the convolution kernel of the trained neural network to obtain the value of the convolution kernel of the trained neural network; S62, subtract convolution kernels whose values ​​are less than a predetermined threshold.

[0091] Through the above steps S61 and S62, in this embodiment of the present disclosure, the convolutional kernels to be removed do not depend on a single sample. Instead, the operation of removing convolutional kernels is performed only after the importance weights of the convolutional kernels of the neural network relative to the sample data set are statistically analyzed. In this way, the removed convolutional kernels are representative of the entire sample data set, avoiding the situation where the convolutional kernels are removed based only on the importance weights calculated for a single sample. In such cases, the removed convolutional kernels may be unimportant to the current sample but important to other kernels. This ensures that a lighter and more efficient network is obtained after pruning, and the pruning results are more accurate and reliable.

[0092] In some embodiments, before step S61, the importance weights of the calculated convolution kernels of the neural network relative to the sample data set are subjected to a second data transformation process, and the value of the importance weights is restricted to the range [0, 1].

[0093] For example, in the second data transformation process, values ​​with importance weights greater than 1 are set to 1, values ​​with importance weights less than 0 are set to 0, and values ​​greater than or equal to 0 and less than or equal to 1 are retained and the above-mentioned second data transformation process is not performed.

[0094] In some embodiments, the convolutional kernels of the trained neural network are pruned, and the importance weights used are regularized importance weights. In this embodiment, before step S61, the method further includes: regularizing the importance weights to obtain regularized importance weights.

[0095] In this embodiment of the disclosure, in order to guide the obtaining of the most sparse importance weights through the squeezing excitation transformation, a regularization can be introduced to make the elements in the importance weights as close to 0 as possible; the strength of the regularization can be used to adaptively balance the pruning intensity, thereby balancing network performance and efficiency.

[0096] Figure 7 The processing flow of the network pruning method according to an exemplary embodiment of this disclosure is illustrated. For example... Figure 7 As shown, the processing flow may include the following steps.

[0097] First, during the training process of the neural network, for the i-th batch sample x... i Its four-dimensional tensor data is a(b, ci, hi, wi).

[0098] Secondly, the importance sub-weights a1(b, c0, 1, 1) of the multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch are obtained through the squeeze excitation transformation. Based on the importance sub-weights of the multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch, the importance weights of the m-th convolutional layer relative to the first i-1 batches are updated to obtain the importance weights a2(c0, 1, 1, 1) of the m-th convolutional layer relative to the first i-th batches, thereby determining the importance weights of the multiple convolutional kernels of the m-th convolutional layer relative to the sample data set.

[0099] Next, after the neural network training is completed, the importance weight a2(c0, 1, 1, 1) of the m-th convolutional layer relative to the previous i batches is multiplied by the convolutional kernel of the trained neural network to obtain the value of the convolutional kernel of the trained neural network, and convolutional kernels with values ​​less than a predetermined threshold are subtracted.

[0100] Then, the pruned neural network will be used to process the i-th batch sample x. i Perform convolution operation to obtain the i-th batch sample x. i The result of convolution operation based on the pruned neural network.

[0101] The network pruning method described in the above embodiments of this disclosure can determine the importance weight of convolutional kernels relative to the sample data set for each batch during the training process of the neural network. The dimension of the importance weight is the same as the number of convolutional kernels, co, with each dimension corresponding to one convolutional kernel, which can be used to characterize the importance or redundancy of the convolutional kernel. The importance weight changes dynamically for different batches. At the end of training, redundant convolutional kernels can be identified from the co convolutional kernels based on the final obtained importance weights. Redundant convolutional kernels are then removed through reparameterization to obtain the trained neural network. Thus, the importance weight of the convolutional kernels can depend on the entire sample data set, improving the universality of the neural network's convolutional kernel importance evaluation results on the global sample dataset, thereby obtaining a more lightweight, efficient, and accurate and reliable network after pruning.

[0102] In this embodiment, reparameterization can be understood as the reconstruction of network parameters in the network inference module by modifying the network parameters in the network training module. Reparameterization has two functions: improving model performance and altering model structure to achieve certain objectives. Therefore, in this embodiment, convolutional kernel reduction can be performed during reparameterization to obtain a pruned network.

[0103] Figure 8 A flowchart illustrating a data processing method according to an embodiment of the invention is shown. Figure 8 As shown, the data processing method includes the following steps.

[0104] S810, the target network is used to process the data to be processed to obtain the processing result of the data to be processed. The target network is obtained by processing according to the network pruning method in the embodiments of this disclosure.

[0105] The data processing method of this disclosure uses a target network obtained by the network pruning method described in the above embodiments. Data processing based on this target network can significantly reduce computational costs, and the processing results of the pruned model are more reliable across the entire dataset.

[0106] Figure 9 This is a block diagram of the network pruning device provided in an embodiment of the present disclosure.

[0107] Reference Figure 9 This disclosure provides a network pruning device 900, which may include the following modules.

[0108] The calculation module 910 is used to determine the importance weight of the convolution kernel of the neural network relative to the sample data set based on N batches of sample data in the sample data set during the training process of the neural network. The neural network includes at least one convolutional layer, each convolutional layer includes multiple convolutional kernels, and N is an integer greater than 1.

[0109] The pruning module 920 is used to prune the convolutional kernels of the trained neural network according to importance weights to obtain the pruned target network. The trained neural network is trained based on N batches of sample data.

[0110] In some embodiments, the computing module 910 includes the following units.

[0111] The weight calculation unit is used to perform a squeezing excitation transformation on the input data of the m-th convolutional layer for the i-th batch of neural network training, so as to obtain the importance sub-weights of multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch. The input data of the first convolutional layer is the sample data of the i-th batch, 1≤i≤N, m≥1 and i and m are integers.

[0112] The weight determination unit is used to update the importance weight of the m-th convolutional layer relative to the first i-1 batches based on the importance sub-weight of the m-th convolutional layer relative to the i-th batch, so as to obtain the importance weight of the m-th convolutional layer relative to the first i batches. Here, the importance weight of the m-th convolutional layer relative to the first 0 batches is the initial importance weight of the multiple convolutional kernels of the m-th convolutional layer.

[0113] The weight determination unit is used to determine the importance weights of multiple convolutional kernels of the m-th convolutional layer relative to the sample data set based on the importance weights of the m-th convolutional layer relative to the previous N batches.

[0114] In some embodiments, the squeeze excitation transformation includes global average pooling and fully connected processing; the weight calculation unit is specifically used to: determine the feature vector of the input data of the m-th convolutional layer; perform global average pooling on the feature vector of the input data of the m-th convolutional layer; and perform at least one fully connected processing on the feature vector after global average pooling to obtain the importance sub-weights of multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch.

[0115] In some embodiments, training each batch of sample data for the neural network includes at least one iterative step; the weight determination unit includes the following sub-units.

[0116] The mean calculation subunit is used to calculate the average value of the importance subweights of the m-th convolutional layer in the i-th batch over the number of samples in the i-th batch.

[0117] The ratio calculation subunit is used at each current iteration step in the neural network training process to calculate the ratio of the average value to the step order of the current iteration step as the importance sub-weight of the m-th convolutional layer relative to the i-th batch at the current iteration step.

[0118] The accumulation unit is used to accumulate the importance sub-weights of the m-th convolutional layer relative to the i-th batch in each iteration step to obtain the importance sub-weights of the m-th convolutional layer relative to the i-th batch.

[0119] The accumulation unit is also used to accumulate the importance sub-weights of the m-th convolutional layer relative to the i-th batch and the importance weights of the (i-1)-th batch to obtain the importance weights of the m-th convolutional layer relative to the previous i batches.

[0120] In some embodiments, the pruning module 910 is specifically used to perform a dot product calculation between the importance weight and the convolution kernel of the trained neural network to obtain the value of the convolution kernel of the trained neural network; and to subtract convolution kernels whose values ​​are less than a predetermined threshold.

[0121] In some embodiments, the importance weight is a regularized importance weight; the network pruning device 900 further includes: a regularization module, used to regularize the importance weight before performing a dot product calculation with the convolution kernel of the trained neural network to obtain a regularized importance weight.

[0122] In some embodiments, the target network is used to perform data processing tasks, which include any one of image processing tasks, speech processing tasks, text processing tasks, and video processing tasks.

[0123] According to the network pruning device of the present disclosure, during the training process of a neural network, the convolution kernels of the neural network are pruned according to the importance weights of the convolution kernels of the neural network relative to the sample data set. This improves the universality of the importance evaluation results of the convolution kernels of the neural network on the global sample dataset, thereby obtaining a more lightweight, efficient, and accurate and reliable network after pruning.

[0124] It should be clarified that this disclosure is not limited to the specific configurations and processes described in the foregoing embodiments and shown in the figures. For the sake of convenience and brevity, detailed descriptions of known methods are omitted here, and the specific working processes of the systems, modules, and units described above can be referred to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0125] Figure 10 This is a block diagram of the data processing apparatus provided in the embodiments of this disclosure.

[0126] Reference Figure 10This disclosure provides a data processing device 1000, which may include the following modules.

[0127] The calculation module 1010 is used to process the data to be processed through the target network to obtain the processing result of the data to be processed, wherein the target network is obtained by processing according to the network pruning method described in the above embodiments of this disclosure.

[0128] The data processing apparatus according to the embodiments of this disclosure can perform data processing on the target network obtained by the network pruning method described in the above embodiments, thereby improving data processing efficiency and reducing computational costs. Furthermore, the processing results of the pruned model according to the embodiments of this disclosure are more reliable.

[0129] This disclosure also provides a processing core, which includes the network pruning device or the data processing device described above.

[0130] This disclosure also provides a processing kernel for loading a neural network model to complete deep learning processing, wherein the convolution kernel in the neural network model is a convolution kernel obtained according to the network pruning method described above.

[0131] Figure 11 This is a block diagram of an electronic device provided in an embodiment of the present disclosure.

[0132] Reference Figure 11 This disclosure provides an electronic device that includes multiple processing cores 1101 and an on-chip network 1102. The multiple processing cores 1101 are all connected to the on-chip network 1102, and the on-chip network 1102 is used to exchange data between the multiple processing cores and external data.

[0133] One or more processing cores 1101 store one or more instructions, and the one or more instructions are executed by one or more processing cores 1101 to enable one or more processing cores 1101 to perform the network pruning method or data processing method described above.

[0134] Furthermore, this disclosure also provides a computer-readable medium storing a computer program thereon, wherein the computer program, when executed by a processing core, implements the network pruning method or data processing method described above.

[0135] It will be understood by those skilled in the art that all or some of the steps, systems, or apparatuses disclosed above, and their functional modules / units, can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned in the above description does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0136] Example embodiments have been disclosed herein, and while specific terminology has been used, it is for illustrative purposes only and should be construed as such, and is not intended to be limiting. In some instances, it will be apparent to those skilled in the art that features, characteristics, and / or elements described in connection with particular embodiments may be used alone, or in combination with features, characteristics, and / or elements described in connection with other embodiments, unless otherwise expressly indicated. Therefore, those skilled in the art will understand that various changes in form and detail may be made without departing from the scope of this disclosure as set forth by the appended claims.

Claims

1. A network pruning method, comprising: During the training of the neural network, the importance weight of the convolutional kernel of the neural network relative to the sample data set is determined based on N batches of sample data in the sample data set. The neural network includes at least one convolutional layer, and each convolutional layer includes multiple convolutional kernels, where N is an integer greater than 1. Based on the importance weights, the convolutional kernels of the trained neural network are pruned to obtain the pruned target network. The trained neural network is trained based on the N batches of sample data. The pruned target network is loaded into the processing kernel so that the processing kernel can perform data processing tasks based on the pruned target network. The data processing tasks include any one of image processing tasks, speech processing tasks, text processing tasks, and video processing tasks. The step of determining the importance weight of the neural network's convolutional kernel relative to the sample data set based on N batches of sample data includes: For the i-th batch of training the neural network, the input data of the m-th convolutional layer is subjected to a squeezing excitation transformation to obtain the importance sub-weights of multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch. The input data of the 1st convolutional layer is the sample data of the i-th batch, 1≤i≤N, m≥1 and i and m are integers. Based on the importance sub-weights of the m-th convolutional layer relative to the i-th batch, the importance weights of the m-th convolutional layer relative to the first i-1 batches are updated to obtain the importance weights of the m-th convolutional layer relative to the first i batches. The importance weights of the m-th convolutional layer relative to the first 0 batches are the initial importance weights of the multiple convolutional kernels of the m-th convolutional layer. Based on the importance weights of the m-th convolutional layer relative to the previous N batches, the importance weights of multiple convolutional kernels of the m-th convolutional layer relative to the sample data set are determined.

2. The method according to claim 1, wherein, The training of each batch of sample data in the neural network includes at least one iterative step; updating the importance weight of the m-th convolutional layer relative to the first i-1 batches based on the importance sub-weights of the m-th convolutional layer relative to the i-th batch, to obtain the importance weight of the m-th convolutional layer relative to the first i batches, includes: Calculate the average importance sub-weights of the m-th convolutional layer in the i-th batch over the number of samples in the i-th batch; In each current iteration step of the neural network training process, the ratio of the average value to the step order of the current iteration step is used as the importance sub-weight of the m-th convolutional layer relative to the i-th batch in the current iteration step; The importance sub-weights of the m-th convolutional layer relative to the i-th batch are accumulated in each iteration step to obtain the importance sub-weights of the m-th convolutional layer relative to the i-th batch; The importance weight of the m-th convolutional layer relative to the i-th batch is summed with the importance weight of the (i-1)-th batch to obtain the importance weight of the m-th convolutional layer relative to the first i batches.

3. The method according to claim 1, wherein, The step of pruning the convolutional kernels of the trained neural network according to the importance weights includes: The importance weights are multiplied by the convolution kernel of the trained neural network to obtain the value of the convolution kernel of the trained neural network. Subtract convolutional kernels whose subtraction value is less than a predetermined threshold.

4. The method according to claim 3, wherein, The importance weight is the importance weight after regularization. Before performing a dot product calculation between the importance weights and the convolution kernel of the trained neural network, the method further includes: The importance weights are regularized to obtain the regularized importance weights.

5. A data processing method, comprising: The target network is used to process the data to be processed to obtain the processing result of the data to be processed, wherein the target network is obtained by the network pruning method according to any one of claims 1-4.

6. A network pruning device, comprising: The calculation module is used to determine the importance weight of the convolution kernel of the neural network relative to the sample data set based on N batches of sample data in the sample data set during the training process of the neural network. The neural network includes at least one convolutional layer, each convolutional layer includes multiple convolutional kernels, and N is an integer greater than 1. The pruning module is used to prune the convolutional kernels of the trained neural network according to the importance weights to obtain the pruned target network. The trained neural network is trained based on the N batches of sample data. The pruned target network is loaded into the processing kernel so that the processing kernel can perform data processing tasks based on the pruned target network. The data processing tasks include any one of image processing tasks, speech processing tasks, text processing tasks, and video processing tasks. The step of determining the importance weight of the neural network's convolutional kernel relative to the sample data set based on N batches of sample data includes: For the i-th batch of training the neural network, the input data of the m-th convolutional layer is subjected to a squeezing excitation transformation to obtain the importance sub-weights of multiple convolutional kernels of the m-th convolutional layer relative to the i-th batch. The input data of the 1st convolutional layer is the sample data of the i-th batch, 1≤i≤N, m≥1 and i and m are integers. Based on the importance sub-weights of the m-th convolutional layer relative to the i-th batch, the importance weights of the m-th convolutional layer relative to the first i-1 batches are updated to obtain the importance weights of the m-th convolutional layer relative to the first i batches. The importance weights of the m-th convolutional layer relative to the first 0 batches are the initial importance weights of the multiple convolutional kernels of the m-th convolutional layer. Based on the importance weights of the m-th convolutional layer relative to the previous N batches, the importance weights of multiple convolutional kernels of the m-th convolutional layer relative to the sample data set are determined.

7. A data processing apparatus, the data processing apparatus being used to process data to be processed through a target network to obtain a processing result of the data to be processed, wherein, The target network is obtained by the network pruning device according to claim 6.

8. A processing core comprising the network pruning apparatus of claim 6 or the data processing apparatus of claim 7.

9. A processing core, said processing core being used to load a neural network model to perform deep learning processing, wherein, The convolution kernel in the neural network model is a convolution kernel obtained by the network pruning method according to any one of claims 1-4.

10. An electronic device, comprising: Multiple processing cores; as well as The on-chip network is configured to interact with data between the multiple processing cores and external data; One or more processing cores store one or more instructions, which are executed by one or more processing cores to enable the one or more processing cores to perform the network pruning method of any one of claims 1-4 or the data processing method of claim 5.

11. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by the processing core, it implements the network pruning method as described in any one of claims 1-4 or the data processing method as described in claim 5.

Images