Neural network processing method and apparatus, storage medium, and electronic device

Abstract

The application relates to a neural network processing method and device, a storage medium and electronic equipment. The neural network processing method comprises the following steps: obtaining a neural network to be processed, the neural network to be processed comprising a convolution kernel to be pruned, the convolution kernel to be pruned corresponding to an input feature map; determining a target pruning direction of the convolution kernel to be pruned; pruning the convolution kernel parameters of the convolution kernel to be pruned according to the target pruning direction to obtain a pruned convolution kernel; and performing convolution operation on the input feature map by using the pruned convolution kernel according to the target pruning direction, and replacing the input feature map by using the output result in the convolution operation process to obtain an output feature map. Therefore, the parameter quantity of the neural network model can be reduced, and the input and output in the model convolution operation process can reuse the same memory space, which greatly reduces the calculation quantity and the access quantity of the memory in the model operation process.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a neural network processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] In recent years, the development of the Internet of Things (IoT) technology has marked the beginning of a fully intelligent era for people's production and lives. The basic principle of IoT is to connect any object to the internet through various information sensing devices according to agreed-upon protocols to achieve information exchange and communication, enabling intelligent identification, positioning, tracking, monitoring, and management. Building upon this foundation, intelligent IoT technology (AIoT), which integrates artificial intelligence, IoT technology, new semiconductor technology, and edge computing, is also flourishing.

[0003] Smart IoT technology primarily refers to the use of small embedded devices to collect information from multiple sensors. This information is then processed by the CPU, GPU, or NPU on the embedded device using deep neural networks to perform specific tasks, such as machine vision, automatic speech recognition, and real-time translation. Smart IoT technology is currently widely used in various industries, including smart manufacturing, autonomous driving, smart furniture, security, and smart offices.

[0004] However, to improve the accuracy of deep learning models, the number of parameters and computational load can reach hundreds of millions. This means that running high-precision models requires high-performance computing equipment and large amounts of memory, posing a significant challenge to their practical application in AIoT. Therefore, reducing the computational load and memory access during model execution is crucial. Summary of the Invention

[0005] The purpose of this application is to provide a neural network processing method, apparatus, storage medium, and electronic device to reduce the amount of computation and memory access during model operation.

[0006] This application provides a neural network processing method, which includes:

[0007] Obtain the neural network to be processed, which includes the convolutional kernel to be pruned and the convolutional kernel to be pruned corresponds to the input feature map;

[0008] Determine the target pruning direction for the convolution kernel to be pruned;

[0009] Prune the convolution kernel parameters of the convolution kernel to be pruned according to the target pruning direction to obtain the pruned convolution kernel;

[0010] Based on the target pruning direction, the input feature map is convolved using a post-pruned convolution kernel. During the convolution operation, the output result is used to replace the input feature map to obtain the output feature map.

[0011] This application embodiment also provides a neural network processing device, which includes:

[0012] The acquisition module is used to acquire the neural network to be processed, which includes the convolutional kernel to be pruned and the convolutional kernel to be pruned has an input feature map.

[0013] The first determining module is used to determine the target pruning direction of the convolution kernel to be pruned;

[0014] The pruning module is used to prune the convolution kernel parameters of the convolution kernel to be pruned according to the target pruning direction, so as to obtain the pruned convolution kernel;

[0015] The convolution operation module is used to perform convolution operations on the input feature map using the pruned convolution kernel according to the target pruning direction, and to replace the input feature map with the output result during the convolution operation to obtain the output feature map.

[0016] The convolution operation module specifically includes:

[0017] The first determining unit is used to determine the first convolution order and the second convolution order according to the target pruning direction;

[0018] The first convolution operation unit is used to perform convolution operations on each pixel of the input feature image one by one using the pruned convolution kernel according to the first convolution order and the second convolution order. During the convolution operation, the pixel value of each pixel is replaced by the output result corresponding to each pixel to obtain the output feature map.

[0019] The convolution operation module specifically includes:

[0020] The second determining unit is used to determine the convolution direction based on the target pruning direction;

[0021] The affine transformation unit is used to perform an affine transformation on the input feature map according to the convolution direction to obtain a deformed input feature map.

[0022] The block unit is used to divide the deformed input feature map into multiple feature map blocks along the horizontal and vertical axes.

[0023] Grouping units are used to divide multiple feature maps into multiple groups based on the convolution direction;

[0024] The second convolution operation unit is used to perform parallel convolution operations on each feature map block in each group according to the convolution direction using the pruned convolution kernel. During the parallel convolution operation, the pixel values ​​of each feature map block in each group are replaced with the output results corresponding to each feature map block in each group to obtain the output feature map.

[0025] Specifically, the affine transformation unit is used for:

[0026] Based on the convolution direction, the pixel coordinates of the input feature map are subjected to an affine transformation using the scheduling algorithm in the polyhedral model to obtain the transformed pixel coordinates of the deformed input feature map.

[0027] The first determining module includes:

[0028] The third determining unit is used to determine multiple pruning directions of the convolution kernel to be pruned;

[0029] The fourth determining unit is used to determine the parameters of multiple target convolution kernels corresponding to each pruning direction from the convolution kernels to be pruned;

[0030] The summation unit is used to sum the absolute values ​​of multiple target convolution kernel parameters corresponding to each pruning direction to obtain the corresponding sum value;

[0031] The fifth determining unit is used to determine the sampling probability corresponding to each pruning direction based on the sum value corresponding to each pruning direction.

[0032] The sixth determining unit is used to determine the target pruning direction from multiple pruning directions based on the sampling probability.

[0033] Specifically, the pruning module is used for:

[0034] Set all convolution kernel parameters in the convolution kernel to be pruned, except for the target convolution kernel parameter corresponding to the target pruning direction, to zero.

[0035] The neural network processing device also includes:

[0036] The second determining module is used to determine the order of the parameters of each convolution kernel retained in the convolution kernel after pruning, based on the target pruning direction.

[0037] The storage module is used to store the parameters of each convolution kernel retained in the pruned convolution kernel according to the target pruning direction and arrangement order.

[0038] This application also provides a computer-readable storage medium storing multiple instructions adapted for loading by a processor to execute any of the above-described neural network processing methods.

[0039] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps in any of the above-described neural network processing methods.

[0040] The neural network processing method, apparatus, storage medium, and electronic device provided in this application acquire a neural network to be processed, which includes a convolutional kernel to be pruned and a corresponding input feature map. Then, the target pruning direction of the convolutional kernel to be pruned is determined, and the kernel parameters are pruned according to the target pruning direction to obtain a pruned convolutional kernel. Subsequently, the pruned convolutional kernel is used to perform convolution operations on the input feature map according to the target pruning direction. During the convolution operation, the output result is used to replace the input feature map to obtain an output feature map. This not only reduces the number of parameters in the neural network model but also allows the input and output to reuse the same memory space during the model's convolution operation, significantly reducing the computational load and memory access during model operation. Attached Figure Description

[0041] The technical solution and other beneficial effects of this application will become apparent from the following detailed description of specific embodiments in conjunction with the accompanying drawings.

[0042] Figure 1 This is a schematic diagram of a neural network processing system provided in an embodiment of this application;

[0043] Figure 2 This is a flowchart illustrating the neural network processing method provided in an embodiment of this application;

[0044] Figure 3 This is a schematic diagram of the structure of the convolution kernel to be pruned and the target pruning direction provided in the embodiments of this application;

[0045] Figure 4 This is another structural schematic diagram of the convolution kernel to be pruned and the target pruning direction provided in the embodiments of this application;

[0046] Figure 5 This is another structural schematic diagram of the convolution kernel to be pruned and the target pruning direction provided in the embodiments of this application;

[0047] Figure 6 This is another structural schematic diagram of the convolution kernel to be pruned and the target pruning direction provided in the embodiments of this application;

[0048] Figure 7 This is a schematic diagram of the structure of the storage list corresponding to the pruned convolution kernel provided in the embodiments of this application;

[0049] Figure 8This is a schematic diagram of the structure of the storage list corresponding to the convolution kernel to be pruned provided in the embodiments of this application;

[0050] Figure 9 This is a schematic diagram of the structure of the storage list corresponding to the pruned convolution kernel provided in another embodiment;

[0051] Figure 10 This is a schematic diagram of the structure of the input feature map provided in the embodiments of this application;

[0052] Figure 11 This is another schematic flowchart of the neural network processing method provided in the embodiments of this application;

[0053] Figure 12 This is another schematic flowchart of the neural network processing method provided in the embodiments of this application;

[0054] Figure 13 This is a schematic diagram of the structure of the deformed input feature map provided in the embodiments of this application;

[0055] Figure 14 yes Figure 13 A schematic diagram of the abstract structure of the deformed input feature map in the image;

[0056] Figure 15 This is a timeline of parallel computing using a multi-core processor provided in an embodiment of this application;

[0057] Figure 16 This is a schematic diagram of the structure of the neural network processing device provided in the embodiments of this application;

[0058] Figure 17 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0059] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] This application provides a neural network processing method, apparatus, storage medium, and electronic device.

[0061] Please see Figure 1 , Figure 1This is a schematic diagram of a neural network processing system provided in an embodiment of this application. The neural network processing system may include any of the neural network processing devices provided in the embodiments of this application. The neural network processing device may be integrated into electronic devices such as terminals or servers. The terminal may be a smartphone, tablet computer, smart Bluetooth device, laptop computer, or personal computer (PC). The server may be a group server. The group server may be a server device used to manage information processing business based on chat groups. In this embodiment, the server may be a single server or a server cluster composed of multiple servers.

[0062] The aforementioned electronic device can acquire a neural network to be processed, which includes a convolutional kernel to be pruned, and the convolutional kernel to be pruned corresponds to an input feature map; determine a target pruning direction based on the convolutional kernel to be pruned; prune the convolutional kernel parameters of the convolutional kernel to be pruned according to the target pruning direction to obtain a pruned convolutional kernel; perform convolution operation on the input feature map using the pruned convolutional kernel according to the target pruning direction, and replace the input feature map with the output result during the convolution operation to obtain an output feature map.

[0063] The neural network to be processed can be a convolutional neural network and may include at least one convolutional layer. The convolutional kernel to be pruned can be any convolutional kernel in each convolutional layer of the neural network to be processed. In one example, the convolutional kernel to be pruned can be a 2D convolutional kernel, and its size can be 3x3, 5x5, 7x7, etc. The input feature map can be the output feature map of the layer above the layer containing the convolutional kernel to be pruned, or it can be a feature map that has not been processed by the convolutional neural network. The target pruning direction can be one of the four directions forming a cross shape in the 2D convolutional kernel.

[0064] Specifically, such as Figure 1As shown, the aforementioned neural network processing system may include a terminal and a server communicatively connected to the terminal. When a user wants to prune the neural network to be processed, they can send a pruning request to the server through their terminal. The server receives the pruning request, obtains the neural network to be processed based on the request, determines the convolutional kernel to be pruned from the neural network, and determines the target pruning direction based on the kernel. Then, the server prunes the kernel parameters of the kernel to be pruned according to the target pruning direction to obtain the pruned convolutional kernel. Further, after completing the pruning operation on the neural network, the server performs a convolution operation on the input feature map using the pruned convolutional kernel according to the target pruning direction, and replaces the input feature map with the output result during the convolution operation to obtain the output feature map.

[0065] like Figure 2 As shown, Figure 2 This is a flowchart illustrating the neural network processing method provided in an embodiment of this application. The specific flow of the neural network processing method can be as follows:

[0066] S101. Obtain the neural network to be processed, which includes the convolutional kernel to be pruned and the convolutional kernel to be pruned corresponds to the input feature map.

[0067] In this embodiment, the neural network to be processed may be a convolutional neural network including one or more convolutional layers, and may include at least one convolutional layer that needs to be pruned. This at least one convolutional layer that needs to be pruned may be some or all of the convolutional layers in the neural network to be processed. The convolutional kernel to be pruned may be each convolutional kernel in each convolutional layer of the neural network to be processed that needs pruning. For each convolutional kernel to be pruned, the input feature map corresponding to the kernel may be the output feature map of the layer preceding the convolutional layer containing the kernel, or it may be a feature map that has not been processed by the neural network to be processed.

[0068] Specifically, both the convolutional kernel to be pruned and the input feature map can be represented as a two-dimensional matrix. In one example, the convolutional kernel to be pruned can be a two-dimensional (2D) convolutional kernel, and its size can be 3x3, 5x5, 7x7, etc. Taking a 3x3 2D convolutional kernel as an example, this 3x3 2D convolutional kernel can be represented as follows: Figure 3 The diagram shows a 3x3 array of convolutional kernel blocks, where one convolutional kernel block in the array corresponds to one of the convolutional kernel parameters to be pruned.

[0069] S102. Determine the target pruning direction for the convolution kernel to be pruned.

[0070] Specifically, for each convolution kernel to be pruned, the target pruning direction can be, but is not limited to, one of the four directions forming a cross shape within that convolution kernel. In some examples, such as Figures 3 to 6 As shown, the target pruning direction C can be one of the following: from bottom to top, from left to right, from top to bottom, or from right to left.

[0071] S103. Prune the convolution kernel parameters of the convolution kernel to be pruned according to the target pruning direction to obtain the pruned convolution kernel.

[0072] Specifically, for each convolutional kernel to be pruned, the neural network processing device can determine multiple convolutional kernel parameters corresponding to the target pruning direction from the convolutional kernel to be pruned as convolutional kernel parameters that need to be retained, and set the other convolutional kernel parameters in the convolutional kernel to be pruned except for the convolutional kernel parameters that need to be retained to zero, so as to obtain the corresponding pruned convolutional kernel.

[0073] Specifically, such as Figure 3 As shown, when the target pruning direction is from bottom to top, the convolution kernel parameters to be retained in the convolution kernel to be pruned that correspond to the target pruning direction can be at least partially located within the isosceles triangle formed by connecting the lower boundary of the convolution kernel to be pruned and the center point of the convolution kernel to be pruned (for example, Figure 3 (The four convolution kernel parameters are marked in dark).

[0074] like Figure 4 As shown, when the target pruning direction is from left to right, the kernel parameters to be retained in the convolution kernel to be pruned that correspond to the target pruning direction can be kernel parameters that are at least partially located within the isosceles triangle formed by connecting the left boundary of the convolution kernel to be pruned and the center point of the convolution kernel to be pruned (e.g., Figure 4 (The four convolution kernel parameters are marked in dark).

[0075] like Figure 5 As shown, when the target pruning direction is from top to bottom, the kernel parameters to be retained in the convolution kernel to be pruned that correspond to the target pruning direction can be kernel parameters that are at least partially located within the isosceles triangle formed by connecting the upper boundary of the convolution kernel to be pruned and the center point of the convolution kernel to be pruned (e.g., Figure 5 (The four convolution kernel parameters are marked in dark).

[0076] like Figure 6As shown, when the target pruning direction is from right to left, the kernel parameters to be retained in the convolution kernel to be pruned that correspond to the target pruning direction can be kernel parameters that are at least partially located within the isosceles triangle formed by connecting the right boundary of the convolution kernel to be pruned and the center point of the convolution kernel to be pruned (e.g., Figure 6 (The four convolution kernel parameters are marked in dark).

[0077] In some specific embodiments, S102 may specifically include:

[0078] S1021. Determine multiple pruning directions for the convolution kernel to be pruned.

[0079] Specifically, for each convolution kernel to be pruned, the multiple pruning directions of the convolution kernel to be pruned can correspond one-to-one with the four boundaries of the convolution kernel to be pruned, and can be specifically the four directions from the center point of the convolution kernel to the center points of the four boundaries of the convolution kernel to be pruned (that is, the bottom-to-top direction, the left-to-right direction, the top-to-bottom direction, and the right-to-left direction).

[0080] S1022. Determine the parameters of multiple target convolution kernels corresponding to each pruning direction from the convolution kernels to be pruned.

[0081] Specifically, for each convolution kernel to be pruned, the target convolution kernel parameter corresponding to each pruning direction can be a convolution kernel parameter that is at least partially located within an isosceles triangle formed by connecting the boundary of the convolution kernel to be pruned corresponding to each pruning direction and the center point of the convolution kernel to be pruned.

[0082] S1023. Sum the absolute values ​​of the multiple target convolution kernel parameters corresponding to each pruning direction to obtain the corresponding sum.

[0083] Specifically, for each convolution kernel to be pruned, since the multiple target convolution kernel parameters corresponding to each pruning direction in the convolution kernel to be pruned all include the convolution kernel parameters at the center point of the convolution kernel to be pruned, when calculating the sum value corresponding to each pruning direction of the convolution kernel to be pruned, we can simply sum the absolute values ​​of the other target convolution kernel parameters (excluding the convolution kernel parameters at the center point of the convolution kernel to be pruned) among the multiple target convolution kernel parameters corresponding to each pruning direction of the convolution kernel to be pruned, and thus obtain the sum value corresponding to each pruning direction of the convolution kernel to be pruned.

[0084] For example, with Figures 3 to 6Taking the convolution kernel to be pruned as an example, the kernel parameter at the center point of the convolution kernel to be pruned is -0.1. In the embodiments where the multiple pruning directions are from bottom to top, from left to right, from top to bottom, and from right to left, the multiple target convolution kernel parameters corresponding to the convolution kernel to be pruned in the bottom-to-top direction can be -0.1, 0.5, -0.5, and -0.2; the multiple target convolution kernel parameters corresponding to the convolution kernel to be pruned in the left-to-right direction can be -0.1, 0.2, 0.3, and 0.5; the multiple target convolution kernel parameters corresponding to the convolution kernel to be pruned in the top-to-bottom direction can be -0.1, 0.2, 0.8, and -0.4; and the multiple target convolution kernel parameters corresponding to the convolution kernel to be pruned in the right-to-left direction can be -0.1, -0.4, 1.2, and -0.2.

[0085] Correspondingly, the sum value from bottom to top can be 1.2, the sum value from left to right can be 1, the sum value from top to bottom can be 1.4, and the sum value from right to left can be 1.6.

[0086] S1024. Determine the sampling probability corresponding to each pruning direction based on the sum of values ​​corresponding to each pruning direction.

[0087] Specifically, for each convolutional kernel to be pruned, the aforementioned neural network processing device can use the Softmax function to normalize the sum of values ​​corresponding to each pruning direction of the convolutional kernel to be pruned, thereby obtaining the sampling probability corresponding to each pruning direction of the convolutional kernel to be pruned.

[0088] The mathematical expression for Softmax can be:

[0089] .

[0090] Continuing from the previous example, for Figures 3 to 6 In the embodiments where the pruning directions are bottom-up, left-to-right, top-down, and right-to-left, the sampling probabilities corresponding to the bottom-up, left-to-right, top-down, and right-to-left directions can be 0.2207, 0.1807, 0.2695, and 0.3292, respectively.

[0091] S1025. Determine the target pruning direction from multiple pruning directions based on the sampling probability.

[0092] Specifically, for each convolutional kernel to be pruned, the aforementioned neural network processing device can randomly select one pruning direction from multiple pruning directions of the convolutional kernel to be pruned as the target pruning direction of the convolutional kernel to be pruned, based on the sampling probability corresponding to each pruning direction of the convolutional kernel to be pruned.

[0093] Furthermore, it is understandable that for each convolution kernel to be pruned, the greater the sampling probability corresponding to each pruning direction of the convolution kernel to be pruned, the more favorable it is for that pruning direction to be selected as the aforementioned target pruning direction.

[0094] Accordingly, S103 may specifically include: setting all convolution kernel parameters in the convolution kernel to be pruned, except for the target convolution kernel parameters corresponding to the target pruning direction, to zero. That is, for each convolution kernel to be pruned, after determining the target pruning direction from the multiple pruning directions of the convolution kernel to be pruned, the neural network processing device can prune all convolution kernel parameters in the convolution kernel to be pruned, except for the multiple target convolution kernel parameters corresponding to the target pruning direction of the convolution kernel to be pruned, to obtain the corresponding pruned convolution kernel.

[0095] In some embodiments, after obtaining the pruned convolutional kernel, when storing the pruned convolutional kernel, only the kernel parameters retained in the pruned convolutional kernel can be stored, without storing the kernel parameters that were pruned, thereby reducing memory overhead. Specifically, to store the pruned convolutional kernel, the neural network processing method may further include:

[0096] Step A. Determine the order of the parameters of each convolution kernel to be retained in the convolution kernel after pruning, based on the target pruning direction.

[0097] Specifically, for each pruned convolutional kernel, when the target pruning direction of the pruned convolutional kernel is known, the positions of each target convolutional kernel parameter (i.e., the retained convolutional kernel parameter) corresponding to its target pruning direction in the pruned convolutional kernel are also known. Furthermore, the aforementioned neural network processing device can sort the retained convolutional kernel parameters in the pruned convolutional kernel according to a first preset order and a second preset order to obtain the arrangement order of the retained convolutional kernel parameters in the pruned convolutional kernel.

[0098] The first preset order can be an ascending order of row numbers, and the second preset order can be an ascending order of column numbers. Furthermore, in one example, for each pruned convolutional kernel, for any two retained convolutional kernel parameters located in the same row, the retained convolutional kernel parameter with the smaller column number is arranged before the retained convolutional kernel parameter with the larger column number. Additionally, for any two retained convolutional kernel parameters located in the same column, the retained convolutional kernel parameter with the smaller row number is arranged before the retained convolutional kernel parameter with the larger row number.

[0099] For example, with Figure 3 Taking the convolution kernel to be pruned and the target pruning direction as an example... Figure 3 The target pruning direction is from bottom to top, and this target pruning direction is in Figure 3 The multiple retained convolution kernel parameters in the convolution kernel to be pruned can be -0.1, 0.5, -0.5, and -0.2. After sorting the retained convolution kernel parameters in the pruned convolution kernel corresponding to the convolution kernel to be pruned according to the first preset order and the second preset order, the arrangement order of the retained convolution kernel parameters in the pruned convolution kernel can be 0.5→-0.5→-0.2→-0.1.

[0100] Step B. Store the parameters of each convolution kernel retained in the pruned convolution kernel according to the target pruning direction and arrangement order.

[0101] Specifically, for each post-pruning convolution kernel, such as Figure 7 As shown, the aforementioned neural network processing device can store the class symbol of the target pruning direction corresponding to the pruned convolutional kernel and the various convolutional kernel parameters retained in the pruned convolutional kernel in a list manner according to the arrangement order of each convolutional kernel parameter retained in the pruned convolutional kernel, thereby obtaining the corresponding storage list.

[0102] It is understandable that for each convolutional kernel to be pruned, when the kernel has multiple pruning directions and the target pruning direction is one of these directions, each pruning direction can correspond to a classifier. This classifier distinguishes each pruning direction from the others; that is, different pruning directions of the kernel have different classifiers. In one example, the classifier for each pruning direction can be represented by one byte.

[0103] In some examples, for each pruned convolution kernel, such as Figure 7As shown, the earlier the sequence number of each retained convolution kernel parameter in the above arrangement order, the earlier the storage position of each retained convolution kernel parameter in the storage list corresponding to the pruned convolution kernel can be.

[0104] Furthermore, it should be noted that compared to Figure 8 The parameter storage method shown is typical for convolutional neural networks or structured pruned convolutional neural networks. Because neural networks are structured, an index value can be used to find the address of the corresponding parameter storage block; and, compared to Figure 9 The general method for saving model parameters after pruning a fine-grained model, as shown, requires saving the index position of each parameter during parameter saving due to the irregularity of the sparsified model parameters, so that the corresponding position of the parameter can be found when the model is loaded. This embodiment prunes the convolution kernels in the neural network according to a specified shape and provides a method for storing the pruned convolution kernels obtained in this embodiment. This not only greatly reduces the number of parameters in the neural network model, but also reduces the amount of additional location information required for parameter storage, which is more conducive to saving memory.

[0105] In the above embodiments, the number of convolution kernels to be pruned can be one or more, and when the number of convolution kernels to be pruned is multiple, the above S101, S102, S103 and S104 can be executed for each of the multiple convolution kernels to be pruned to obtain the pruned convolution kernel corresponding to each convolution kernel to be pruned.

[0106] Specifically, in order to determine the convolutional kernels to be pruned from the neural network to be processed, the neural network processing device can determine one or more convolutional layers to be pruned from the neural network to be processed. Furthermore, each convolutional kernel included in each convolutional layer to be pruned can be a convolutional kernel to be pruned.

[0107] In embodiments where there are multiple convolutional layers to be pruned, a sequence of layers to be pruned (e.g., [convolutional layer 1, convolutional layer 2, ..., convolutional layer N]) can be obtained by ranking the convolutional layers to be pruned in the neural network to be processed from shallow to deep. Then, the neural network processing device can prune each convolutional layer (e.g., convolutional layer 1) in the sequence of layers to be pruned sequentially. Specifically, pruning each convolutional layer in the sequence of layers to be pruned can include: sequentially executing S101, S102, S103, and S104 for each convolutional kernel (i.e., the convolutional kernel to be pruned) in the convolutional layer to be pruned according to a preset processing order, to obtain the pruned convolutional kernel corresponding to each convolutional kernel to be pruned in the convolutional layer to be pruned.

[0108] In some specific embodiments, the above-mentioned neural network processing method may further include: for each pruned convolutional kernel obtained, updating the corresponding convolutional kernel in the neural network to be processed using the currently obtained pruned convolutional kernel, and retraining the updated neural network to be processed until convergence, thereby enabling the updating of the convolutional kernel parameters retained in the currently obtained pruned convolutional kernel, so as to ensure the accuracy of the output results obtained by subsequent convolution operations using the pruned convolutional kernel.

[0109] Furthermore, in specific implementation, the initial neural network to be processed can be a neural network model obtained after model training on a specific task. Moreover, the aforementioned retraining of the updated neural network to be processed can specifically include: retraining the updated neural network to be processed using the same method as the method used to train the initial neural network to be processed.

[0110] S104. Based on the target pruning direction, perform convolution operation on the input feature map using the pruned convolution kernel, and replace the input feature map with the output result during the convolution operation to obtain the output feature map.

[0111] In this embodiment, after pruning all the convolutional kernels to be pruned in the neural network to be processed, for each pruned convolutional kernel obtained, the neural network processing device can perform convolution operations on the corresponding input feature map using the pruned convolutional kernel according to the target pruning direction corresponding to the pruned convolutional kernel, and replace the input feature map with the output result during the convolution operation to obtain the corresponding output feature map. Furthermore, the input feature map and output feature map corresponding to the same pruned convolutional kernel can have the same size, for example, H×W, where H represents the height of the input feature map and the output feature map, and W represents the width of the input feature map and the output feature map.

[0112] Specifically, such as Figure 10 As shown, the input feature map described above can consist of several pixels, and these pixels can be arranged in rows and columns. In one example, such as... Figure 10 As shown, the input feature map can be a 6-row, 9-column pixel matrix.

[0113] Furthermore, for each pruned convolutional kernel, the target pruning direction corresponding to the pruned convolutional kernel can indicate the pixel position in the input feature map corresponding to the pruned convolutional kernel when calculating the output result of any pixel in the input feature map corresponding to the pruned convolutional kernel.

[0114] Among them, Figure 10In the above-mentioned input feature map, an arrow pointing from one pixel to another can be used to indicate that the input result of the pixel located at the tip of the arrow depends on the pixel located at the non-tip of the arrow.

[0115] For example, for each pruned convolutional kernel, if the target pruning direction corresponding to the pruned convolutional kernel is from bottom to top, then in the input feature map corresponding to the pruned convolutional kernel, the output of each pixel will only depend on the pixels in the row below it, and not on the pixels in the row above it. Therefore, when performing convolution operations on the input feature map using pruned convolutional kernels, convolution operations can be performed on each part of the input feature map sequentially from top to bottom. Furthermore, after completing the convolution operation on each part of the input feature map, the output of that part can be used to replace that part in the input feature map without affecting the output of the remaining parts. This allows convolution operations to be performed in the memory of the input feature map without using additional output space. This helps reduce the requirements for high-speed cache in computing hardware, making it possible to deploy deep learning models on lower-end computing hardware.

[0116] In some embodiments, such as Figure 11 As shown, the above S104 can specifically include:

[0117] S1-1. Determine the first convolution order and the second convolution order based on the target pruning direction.

[0118] The first convolution order can be used to instruct convolution operations to be performed row-by-row on each row of pixels in the corresponding input feature map, or it can be used to instruct convolution operations to be performed column-by-column on each column of pixels in the corresponding input feature map. Specifically, when the first convolution order instructs convolution operations to be performed row-by-row on each row of pixels in the corresponding input feature map, the first convolution order can be the order in which convolution operations are performed on each row of pixels in the corresponding input feature map, and the second convolution order can be the order in which convolution operations are performed on each pixel within each row of pixels in the corresponding input feature map. Similarly, when the first convolution order instructs convolution operations to be performed column-by-column on each column of pixels in the corresponding input feature map, the first convolution order can be the order in which convolution operations are performed on each column of pixels in the corresponding input feature map, and the second convolution order can be the order in which convolution operations are performed on each pixel within each column of pixels in the corresponding input feature map.

[0119] In one example, if the target pruning direction is from bottom to top, the first convolution order can be used to instruct convolution operations to be performed row by row on each row of pixels in the corresponding input feature map, and the first and second convolution orders are respectively in ascending order of row number and ascending order of column number; if the target pruning direction is from top to bottom, the first convolution order can be used to instruct convolution operations to be performed row by row on each row of pixels in the corresponding input feature map, and the first and second convolution orders are respectively in descending order of row number and descending order of column number. The order of convolution is as follows: If the target pruning direction is from left to right, the first convolution order can be used to instruct convolution operations to be performed on each column of pixels in the corresponding input feature map column by column, and the first convolution order and the second convolution order are respectively in descending order of column number and ascending order of row number; if the target pruning direction is from right to left, the first convolution order can be used to instruct convolution operations to be performed on each row of pixels in the corresponding input feature map row by row, and the first convolution order and the second convolution order are respectively in ascending order of column number and descending order of row number.

[0120] S1-2. Following the first and second convolution order, each pixel of the input feature image is convolved using the pruned convolution kernel. During the convolution operation, the pixel value of each pixel is replaced using the output result corresponding to each pixel to obtain the output feature map.

[0121] It is understandable that, for each pruned convolution kernel, when performing a convolution operation on any pixel in the corresponding input feature map using that pruned convolution kernel, the input result corresponding to that pixel is only related to the pixels that are located after that pixel in the first convolution order and the pixels that are located after that pixel in the second convolution order, and is not related to the pixels that are located before that pixel in the first convolution order and the pixels that are located before that pixel in the second convolution order.

[0122] Furthermore, for each pruned convolution kernel, when performing convolution operations on each pixel in the corresponding input feature image according to the first convolution order and the second convolution order, the pixel value of the pixel in the above-mentioned input feature map can be replaced by the output result corresponding to each pixel after obtaining the output result of each pixel. Thus, after completing the convolution operation on all pixels of the above-mentioned input feature image, the input feature map obtained by replacing the pixel values ​​of all pixels in the corresponding input feature map with the corresponding output result can be obtained, and the convolution operation can be completed in the memory of the above-mentioned input feature map without using additional output space.

[0123] In other embodiments, such as Figure 12 As shown, the above S104 can specifically include:

[0124] S2-1. Determine the convolution direction based on the target pruning direction.

[0125] In this embodiment, for each pruned convolutional kernel, the convolution direction corresponding to the pruned convolutional kernel is not parallel to the target pruning direction corresponding to the pruned convolutional kernel. In one example, the angle between the convolution direction corresponding to the pruned convolutional kernel and the target pruning direction corresponding to the pruned convolutional kernel can be 45 degrees or 135 degrees. Furthermore, it is understood that when performing convolution operations on the corresponding deformed input feature map using the pruned convolutional kernel, convolution operations can be performed sequentially on each part of the deformed input feature map along the aforementioned convolution direction. After completing the convolution operation on each part of the input feature map, the output result of that part can be used to replace that part in the deformed input feature map without affecting the output result of the remaining parts of the deformed input feature map. This allows the convolution operation to be completed in the memory of the deformed input feature map without using additional output space.

[0126] Specifically, for each pruned convolution kernel, if the target pruning direction corresponding to the pruned convolution kernel is from bottom to top, then the convolution direction corresponding to the pruned convolution kernel can be from the upper right to the lower left; if the target pruning direction corresponding to the pruned convolution kernel is from left to right, then the convolution direction corresponding to the pruned convolution kernel can be from the lower right to the upper left; if the target pruning direction corresponding to the pruned convolution kernel is from top to bottom, then the convolution direction corresponding to the pruned convolution kernel can be from the lower left to the upper right; if the target pruning direction corresponding to the pruned convolution kernel is from right to left, then the convolution direction corresponding to the pruned convolution kernel can be from the upper left to the lower right.

[0127] S2-2. Perform an affine transformation on the input feature map according to the convolution direction to obtain a deformed input feature map.

[0128] Specifically, for each pruned convolutional kernel, the aforementioned neural network processing device can perform an affine transformation on the pixel coordinates of the input feature map corresponding to the pruned convolutional kernel, based on the convolution direction corresponding to the kernel, using a scheduling algorithm in the polyhedral model, to obtain the transformed pixel coordinates of the deformed input feature map. In one example, the aforementioned scheduling algorithm can be specifically the Pluto algorithm.

[0129] In one specific embodiment, for each pruned convolution kernel, when the target pruning direction corresponding to the pruned convolution kernel is from bottom to top, such as Figure 10As shown, the pixel coordinates of the input feature map corresponding to the pruned convolutional kernel can be represented as (h, w), and the aforementioned neural network processing device can map the pixel coordinates of the input feature map corresponding to the pruned convolutional kernel from (h, w) to (h, w + h), thus obtaining the pixel coordinates of the input feature map corresponding to the pruned convolutional kernel as shown in the figure. Figure 13 The deformed input feature map is shown.

[0130] Specifically, for each pruned convolutional kernel, in the deformed input feature map corresponding to the pruned convolutional kernel, the output of each pixel will only depend on the pixels in front of it in the corresponding convolutional direction, and will not depend on the pixels behind it in the corresponding convolutional direction.

[0131] For example, for each post-pruning convolution kernel, such as Figure 13 As shown, if the target pruning direction corresponding to the pruned convolution kernel is from bottom to top, then the convolution direction corresponding to the pruned convolution kernel can be from right-top-bottom to left-bottom. In the deformed input feature map corresponding to the pruned convolution kernel, the output of each pixel will only depend on the pixel located to its lower left, and will not depend on the pixel located to its upper right.

[0132] Therefore, by using pruned convolutional kernels to perform convolution operations on the deformed input feature map, convolution operations can be performed sequentially on each part of the input feature map along the convolution direction. After each part of the input feature map is convolved, the output of that part can be used to replace that part in the input feature map without affecting the output of the remaining parts of the input feature map. This allows convolution operations to be performed in the memory of the input feature map without using additional output space. This helps reduce the requirements for high-speed cache in computing hardware, making it possible to deploy deep learning models on lower-end computing hardware.

[0133] S2-3. Divide the deformed input feature map into multiple feature map blocks along the horizontal and vertical axes.

[0134] In this embodiment, the horizontal direction of the pixel coordinates (or pixel transformation coordinates) in the deformed input feature map is the same as the horizontal direction of the pixel coordinates in the input feature map; for example, both can be from bottom to top. Similarly, the vertical direction of the pixel transformation coordinates in the deformed input feature map is the same as the vertical direction of the pixel coordinates in the input feature map; for example, both can be from left to right.

[0135] Specifically, the aforementioned neural network processing device can divide the deformed input feature map into a feature map block array with a preset number of rows and columns along the horizontal and vertical axes. Figure 13Taking the deformed input feature map shown as an example, this deformed input feature map can be divided into a 2x5 feature map block array. Furthermore, after dividing the deformed input feature map into multiple feature map blocks along the horizontal and vertical axes, each feature map block in the deformed input feature map can be abstracted into a corresponding coordinate point without changing the positional relationship between the various feature map blocks. For example, Figure 13 The eight feature map blocks of the deformed input feature map can be abstracted into eight points in another coordinate system, namely (2,2), (3,2), (4,2), (5,2), (1,1), (2,1), (3,2) and (4,2).

[0136] S2-4. Divide multiple feature maps into multiple groups based on the convolution direction.

[0137] Specifically, the aforementioned neural network processing device can group one or more feature map blocks whose center points are located on the same straight line perpendicular to the aforementioned convolution direction, thereby obtaining multiple groups of feature map blocks.

[0138] For example, with Figure 14 Taking the deformed input feature map shown as an example, the convolution direction corresponding to this deformed input feature map is from the upper right to the lower left, and is parallel to the vertical direction (that is, Figure 14 The i-axis direction in the image is at a 135-degree angle to the other two, and the multiple feature maps included in the deformed input feature map are divided into 6 groups. The first group consists of (5,2), the second group consists of (4,2), the third group consists of (3,2) and (4,1), the fourth group consists of (2,2) and (3,1), the fifth group consists of (2,1), and the sixth group consists of (1,1).

[0139] S2-5. Based on the convolution direction, the pruned convolution kernel performs parallel convolution operations on each feature map block in each group. During the parallel convolution operation, the pixel values ​​of each feature map block in each group are replaced with the output results corresponding to each feature map block in each group to obtain the output feature map.

[0140] Specifically, for each pruned convolutional kernel, after dividing the multiple feature maps in the pruned kernel into multiple groups according to the corresponding convolution direction, the number of feature map blocks in each group can be one or more. Furthermore, the output results corresponding to the convolution operation of each feature map block in the same group do not affect each other.

[0141] It is understandable that, for each pruned convolution kernel, when performing convolution operations on each group of feature map blocks in the corresponding input feature map using the pruned convolution kernel, the input result corresponding to each group of feature map blocks is only related to the group of feature map blocks that are in front of and adjacent to it in the first convolution order, and is not related to the remaining group of feature map blocks.

[0142] Furthermore, since the output results of convolution operations on each feature map block within the same group do not affect each other, when performing convolution operations on each feature map block in each group, multiple processor cores can perform parallel convolution operations on each feature map block in each group to improve the efficiency of convolution operations.

[0143] Specifically, in an embodiment where multiple processor cores perform parallel convolution operations on each feature map block within each group of feature map blocks, the number of processor cores used can be equal to the number of feature map blocks in the feature map block group with the most feature map blocks. Furthermore, the multiple processor cores used can belong to the same processor, and these multiple cores access the same memory address with identical access latency.

[0144] In one example, with Figure 14 Taking the deformed input feature map shown as an example, after dividing the multiple feature maps included in the deformed input feature map into 6 groups (that is, the first group composed of (5,2), the second group composed of (4,2), the third group composed of (3,2) and (4,1), the fourth group composed of (2,2) and (3,1), the fifth group composed of (2,1), and the sixth group composed of (1,1), as shown Figure 15 As shown, parallel convolution operations can be performed on each feature map block within each group of feature map blocks using two processor cores (i.e., processor core P1 and processor core P2). Figure 15 In the table, T represents the time value, P1 and P2 are two different processor cores, and the non-empty values ​​in the table represent the feature map blocks calculated on processor cores P1 / P1 at a certain time.

[0145] As can be seen from the above, the neural network processing method provided in this embodiment obtains a neural network to be processed, which includes a convolutional kernel to be pruned and a corresponding input feature map. Then, a target pruning direction is determined based on the convolutional kernel to be pruned, and the convolutional kernel parameters of the convolutional kernel to be pruned are pruned according to the target pruning direction to obtain a pruned convolutional kernel. After that, the input feature map is convolved using the pruned convolutional kernel according to the target pruning direction, and the input feature map is replaced with the output result during the convolution operation to obtain an output feature map. This not only reduces the number of parameters in the neural network model, but also allows the input and output in the model convolution operation to reuse the same memory space, which greatly reduces the amount of computation and memory access during model operation.

[0146] Based on the methods described in the above embodiments, this embodiment will further describe the method from the perspective of a neural network processing device. Please refer to [link / reference]. Figure 16 , Figure 16 The neural network processing apparatus provided in the embodiments of this application is described in detail. The neural network processing apparatus includes: an acquisition module 301, a first determination module 302, a pruning module 303, and a convolution operation module 304, wherein:

[0147] (1) Acquisition module 301

[0148] The acquisition module 301 is used to acquire the neural network to be processed, which includes a convolutional kernel to be pruned and a corresponding input feature map.

[0149] (2) First Determining Module 302

[0150] The first determining module 302 is used to determine the target pruning direction of the convolution kernel to be pruned.

[0151] (3) Pruning module 303

[0152] The pruning module 303 is used to prune the convolution kernel parameters of the convolution kernel to be pruned according to the target pruning direction, so as to obtain the pruned convolution kernel.

[0153] (4) Convolution operation module 304

[0154] The convolution operation module 304 is used to perform convolution operation on the input feature map using the pruned convolution kernel according to the target pruning direction, and to replace the input feature map with the output result during the convolution operation to obtain the output feature map.

[0155] In some embodiments, the convolution operation module 304 may specifically include:

[0156] The first determining unit is used to determine the first convolution order and the second convolution order according to the target pruning direction;

[0157] The first convolution operation unit is used to perform convolution operations on each pixel of the input feature image one by one using the pruned convolution kernel according to the first convolution order and the second convolution order. During the convolution operation, the pixel value of each pixel is replaced by the output result corresponding to each pixel to obtain the output feature map.

[0158] In other embodiments, the convolution operation module 304 described above may specifically include:

[0159] The second determining unit is used to determine the convolution direction based on the target pruning direction;

[0160] The affine transformation unit is used to perform an affine transformation on the input feature map according to the convolution direction to obtain a deformed input feature map.

[0161] The block unit is used to divide the deformed input feature map into multiple feature map blocks along the horizontal and vertical axes.

[0162] Grouping units are used to divide multiple feature maps into multiple groups based on the convolution direction;

[0163] The second convolution operation unit is used to perform parallel convolution operations on each feature map block in each group according to the convolution direction using the pruned convolution kernel. During the parallel convolution operation, the pixel values ​​of each feature map block in each group are replaced with the output results corresponding to each feature map block in each group to obtain the output feature map.

[0164] In one specific embodiment, the above-described affine transformation unit can be specifically used for:

[0165] Based on the convolution direction, the pixel coordinates of the input feature map are subjected to an affine transformation using the scheduling algorithm in the polyhedral model to obtain the transformed pixel coordinates of the deformed input feature map.

[0166] In some embodiments, the first determining module 302 may specifically include:

[0167] The third determining unit is used to determine multiple pruning directions of the convolution kernel to be pruned;

[0168] The fourth determining unit is used to determine the parameters of multiple target convolution kernels corresponding to each pruning direction from the convolution kernels to be pruned;

[0169] The summation unit is used to sum the absolute values ​​of multiple target convolution kernel parameters corresponding to each pruning direction to obtain the corresponding sum value;

[0170] The fifth determining unit is used to determine the sampling probability corresponding to each pruning direction based on the sum value corresponding to each pruning direction.

[0171] The sixth determining unit is used to determine the target pruning direction from multiple pruning directions based on the sampling probability.

[0172] Accordingly, the pruning module 303 described above can be specifically used for:

[0173] Set all convolution kernel parameters in the convolution kernel to be pruned, except for the target convolution kernel parameter corresponding to the target pruning direction, to zero.

[0174] In the above embodiments, the neural network processing device may further include:

[0175] (5) Second acquisition module

[0176] The second determining module is used to determine the order of the parameters of each convolution kernel retained in the convolution kernel after pruning, based on the target pruning direction.

[0177] (6) Storage module

[0178] The storage module is used to store the parameters of each convolution kernel retained in the pruned convolution kernel according to the target pruning direction and arrangement order.

[0179] In practice, each of the above units and modules can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units and modules, please refer to the previous method implementation examples, which will not be repeated here.

[0180] As can be seen from the above, the neural network processing device provided in this embodiment includes an acquisition module for acquiring a neural network to be processed, the neural network to be processed including a convolutional kernel to be pruned, and the convolutional kernel to be pruned corresponding to an input feature map; a first determination module for determining the target pruning direction of the convolutional kernel to be pruned; a pruning module for pruning the convolutional kernel parameters of the convolutional kernel to be pruned according to the target pruning direction to obtain a pruned convolutional kernel; and a convolution operation module for performing convolution operation on the input feature map using the pruned convolutional kernel according to the target pruning direction, and replacing the input feature map with the output result during the convolution operation to obtain an output feature map. This not only reduces the number of parameters in the neural network model, but also allows the input and output in the model convolution operation process to reuse the same memory space, which greatly reduces the amount of computation and memory access during model operation.

[0181] Accordingly, embodiments of this application also provide an electronic device, wherein the electronic device may be a terminal or a server, such as... Figure 17 As shown, it illustrates a structural schematic diagram of the electronic device involved in the embodiments of this application, specifically:

[0182] The electronic device 400 includes a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, and a computer program stored in the memory 402 and executable on the processor. The processor 401 and the memory 402 are electrically connected. Those skilled in the art will understand that the electronic device structure shown in the figures does not constitute a limitation on the electronic device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0183] The processor 401 is the control center of the electronic device 400. It connects various parts of the electronic device 400 through various interfaces and lines. By running or loading software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, it performs various functions of the electronic device 400 and processes data, thereby monitoring the electronic device 400 as a whole.

[0184] In this embodiment, the processor 401 in the electronic device 400 loads the instructions corresponding to the processes of one or more applications into the memory 402 according to the following steps, and the processor 401 runs the applications stored in the memory 402 to realize various functions:

[0185] Obtain the neural network to be processed, which includes the convolutional kernel to be pruned and the convolutional kernel to be pruned corresponds to the input feature map;

[0186] Determine the target pruning direction for the convolution kernel to be pruned;

[0187] Prune the convolution kernel parameters of the convolution kernel to be pruned according to the target pruning direction to obtain the pruned convolution kernel;

[0188] Based on the target pruning direction, the input feature map is convolved using a post-pruned convolution kernel. During the convolution operation, the output result is used to replace the input feature map to obtain the output feature map.

[0189] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0190] Optional, such as Figure 17 As shown, the electronic device 400 also includes: a touch display screen 403, a radio frequency circuit 404, an audio circuit 405, an input unit 406, and a power supply 407. The processor 401 is electrically connected to the touch display screen 403, the radio frequency circuit 404, the audio circuit 405, the input unit 406, and the power supply 407. Those skilled in the art will understand that... Figure 17The electronic device structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0191] The touch display screen 403 can be used to display a graphical user interface (GUI) and receive operation commands generated by the user interacting with the GUI. The touch display screen 403 may include a display panel and a touch panel. The display panel can be used to display information input by the user or information provided to the user, as well as various graphical user interfaces of the electronic device. These graphical user interfaces can be composed of graphics, text, icons, video, and any combination thereof. Optionally, the display panel can be configured using a liquid crystal display (LCD), organic light-emitting diode (OLED), or other similar technologies. The touch panel can be used to collect touch operations performed by the user on or near it (such as operations performed by the user using a finger, stylus, or any suitable object or accessory on or near the touch panel), generate corresponding operation commands, and execute the corresponding program according to the operation commands. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. The touch detection device detects the user's touch location and the signal generated by the touch operation, transmitting the signal to the touch controller. The touch controller receives touch information from the touch detection device, converts it into touch point coordinates, and sends it to the processor 401. It can also receive and execute commands from the processor 401. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it transmits the information to the processor 401 to determine the type of touch event. Subsequently, the processor 401 provides corresponding visual output on the display panel based on the type of touch event. In this embodiment, the touch panel and the display panel can be integrated into the touch display screen 403 to achieve input and output functions. However, in some embodiments, the touch panel and the touch display screen 403 can be implemented as two independent components to achieve input and output functions. That is, the touch display screen 403 can also be used as part of the input unit 406 to achieve input functions.

[0192] In this embodiment of the application, the processor 401 executes a game application to generate a virtual three-dimensional scene on the touch screen 403. The scene includes a graphical user interface (UI) and a second spatial orientation indicator. The second spatial orientation indicator displays a spatial orientation mark corresponding to the target object. The spatial orientation mark is used to indicate the location of the target object.

[0193] The touch display screen 403 can be used to present images of virtual three-dimensional scenes, as well as graphical user interfaces and receive operation commands generated by the user interacting with the graphical user interface.

[0194] The radio frequency circuit 404 can be used to transmit and receive radio frequency signals to establish wireless communication with network devices or other electronic devices, and to transmit and receive signals with network devices or other electronic devices.

[0195] Audio circuitry 405 can be used to provide an audio interface between a user and electronic device 400 via a speaker and microphone. Audio circuitry 405 can convert received audio data into electrical signals and transmit them to the speaker, where the speaker converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuitry 405, converted back into audio data, and then processed by processor 401 before being transmitted via radio frequency circuitry 404 to, for example, another electronic device, or output to memory 402 for further processing. Audio circuitry 405 may also include an earphone jack to provide communication between peripheral headphones and electronic device 400.

[0196] The input unit 406 can be used to receive input numbers, characters, or user characteristic information (such as fingerprints, iris, facial information, etc.), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0197] Power supply 407 is used to supply power to various components of electronic device 400. Optionally, power supply 407 can be logically connected to processor 401 through a power management system, thereby enabling functions such as charging, discharging, and power consumption management through the power management system. Power supply 407 may also include one or more DC or AC power supplies, recharging systems, power fault detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0198] although Figure 17 As not shown in the diagram, the electronic device 400 may also include a camera, sensor, wireless fidelity module, Bluetooth module, etc., which will not be described in detail here.

[0199] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0200] As can be seen from the above, the electronic device provided in this embodiment can not only reduce the number of parameters in the neural network model, but also enable the input and output in the model convolution operation process to reuse the same memory space, which greatly reduces the amount of computation and memory access during the model operation.

[0201] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0202] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of computer programs that can be loaded by a processor to execute steps in any of the neural network processing methods provided in embodiments of this application. For example, the computer program can execute the following steps:

[0203] Obtain the neural network to be processed, which includes the convolutional kernel to be pruned and the convolutional kernel to be pruned corresponds to the input feature map;

[0204] Determine the target pruning direction for the convolution kernel to be pruned;

[0205] Prune the convolution kernel parameters of the convolution kernel to be pruned according to the target pruning direction to obtain the pruned convolution kernel;

[0206] Based on the target pruning direction, the input feature map is convolved using a post-pruned convolution kernel. During the convolution operation, the output result is used to replace the input feature map to obtain the output feature map.

[0207] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0208] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0209] Since the computer program stored in the storage medium can execute the steps of any of the neural network processing methods provided in the embodiments of this application, the beneficial effects that any of the neural network processing methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0210] The foregoing has provided a detailed description of a neural network processing method, apparatus, storage medium, and electronic device provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A neural network processing method, characterized in that, include: Obtain a neural network to be processed, the neural network to be processed includes a convolutional kernel to be pruned, the convolutional kernel to be pruned corresponds to an input feature map; wherein, the input feature map is the output feature map of the layer above the convolutional layer where the convolutional kernel to be pruned is located, or is a feature map that has not been processed by the neural network to be processed; the input feature map is a pixel matrix; Determine the target pruning direction for the convolution kernel to be pruned; The convolution kernel parameters of the convolution kernel to be pruned are pruned according to the target pruning direction to obtain the pruned convolution kernel; According to the target pruning direction, the input feature map is convolved using the pruned convolution kernel, and the output result is used to replace the input feature map during the convolution operation to obtain the output feature map; wherein, the target pruning direction corresponding to the pruned convolution kernel is used to indicate: when calculating the output result of any pixel in the input feature map corresponding to the pruned convolution kernel, the position of the pixel in the input feature map corresponding to the pruned convolution kernel that the pruned convolution kernel depends on for one convolution operation between the pruned convolution kernel and the pixel.

2. The neural network processing method according to claim 1, characterized in that, The step of performing convolution operations on the input feature map using the pruned convolution kernel according to the target pruning direction, and replacing the input feature map with the output result during the convolution operation to obtain the output feature map, specifically includes: The first convolution order and the second convolution order are determined according to the target pruning direction; Following the first convolution order and the second convolution order, the pruned convolution kernels are used to perform convolution operations on each pixel of the input feature image one by one. During the convolution operation, the pixel value of each pixel is replaced by the output result corresponding to each pixel to obtain the output feature map.

3. The neural network processing method according to claim 1, characterized in that, The step of performing convolution operations on the input feature map using the pruned convolution kernel according to the target pruning direction, and replacing the input feature map with the output result during the convolution operation to obtain the output feature map, specifically includes: The convolution direction is determined based on the target pruning direction; Affine transformation is performed on the input feature map according to the convolution direction to obtain a deformed input feature map; The deformed input feature map is divided into multiple feature map blocks along the horizontal and vertical axes; The multiple feature maps are divided into multiple groups based on the convolution direction; According to the convolution direction, the pruned convolution kernel is used to perform parallel convolution operations on each feature map block in each group. During the parallel convolution operation, the pixel values ​​of each feature map block in each group are replaced by the output results corresponding to each feature map block in each group to obtain the output feature map.

4. The neural network processing method according to claim 3, characterized in that, The step of performing an affine transformation on the input feature map according to the convolution direction to obtain a deformed input feature map specifically includes: Based on the convolution direction, the pixel coordinates of the input feature map are subjected to an affine transformation using a scheduling algorithm in a polyhedral model to obtain the transformed pixel coordinates of the deformed input feature map.

5. The neural network processing method according to claim 1, characterized in that, Determining the target pruning direction of the convolution kernel to be pruned specifically includes: Determine multiple pruning directions for the convolution kernel to be pruned; Determine multiple target convolution kernel parameters corresponding to each pruning direction from the convolution kernels to be pruned; The absolute values ​​of the multiple target convolution kernel parameters corresponding to each pruning direction are summed to obtain the corresponding sum value; Based on the sum value corresponding to each pruning direction, determine the sampling probability corresponding to each pruning direction; The target pruning direction is determined from the plurality of pruning directions based on the sampling probability.

6. The neural network processing method according to claim 5, characterized in that, The step of pruning the convolution kernel parameters of the convolution kernel to be pruned according to the target pruning direction to obtain the pruned convolution kernel specifically includes: Set all convolution kernel parameters in the convolution kernel to be pruned, except for the target convolution kernel parameter corresponding to the target pruning direction, to zero.

7. The neural network processing method according to claim 1, characterized in that, The neural network processing method further includes: The order in which the parameters of each convolution kernel are retained in the pruned convolution kernel is determined according to the target pruning direction. The parameters of each convolution kernel retained in the pruned convolution kernel are stored according to the target pruning direction and the arrangement order.

8. A neural network processing device, characterized in that, include: An acquisition module is used to acquire a neural network to be processed, wherein the neural network to be processed includes a convolutional kernel to be pruned, and the convolutional kernel to be pruned corresponds to an input feature map; wherein the input feature map is the output feature map of the layer above the convolutional layer where the convolutional kernel to be pruned is located, or is a feature map that has not been processed by the neural network to be processed; the input feature map is a pixel matrix; The determination module is used to determine the target pruning direction of the convolution kernel to be pruned; The pruning module is used to prune the convolution kernel parameters of the convolution kernel to be pruned according to the target pruning direction, so as to obtain the pruned convolution kernel; The convolution operation module is used to perform convolution operations on the input feature map using the pruned convolution kernel according to the target pruning direction, and to replace the input feature map with the output result during the convolution operation to obtain the output feature map; wherein, the target pruning direction corresponding to the pruned convolution kernel is used to indicate: when calculating the output result of any pixel in the input feature map corresponding to the pruned convolution kernel, the position of the pixel in the input feature map corresponding to the pruned convolution kernel that the pruned convolution kernel depends on for one convolution operation between the pruned convolution kernel and the pixel.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program adapted for loading by a processor to perform the steps of the neural network processing method as described in any one of claims 1-7.

10. An electronic device, characterized in that, The electronic device includes a memory and a processor. The memory stores a computer program, and the processor executes the steps of the neural network processing method as described in any one of claims 1-7 by calling the computer program stored in the memory.

Images