Convolution data processing method and device, equipment and storage medium
By combining and cache the input feature maps and convolution kernel elements of the convolution neural network, the resource waste problem caused by data mismatch in convolutional calculations is solved, and more efficient computing resource utilization and computing efficiency are achieved.
Patent Information
- Application Number
- CN202510213568.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-02-25
- Publication Date
- 2025-06-13
AI Technical Summary
When computing, the existing convolutional neural networks need to read in a large amount of garbage data and perform redundant calculations due to the mismatch of the size of the input channel, output channel and feature map, resulting in wasted computing resources.
By combining the pixels and convolution kernel elements of the input feature map, cache them according to the parallelism of the input channel, reducing the storage and calculation of redundant data. The specific methods include: reading a preset number of pixels and convolution kernel elements, merging them into merged data, writing to cache units, and performing convolution calculations during calculation.
It effectively reduces the read in invalid data, improves the utilization rate of cache resources, and improves the efficiency of convolutional calculation.
Smart Images
Figure CN120146107A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and particularly relates to a convolutional data processing method, apparatus, device, and storage medium. Background Art
[0002] CNN (Convolutional Neural Network) is a deep learning model widely used in fields such as image recognition, speech recognition, and natural language processing. The convolutional layer is the core layer of CNN, and its main operation is the multiplication and addition calculation between image data and convolutional kernels. The computational amount of the convolutional layer accounts for a large proportion of the total computational amount of CNN. Therefore, the performance bottleneck of CNN network computing is generally also in convolutional computing.
[0003] However, the input channels, output channels, and sizes of feature maps of different convolutional layers are not the same, while the hardware structure for realizing parallel acceleration is often relatively fixed. Therefore, when the neural network performs calculations, to meet the alignment requirements of parallel computing, a large amount of garbage data needs to be read in, a large amount of redundant calculations need to be done, and a large amount of invalid data needs to be output, thus causing waste of neural network computing resources. Summary of the Invention
[0004] Embodiments of this application provide a convolutional data processing method, apparatus, device, and storage medium, which can reduce the number of invalid data read in, thereby saving neural network computing resources. The technical solution is as follows:
[0005] According to the first aspect of the embodiments of this application, a convolutional data processing method is provided. The method includes:
[0006] Read a first preset number of pixels and a second preset number of convolutional kernel elements; the first preset number of pixels is a part of the input feature map; the second preset number of convolutional kernel elements is a part of a convolutional kernel;
[0007] When a first parameter is less than the input channel parallelism, merge the first preset number of pixels to obtain first merged data; write the first merged data into a first cache unit; the first parameter indicates the product of the first preset number and the number of input channels;
[0008] When a second parameter is less than the input channel parallelism, merge the second preset number of convolutional kernel elements to obtain second merged data; write the second merged data into a second cache unit; the second parameter indicates the product of the second preset number and the number of input channels;
[0009] For a pixel in the first cache unit, read the pixel from the first cache unit and read the second merged data from the second cache unit;
[0010] Obtain the second preset number of the pixels, and perform a convolution calculation on the second preset number of the pixels and the second merged data to obtain a convolution value; the convolution value is a part of the output feature map.
[0011] In a possible implementation manner, the writing the first merged data into the first buffer unit includes:
[0012] When the first parameter is less than the input channel parallelism, write first padding data into the first buffer unit; the number of the padding channel data included in the first padding data and the number of the pixel channel data included in the first merged data sum up to the input channel parallelism; each of the pixels includes a plurality of the pixel channel data; the first padding data includes a plurality of the padding channel data.
[0013] In a possible implementation manner, the method further includes:
[0014] When the first parameter is equal to the input channel parallelism, the number of the pixel channel data included in the first merged data is equal to the input channel parallelism.
[0015] In a possible implementation manner, the method further includes:
[0016] When the first parameter is greater than the input channel parallelism, divide the first preset number of the pixels into multiple groups; each group includes a sub-preset number of the pixels;
[0017] Merge the sub-preset number of the pixels in each group to obtain a plurality of sub-merged data;
[0018] Write each of the sub-merged data and a sub-padding data into the first buffer unit; the number of the pixel channel data included in each of the sub-merged data and the number of the padding channel data included in a sub-padding data sum up to the input channel parallelism.
[0019] In a possible implementation manner, the writing the second merged data into the second buffer unit includes:
[0020] When the second parameter is less than the input channel parallelism, write second padding data into the second buffer unit; the number of the padding channel data included in the second padding data and the number of the convolution kernel channel data included in the second merged data sum up to the input channel parallelism; each of the convolution kernels includes a plurality of the convolution kernel channel data.
[0021] In a possible implementation, the method further includes:
[0022] When the second parameter is equal to the input channel parallelism, the number of the convolution kernel channel data included in the second merged data is equal to the input channel parallelism.
[0023] In a possible implementation, the second preset number indicates the width of the convolution kernel.
[0024] In a possible implementation, each pixel includes 3 pieces of pixel channel data; each convolution kernel element includes 3 pieces of convolution kernel channel data.
[0025] In a possible implementation, the method further includes:
[0026] Read a first number of the first merged data and the first padding data from the first cache unit; the first number is (pixel parallelism / first preset number); and read k rows of the second merged data and the second padding data from the second cache unit; each row of the second merged data includes a second number of the second merged data and the second padding data; the second number indicates the output channel parallelism; k is an integer greater than or equal to 1 and less than or equal to the height; the height is the height of the convolution kernel;
[0027] Copy the i-th pixel in the first merged data to obtain the i-th copied data; the i-th copied data includes the second preset number of the i-th pixels; i is an integer greater than or equal to 1 and less than or equal to the first preset number;
[0028] Use the i-th copied data and the third padding data as the i-th arranged data; the sum of the number of the pixel channel data included in the i-th copied data and the number of the padding channel data included in the third padding data is equal to the input channel parallelism;
[0029] Perform convolution calculation on the i-th arranged data and each of the second merged data and the second padding data in the k-th row to obtain the output feature map.
[0030] According to a second aspect of the embodiments of the present application, there is provided a convolution data processing device, the device includes:
[0031] A first reading module, configured to read a first preset number of pixels and a second preset number of convolution kernel elements; the first preset number of pixels is a part of an input feature map; the second preset number of convolution kernel elements is a part of a convolution kernel;
[0032] The first shaping module is configured to, when a first parameter is less than the input channel parallelism, merge the first preset number of the pixels to obtain first merged data; write the first merged data into a first cache unit; the first parameter indicates the product of the first preset number and the number of input channels;
[0033] The second shaping module is configured to, when a second parameter is less than the input channel parallelism, merge the second preset number of the convolutional kernel elements to obtain second merged data; write the second merged data into a second cache unit; the second parameter indicates the product of the second preset number and the number of input channels;
[0034] The second reading module is configured to, for one pixel of the first cache unit, read the pixel from the first cache unit and read the second merged data from the second cache unit;
[0035] The calculation module is configured to obtain the second preset number of the pixels, perform a convolution calculation on the second preset number of the pixels and the second merged data to obtain a convolution value; the convolution value is a part of the output feature map.
[0036] According to a third aspect of the embodiments of the present application, there is provided a computer device, which includes a processor and a memory, the memory is used to store at least one program, and the at least one program is loaded and executed by the processor to perform the convolution data processing method described above.
[0037] According to a fourth aspect of the embodiments of the present application, there is provided a computer-readable storage medium, in which at least one program is stored, and the at least one program is loaded and executed by a processor to implement the convolution data processing method described above.
[0038] In the embodiments of the present application, a convolution data processing method is provided. Before caching pixels and convolutional kernel elements, the pixels are merged based on the input channel parallelism to obtain a plurality of first merged data; the convolutional kernel elements are merged based on the input channel parallelism to obtain a plurality of second merged data; each first merged data is written into a first cache unit, and each second merged data is written into a second cache unit; compared with the related art, each time one pixel and a large amount of garbage data are written into the first cache unit, and one convolutional kernel element and a large amount of garbage data are written into the second cache unit; the embodiments of the present application reduce the garbage data in the first cache unit and the second cache unit, thereby improving the utilization rate of cache resources. In addition, in the embodiments of the present application, before performing convolution calculation, each first merged data is copied, so that compared with the related art, more pixels can be calculated in each clock cycle, thereby improving the calculation efficiency. Description of the Drawings
[0039] To more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on these drawings.
[0040] Figure 1 is a schematic diagram of an implementation environment provided according to an embodiment of the present application;
[0041] Figure 2 is a schematic diagram of the NPU performing convolution calculation;
[0042] Figure 3 is a schematic flowchart of a convolution data processing method provided according to an embodiment of the present application;
[0043] Figure 4 is a schematic diagram of merging pixels provided according to an embodiment of the present application;
[0044] Figure 5 is a schematic diagram of merging convolution kernel elements provided according to an embodiment of the present application;
[0045] Figure 6 is a schematic diagram of the arrangement of the first merged data provided according to an embodiment of the present application;
[0046] Figure 7 is a schematic diagram of the NPU structure for performing the convolution data processing method provided according to an embodiment of the present application;
[0047] Figure 8 is a schematic diagram of the structure of a convolution data processing device provided according to an embodiment of the present application;
[0048] Figure 9 is a schematic diagram of the structure of a terminal provided according to an embodiment of the present application;
[0049] Figure 10 is a schematic diagram of the structure of a server provided according to an embodiment of the present application. Detailed implementation manners
[0050] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe the embodiments of the present application in detail with reference to the drawings.
[0051] Exemplary embodiments will be described in detail herein, and examples thereof are shown in the accompanying drawings. When the following description refers to the accompanying drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application.
[0052] In the present application, terms such as "first" and "second" are used to distinguish between identical or similar items with substantially the same function and effect. It should be understood that there is no logical or temporal dependency between "first", "second", and "nth", nor are the quantity and execution order limited. It should also be understood that although the following description uses terms such as first and second to describe various elements, these elements should not be limited by the terms.
[0053] These terms are only used to distinguish one element from another. For example, without departing from the scope of various examples, the first action can be called the second action, and similarly, the second action can also be called the first action. Both the first action and the second action can be actions, and in some cases, they can be separate and different actions.
[0054] Wherein, at least one means one or more than one. For example, at least one action can be one action, two actions, three actions, etc., any integer greater than or equal to one. And multiple means two or more than two. For example, multiple actions can be two actions, three actions, etc., any integer greater than or equal to two.
[0055] Figure 1 It is a schematic diagram of an implementation environment provided according to an embodiment of the present application. The implementation environment may include a terminal 101 and a server 102.
[0056] In the terminal 101, a hardware structure for convolution calculation is provided. The hardware structure can be any one of an NPU (Neural-network Processing Unit), a GPU (Graphics Processing Unit), and an SOC (System on Chip).
[0057] The terminal 101 can be a smart phone, a wearable device, a personal computer, a laptop computer, a tablet computer, a smart TV, a vehicle-mounted terminal, etc. having a hardware structure for convolution calculation.
[0058] The server 102 can be a single server, a server cluster composed of multiple servers, or a cloud processing center.
[0059] The terminal 101 is connected to the server 102 via a wired or wireless network.
[0060] In some embodiments, the wireless network or the wired network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to any combination of a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a private network or a virtual private network. In some embodiments, technologies and / or formats including HyperText Mark-up Language (HTML), Extensible Markup Language (XML), etc. are used to represent the data exchanged through the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some of the links. In other embodiments, custom and / or proprietary data communication technologies can also be used to replace or supplement the above data communication technologies.
[0061] In the related art, the computational amount of the convolutional layer often accounts for a large proportion of the total computational amount of the CNN. Therefore, the performance bottleneck of the CNN network calculation is generally also in the convolutional calculation. With the increasingly wide application of neural networks on the edge side, the requirements for the performance of the CNN network by edge-side devices (such as smart phones, smart speakers, smart cameras, etc.) are also getting higher and higher, and more and more edge-side devices will integrate NPUs (Neural Processing Units). Among them, the parameters of the convolutional layer mainly include the convolution kernel (kernel), the number of input channels (input_channels, ic), the number of output channels (output_channels, oc), the convolution stride (stride), and the padding (padding). The following explains these parameters:
[0062] The convolutional kernel is used to perform a convolution operation on the input data to extract features. The convolutional kernel is usually a four-dimensional tensor. The first dimension (oc) represents the number of convolutional kernels, the second dimension (kh) and the third dimension (kw) represent the size of the convolutional kernel. The fourth dimension (ic) represents the number of input channels. The number of input channels represents the number of channels of the input feature map. For example, in an RGB (Red, Green, Blue) image classification task, the number of input channels is 3, indicating that each pixel of the input feature map consists of 3-channel (Red, Green, Blue) data. The number of output channels represents the number of output channels of the convolution operation. The number of output channels usually equals the number of convolutional kernels. The convolution stride is the stride at which the convolutional kernel moves during the convolution operation, used to control the size of the output feature map. The convolution stride is usually a pair, representing the stride of the convolutional kernel in the horizontal and vertical directions respectively. Padding is an operation in the convolution operation, used to pad certain values at the edges of the input data to enlarge the size of the input feature map. Padding is usually a pair, representing the padding size in the horizontal and vertical directions respectively.
[0063] In one example, the NPU is a hardware accelerator specifically designed for convolution calculations. Compared with general-purpose processors, the NPU can accelerate the convolution calculation process through highly optimized hardware structures and algorithms, thereby improving the calculation efficiency and performance. When the NPU accelerates convolution calculations, the most commonly used method is parallel computing. From the parameters of the convolutional layer, it can be concluded that calculations can be performed simultaneously in multiple dimensions: parallel computing of multiple input channels and parallel computing of multiple output channels.
[0064] In addition, quantization is also a commonly used method for accelerating convolution calculations. Quantization of convolution calculations is a technique that converts floating-point numbers in convolution calculations into fixed-point numbers or integers, which can reduce the complexity and storage requirements of convolution calculations while ensuring accuracy, thereby improving the running speed and efficiency of neural networks. The quantization precision refers to the number of bits of the fixed-point number or integer after quantization, usually including 4bit, 8bit, 16bit, etc. For different application requirements, different quantization precisions can be selected. The NPU often uses both quantization and parallel computing methods to accelerate convolution calculations.
[0065] However, the number of input channels, output channels, and the size of the feature maps of different convolutional layers are not the same, while the hardware structure for achieving parallel acceleration is often relatively fixed. Assume that the parallelism degrees of the convolutional computing array in the three dimensions of the number of input channels, output channels, and pixels are PI (Input Channel Parallelism), PO (Output Channel Parallelism), and PP (Pixel Parallelism), respectively. Since the quantization precision is generally 4-bit, 8-bit, or 16-bit, and for the neatness of data, PI, PO, and PP are generally powers of 2. For example, the commonly used PI and PO are 16, 32, or 64, etc., and PP is 4, 8, or 16, etc. When performing convolutional calculations, the alignment requirements of parallel computing need to be followed. The alignment requirements of parallel computing indicate that the number of input channels, output channels, and the width (ow) of the output feature map are integer multiples of PI, PO, and PP, respectively. When the number of input channels is not an integer multiple of PI, or the number of output channels is not an integer multiple of PO, or (ow) is not an integer multiple of PP, it is often necessary to read in some invalid data, thus performing some redundant calculations and then outputting some invalid data.
[0066] In a CNN, the number of input channels of the first convolutional layer is generally relatively small (commonly 3 or 4), while PO is generally relatively large, and the number of input channels is not an integer multiple of PI. To meet the alignment requirements of parallel computing, some garbage data needs to be padded so that the number of input channels is aligned to PI. Also, since the size of the input feature map (IFM) of the first convolutional layer is generally relatively large. For example, the size of the first IFM of the resnet50 network (Residual Network 50) is 224x224. Padding the number of input channels of each pixel to PI will waste a large amount of feature map cache (IFMmem) resources. Similarly, to meet the alignment requirements of parallel computing, garbage data also needs to be supplemented in the input channels of the convolutional kernel to align it to PI, which will also waste a large amount of weight cache (Weightmem) resources. In addition, due to the inclusion of garbage data in the input channel data, it causes a waste of the computing resources of the neural network. Here, the garbage data can also be expressed as padding data. For example, when the number of input channels is 4 and padded to 16 input channels, 12 / 16 = 3 / 4 of the computing resources are wasted, thus reducing the computing efficiency of the neural network.
[0067] Figure 2 is a schematic diagram of the NPU performing convolutional calculations.
[0068] The following combines Figure 2 to make an exemplary description of the convolutional calculations of the NPU.
[0069] In one example, due to the limited on-chip storage space of the NPU, a large-capacity input feature map and convolution kernels are stored in an off-chip storage unit. The NPU reads the input feature map and convolution kernels from the off-chip storage unit. When the NPU reads the input feature map data and convolution kernels from the off-chip storage unit, each pixel of the input feature map is cached in the IFMmem, and the convolution kernel elements are cached in the Weightmem. To meet the alignment requirements for parallel computing, PP pixels are read from the IFMmem, corresponding to PI input channels. PO convolution kernels are read from the Weightmem. Corresponding to PI input channels and PO output channels, the convolution kernels and pixels are output to the convolution calculation unit for convolution calculation. However, when the number of input channels of the convolution layer is small, a large amount of garbage data needs to be stored in the IFM mem and Weight mem. This will result in a large proportion of garbage data, thus causing waste of cache resources and computing resources, and further reducing the computing efficiency.
[0070] To solve the above technical problems, an embodiment of the present application provides a convolution data processing method.
[0071] Figure 3 It is a schematic flowchart of a convolution data processing method provided by an embodiment of the present application. As Figure 3 shown, in the embodiment of the present application, it is described by taking an application to a terminal with a hardware structure for convolution calculation as an example. The method includes the following steps:
[0072] First, the caching process is exemplarily described in combination with step 301 and step 303.
[0073] In step 301, the terminal reads a first preset number of pixels and a second preset number of convolution kernel elements.
[0074] Among them, the first preset number of pixels is a part of the input feature map; the second preset number of convolution kernel elements is a part of a convolution kernel.
[0075] In some embodiments, since the input feature map and the convolutional kernel are stored in an off-chip storage unit, when the terminal performs convolutional calculations on the input feature map and the convolutional kernel, it needs to read the input feature map and the convolutional kernel from the off-chip storage unit. The off-chip storage unit includes multiple storage addresses. Each storage address is used to store a pixel or a convolutional kernel element. Each convolutional kernel includes multiple convolutional kernel elements. Optionally, the second preset quantity indicates the width of the convolutional kernel. That is, the terminal reads a row of convolutional kernel elements in a convolutional kernel from the off-chip storage unit each time. For example: Each pixel includes 3 pixel channel data. Each convolutional kernel element includes 3 convolutional kernel channel data. A convolutional kernel with a size of 3 (width) × 3 (height) includes 9 convolutional kernel elements. Another example: When PI is 16, the bit width of the data port of the off-chip storage unit is 128 bit. That is, each storage address can store at most 16 complete pixel channel data or convolutional kernel channel data. The bit widths of the data ports of the first buffer unit and the second buffer unit are 128 bit.
[0076] In step 302, when the first parameter is less than the input channel parallelism, merge the first preset quantity of pixels to obtain first merged data; write the merged data into the first buffer unit.
[0077] Wherein, the first parameter indicates the product of the first preset quantity and the number of input channels.
[0078] In the related art, after the terminal obtains the input feature map and the convolutional kernel from the off-chip storage unit, it caches the input feature map and the convolutional kernel into the first buffer unit and the second buffer unit respectively in the same storage format as the off-chip storage unit. That is, each first address in the first buffer unit stores a pixel. Each second address in the second buffer unit stores a convolutional kernel element. When the number of input channels of the convolutional layer is small, that is, when the first address or the second address is not fully utilized, a large amount of garbage data will be filled in the first buffer unit or the second buffer unit.
[0079] To solve the above problems, in the embodiments of the present application, after reading the first preset quantity of pixels from the off-chip storage unit, if the first parameter is less than the input channel parallelism, merge the first preset quantity of pixels to obtain a first merged data. Then store the first merged data in a first address. Compared with the related art where only one pixel is stored in a first address, in the embodiments of the present application, the first preset quantity of pixels is stored in a first address. Accordingly, the amount of garbage data to be filled in each first address is reduced, thereby improving the space utilization rate of the first buffer unit. The first preset quantity is an integer greater than 1. Similarly, the amount of garbage data filled in each second address is also reduced.
[0080] It can be understood that the more garbage data stored in the first cache unit, the lower the space utilization rate.
[0081] In some embodiments, step 302 further includes: when the first parameter is less than the input channel parallelism, writing the first padding data into the first cache unit; the number of padding channel data included in the first padding data and the number of pixel channel data included in the first merged data sum up to the input channel parallelism; each pixel includes multiple pixel channel data; the first padding data includes multiple padding channel data. For example: Since the input channel parallelism is 16. When the first preset number is 4 and the number of pixel channel data included in each pixel is 3. Then the first parameter 12 (4×3 = 12) is less than the input channel parallelism 16. Merging these 4 pixels, that is, merging 12 pixel channel data, to obtain a first merged data. Since the size of each pixel channel data is 8 bit and the size of each pixel is 24 bit, the size of the first merged data is 96 bit. Therefore, 4 padding channel data need to be supplemented, and the size of each padding channel data is 8 bit. That is, 32 bit of the first padding data is supplemented. In this case, the utilization rate of the first address is 3 / 4. Compared with the related art, the embodiments of the present application reduce the size of the first padding data and improve the utilization rate of the first address. Each bit of the first padding data can be "0".
[0082] In some embodiments, when the first parameter is equal to the input channel parallelism, the number of pixel channel data included in the first merged data is equal to the input channel parallelism. Optionally, in different input feature maps, the number of pixel channel data included in each pixel may also be different. In some cases, multiple pixels may exactly fill a first address. For example: Each pixel includes 4 pixel channel data. Then when 4 pixels are written into each first address, the first address is exactly fully utilized without padding garbage data. That is, in this case, the space utilization rate of the first cache unit is 100%. In addition, since there is no need to pad garbage data, there is no need to perform convolution calculations on the garbage data, thus saving computing resources and further improving the computing efficiency.
[0083] Figure 4 It is a schematic diagram of merging pixels according to an embodiment of the present application.
[0084] The following combines Figure 4 to make an exemplary description of merging pixels.
[0085] In some embodiments, when the first parameter is greater than the input channel parallelism, a first preset number of pixels are divided into multiple groups; each group includes a sub-preset number of pixels; the sub-preset number of pixels in each group are combined to obtain multiple sub-combined data; each sub-combined data and a sub-padding data are written into the first buffer unit; the sum of the number of pixel channel data included in each sub-combined data and the number of padding channel data included in a sub-padding data is equal to the input channel parallelism. Optionally, each group includes the same number of pixels, so that the same number of pixels are stored in each first address, thereby ensuring the neatness of the data. For example: each storage address in the off-chip storage unit is used to store one pixel. The pixel includes ic0 (the zeroth pixel channel data), ic1 (the first pixel channel data), and ic2 (the second pixel channel data). For example, each storage address can store 128bit. Therefore, 104bit of garbage data needs to be filled. The 104bit of garbage data is represented by "104'b0". That is, each bit value is "0". In the related art, the storage format of the first buffer unit is the same as that of the off-chip storage unit. In the embodiments of the present application, since the input channel parallelism is 16. When the first preset number is 8, the number of input channels is 3, and the first parameter is 24, the input channel parallelism is less than the first parameter. To maintain data symmetry, these 8 pixels are divided into two groups. The first group includes pixel n0, pixel n1, pixel n2, and pixel n3. The second group includes pixel n4, pixel n5, pixel n6, and pixel n7. That is, the sub-preset number is 4. Taking the first group as an example, pixel n0, pixel n1, pixel n2, and pixel n3, that is, 12 pixel channel data are combined to obtain a sub-combined data. The size of the sub-combined data is 96bit. Therefore, 32bit of sub-padding data is also required. In this case, the utilization rate of the first address is 3 / 4. That is, compared with filling 104bit of garbage data in the related art, the size of the first padding data is reduced, and the space utilization rate of the first buffer unit is improved. Similarly, the first preset number can also be 16 or the like according to actual needs.
[0086] In step 303, when the second parameter is less than the input channel parallelism, a second preset number of convolution kernel elements are combined to obtain a second combined data; the second combined data is written into the second buffer unit.
[0087] Wherein, the second parameter indicates the product of the second preset number and the number of input channels.
[0088] In some embodiments, the terminal simultaneously executes the above step 302 and step 303.
[0089] In some embodiments, after reading a second preset number of convolutional kernel elements from an off-chip storage unit, if the second parameter is less than the input channel parallelism, the second preset number of convolutional kernel elements are merged to obtain a second merged data. Then, the second merged data is stored in a second address. Compared with the related art where only one convolutional kernel element is stored in one second address, in the embodiments of the present application, the second preset number of convolutional kernel elements are stored in one second address. Accordingly, the amount of garbage data to be filled in each second address is reduced, thereby improving the space utilization rate of the second cache unit. The second preset number is an integer greater than 1.
[0090] It should be noted that there is no essential difference between the first address and the second address, except for the convenience of description and distinction.
[0091] Figure 5 It is a schematic diagram of merging convolutional kernel elements according to an embodiment of the present application.
[0092] The following combines Figure 5 to give an exemplary illustration of merging convolutional kernel elements.
[0093] In some embodiments, step 303 above further includes:
[0094] When the second parameter is less than the input channel parallelism, the second padding data is written into the second buffer unit; the sum of the number of padding channel data included in the second padding data and the number of convolution kernel channel data included in the second merged data is equal to the input channel parallelism; each convolution kernel includes multiple convolution kernel channel data. Optionally, when the second parameter is less than the input channel parallelism, that is, after the second merged data is written into the second address, the second address is not yet full. Therefore, it is necessary to write the second padding data into the second address. For example: each bit of the second padding data is "0". Another example: each storage address in the off-chip storage unit is used to store a convolution kernel element. The convolution kernel element includes ic0 (the zeroth convolution kernel channel data), ic1 (the first convolution kernel channel data), and ic2 (the second convolution kernel channel data). The size of each convolution kernel channel data is 8 bit. Since each storage address can store 128 bit. Therefore, 104 bit of garbage data needs to be filled. The 104 bit of garbage data is represented by "104'b0", that is, the value of each bit is "0". For ease of description, ocj is used to represent the jth output channel. j is an integer greater than or equal to 0 and less than or equal to (PO - 1). "kh" represents the height of the convolution kernel. "kw" represents the width of the convolution kernel. In the related art, the storage format of the second buffer unit is the same as that of the off-chip storage unit. In the embodiments of the present application, when PO is 16, each row of the second merged data includes 16 second merged data. For example: the oc0 - kh0 row includes 16 second merged data. Each second merged data is composed of the first convolution kernel element kw0 in the first row of the convolution kernel, the second convolution kernel element kw1 in the first row of the convolution kernel, and the third convolution kernel element kw2 in the first row of the convolution kernel. That is, 9 convolution kernel channel data are merged to obtain one second merged data. That is, the second parameter 9 (the second preset quantity is 3, and the number of input channels is 3) is less than the input channel parallelism 16. 7 padding channel data also need to be supplemented. The size of the second merged data is 72 bit. Then 56 bit of second padding data is also needed. In this case, the utilization rate of the second address is 9 / 16. That is, compared with the related art, the size of the second padding data is reduced, and the space utilization rate of the second buffer unit is improved.
[0095] In some embodiments, when the second parameter is equal to the input channel parallelism, the number of convolution kernel channel data included in the second merged data is equal to the input channel parallelism. For example: when PI is 16. The size of the convolution kernel is 2×2. Each convolution kernel element includes 4 convolution kernel input channels, then 4 convolution kernel elements are stored in one second address. That is, the utilization rate of the second address reaches 100%.
[0096] The convolution calculation will be exemplarily described below in combination with step 304 and step 305.
[0097] In step 304, for a pixel of the first buffer unit, read the pixel from the first buffer unit and read the second merged data from the second buffer unit.
[0098] In some embodiments, the terminal reads a first quantity of first merged data and first padding data from the first buffer unit; the first quantity is (pixel parallelism / first preset quantity); and reads k rows of second merged data and second padding data from the second buffer unit; each row of the second merged data includes a second quantity of second merged data and second padding data; the second quantity indicates the output channel parallelism; k is an integer greater than or equal to 1 and less than or equal to the height; the height is the height of the convolutional kernel. Duplicate the i-th pixel in the first merged data to obtain the i-th duplicated data; the i-th duplicated data includes a second preset quantity of the i-th pixel; i is an integer greater than or equal to 1 and less than or equal to the first preset quantity.
[0099] In step 305, the terminal obtains a second preset quantity of pixels, performs a convolution calculation on the second preset quantity of pixels and the second merged data to obtain a convolution value. Among them, the convolution value is a part of the output feature map.
[0100] Figure 6 It is a schematic diagram of the arrangement of a first merged data provided according to an embodiment of the present application.
[0101] The following combines Figure 6 to give an exemplary description of the arrangement of the first merged data.
[0102] In some embodiments, to meet the alignment requirements of parallel computing, the terminal uses the i-th replicated data and the third padding data as the i-th arranged data; the sum of the number of pixel channel data included in the i-th replicated data and the number of padding channel data included in the third padding data is equal to the input channel parallelism. The i-th arranged data is convolved with each second merged data and the second padding data in the k-th row to obtain the output feature map. Optionally, each replicated data is arranged at the data port of the first buffer unit and transmitted to the convolution calculation unit through the data port of the first buffer unit. When the number of pixel channel data included in the replicated data is less than the input channel parallelism, a third padding data is added to the replicated data to obtain an arranged data. Each third padding data corresponds to 3 padding channel data, that is, garbage data. The number of pixel channel data included in an arranged data is equal to the input channel parallelism. Multiple arranged data are output to the convolution calculation unit. Each arranged data is convolved with each second merged data. For example: PI is 16, and the number of pixel channel data included in each replicated data is 9, that is, the number of pixel channel data included in each replicated data is less than PI, so 7 padding channel data need to be added. That is, 56-bit third padding data needs to be added. Each bit of the third padding data is "0".
[0103] Optionally, the k rows of second merged data are arranged at the data port of the second buffer unit. Each second merged data corresponds to a second padding data; the k rows of second merged data and the second padding data are output to the convolution calculation unit. Optionally, the first arranged data is replicated multiple times to obtain a total of PO first arranged data. For example: Taking one first arranged data as an example for explanation. The first pixel n0 is convolved with the convolution kernel element kw0. The second pixel n1 is convolved with the convolution kernel element kw1. The third pixel n0 is convolved with the convolution kernel element kw2. Similarly, each first arranged data is convolved.
[0104] In the related art, in each clock cycle, the terminal can calculate PP pixels, PO output channels, and PI input channels. In the embodiments of the present application, in each clock cycle, the terminal can calculate PP×the second preset number of pixels, PO output channels, and PI input channels. Therefore, compared with the related art, the calculation efficiency of the embodiments of the present application is greatly improved.
[0105] Figure 7 It is a schematic diagram of an NPU structure for performing a convolution data processing method according to an embodiment of the present application.
[0106] The following combines Figure 7 to exemplarily illustrate the NPU for performing the convolution data processing method.
[0107] In some embodiments, the NPU is electrically connected to an off-chip storage unit. The NPU includes a first processor, a second processor, a first cache unit, a second cache unit, and a convolution calculation unit. Among them, the first processor is electrically connected to the first cache unit and the off-chip storage unit. The second processor is electrically connected to the second cache unit and the off-chip storage unit. The first cache unit and the second cache unit are respectively electrically connected to the convolution calculation unit. Among them, the first processor, the second processor, the first cache unit, the second cache unit, and the convolution calculation unit are all hardware structures. For example: the first cache unit is IFMmem, and the second cache unit is Weight mem.
[0108] In one example, the first processor reads a first preset number of pixels from the off-chip cache unit. The first preset number of pixels are merged to obtain first merged data. The first merged data is written to a first address in the first cache unit. At the same time, the second processor reads a second preset number of convolution kernel elements from the off-chip cache unit. The second preset number of convolution kernel elements are merged to obtain second merged data. The second merged data is written to a second address in the second cache unit.
[0109] In one example, the NPU reads a first number of first merged data from the first cache unit and a second number of second merged data from the second cache unit. Each pixel in the first merged data is copied a second preset number of times to obtain a plurality of arranged data. Based on the input parallelism and the output parallelism, the plurality of arranged data and the second merged data are output to the convolution calculation unit. The convolution calculation unit performs convolution calculation on each pixel and convolution kernel element based on preset parameters to obtain an output feature map. It should be noted that the preset parameters indicate that each pixel channel data in a pixel is convolved with each convolution kernel channel data in any one convolution kernel element. That is, the preset parameters indicate the specific process of convolution calculation.
[0110] In the embodiments of the present application, pixels are merged based on the input channel parallelism to obtain a plurality of first merged data; convolution kernel elements are merged based on the input channel parallelism to obtain a plurality of second merged data; each first merged data is written to the first cache unit, and each second merged data is written to the second cache unit; compared with the related art where each time a pixel and a large amount of garbage data are written to the first cache unit, and a convolution kernel element and a large amount of garbage data are written to the second cache unit; the embodiments of the present application reduce the garbage data in the first cache unit and the second cache unit, thereby improving the utilization rate of cache resources. In addition, in the embodiments of the present application, each first merged data is copied before convolution calculation, so that compared with the related art, more pixels can be calculated per clock cycle, thereby improving the calculation efficiency.
[0111] Figure 8It is a schematic structural diagram of a convolutional data processing device 800 provided according to an embodiment of the present application. The device includes:
[0112] A first reading module 801, configured to read a first preset number of pixels and a second preset number of convolutional kernel elements; the first preset number of pixels is a part of an input feature map; the second preset number of convolutional kernel elements is a part of a convolutional kernel;
[0113] A first shaping module 802, configured to, when a first parameter is less than the input channel parallelism, merge the first preset number of pixels to obtain first merged data; write the first merged data into a first buffer unit; the first parameter indicates the product of the first preset number and the number of input channels;
[0114] A second shaping module 803, configured to, when a second parameter is less than the input channel parallelism, merge the second preset number of convolutional kernel elements to obtain second merged data; write the second merged data into a second buffer unit; the second parameter indicates the product of the second preset number and the number of input channels;
[0115] A second reading module 804, configured to, for a pixel in the first buffer unit, read the pixel from the first buffer unit and read the second merged data from the second buffer unit;
[0116] A calculation module 805, configured to obtain a second preset number of pixels, perform a convolution calculation on the second preset number of pixels and the second merged data to obtain a convolution value; the convolution value is a part of an output feature map.
[0117] In some embodiments, the device is further configured to:
[0118] When the first parameter is less than the input channel parallelism, write first padding data into the first buffer unit; the number of padding channel data included in the first padding data and the number of pixel channel data included in the first merged data sum up to the input channel parallelism; each pixel includes multiple pixel channel data; the first padding data includes multiple padding channel data.
[0119] In some embodiments, the device is further configured to:
[0120] When the first parameter is equal to the input channel parallelism, the number of pixel channel data included in the first merged data is equal to the input channel parallelism.
[0121] In some embodiments, the device is further configured to:
[0122] When the first parameter is greater than the input channel parallelism, divide the first preset number of pixels into multiple groups; each group includes a sub-preset number of pixels;
[0123] Merge the sub-preset number of pixels in each group to obtain multiple sub-merged data;
[0124] Write each sub - merged data and a sub - padding data into the first cache unit; the sum of the number of pixel channel data included in each sub - merged data and the number of padding channel data included in a sub - padding data is equal to the input channel parallelism.
[0125] In some embodiments, the apparatus is further configured to:
[0126] When the second parameter is less than the input channel parallelism, write the second padding data into the second cache unit; the sum of the number of padding channel data included in the second padding data and the number of convolution kernel channel data included in the second merged data is equal to the input channel parallelism; each convolution kernel includes multiple convolution kernel channel data.
[0127] In some embodiments, the apparatus is further configured to:
[0128] When the second parameter is equal to the input channel parallelism, the number of convolution kernel channel data included in the second merged data is equal to the input channel parallelism.
[0129] In some embodiments, the second preset quantity indicates the width of the convolution kernel.
[0130] In some embodiments, each pixel includes 3 pixel channel data; each convolution kernel element includes 3 convolution kernel channel data.
[0131] In some embodiments, the apparatus is further configured to:
[0132] Read a first quantity of first merged data and first padding data from the first cache unit; the first quantity is (pixel parallelism / first preset quantity); and read k rows of second merged data and second padding data from the second cache unit; each row of the second merged data includes a second quantity of second merged data and second padding data; the second quantity indicates the output channel parallelism; k is an integer greater than or equal to 1 and less than or equal to the height; the height is the height of the convolution kernel;
[0133] Copy the i - th pixel in the first merged data to obtain the i - th copied data; the i - th copied data includes a second preset quantity of the i - th pixel; i is an integer greater than or equal to 1 and less than or equal to the first preset quantity;
[0134] Use the i - th copied data and the third padding data as the i - th arranged data; the sum of the number of pixel channel data included in the i - th copied data and the number of padding channel data included in the third padding data is equal to the input channel parallelism;
[0135] Perform convolution calculation on the i - th arranged data and each row of the second merged data and second padding data in the k - th row to obtain the output feature map.
[0136] It should be noted that when the convolutional data processing device provided in the above embodiment executes the corresponding steps, only the division of the above functional modules is used for illustration. In actual applications, the above functions can be allocated to different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the functions described above. In addition, the convolutional data processing device provided in the above embodiment and the convolutional data processing method embodiment belong to the same concept. For the specific implementation process, please refer to the method embodiment and will not be elaborated here.
[0137] In the embodiment of the present application, pixels are merged based on the input channel parallelism to obtain a plurality of first merged data; convolutional kernel elements are merged based on the input channel parallelism to obtain a plurality of second merged data; each first merged data is written into a first cache unit, and each second merged data is written into a second cache unit; compared with the related art where one pixel and a large amount of garbage data are written into the first cache unit each time, and one convolutional kernel element and a large amount of garbage data are written into the second cache unit; the embodiment of the present application reduces the garbage data in the first cache unit and the second cache unit, thereby improving the utilization rate of cache resources. In addition, in the embodiment of the present application, before performing convolutional calculation, each first merged data is copied, so that compared with the related art, more pixels can be calculated per clock cycle, thereby improving the calculation efficiency.
[0138] The embodiment of the present application also provides a computer device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, the above method is implemented.
[0139] Taking the computer device as a terminal as an example, Figure 9 is a schematic structural diagram of a terminal provided by an embodiment of the present application. Refer to Figure 9 , the terminal 900 may be: a smart phone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 (Moving Picture Experts Group Audio Layer IV) player, a notebook computer or a desktop computer. The terminal 900 may also be referred to by other names such as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, etc.
[0140] Generally, the terminal 900 includes: a processor 901 and a memory 902.
[0141] The processor 901 may include one or more processing cores, such as a quad-core processor, a penta-core processor, etc. The processor 901 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU (Graphics Processing Unit), and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may further include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.
[0142] The memory 902 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 902 may further include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 is used to store at least one program code, and the at least one program code is used to be executed by the processor 901 to implement the process executed by the terminal in the method embodiments provided in this application for the above-mentioned method.
[0143] In some embodiments, the terminal 900 may further optionally include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902, and the peripheral device interface 903 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 903 through a bus, signal lines, or a circuit board. Specifically, the peripheral device includes at least one of a display screen 904, a camera assembly 905, an audio circuit 906, and a power supply 907.
[0144] The peripheral device interface 903 can be used to connect at least one I / O (Input / Output) related peripheral device to the processor 901 and the memory 902. In some embodiments, the processor 901, the memory 902, and the peripheral device interface 903 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 901, the memory 902, and the peripheral device interface 903 can be implemented on separate chips or circuit boards, and the embodiments of the present application do not limit this.
[0145] The display screen 904 is used to display the UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 904 is a touch display screen, the display screen 904 also has the ability to collect touch signals on or above the surface of the display screen 904. The touch signal can be input to the processor 901 as a control signal for processing. At this time, the display screen 904 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 904, which is arranged on the front panel of the terminal 900; in some other embodiments, there can be at least two display screens 904, which are respectively arranged on different surfaces of the terminal 900 or in a foldable design; in some other embodiments, the display screen 904 can be a flexible display screen, which is arranged on the curved surface or the folding surface of the terminal 900. Even, the display screen 904 can also be set as an irregular non-rectangular shape, that is, a special-shaped screen. The display screen 904 can be prepared using materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).
[0146] The camera assembly 905 is used to collect images or videos. In some embodiments, the camera assembly 905 includes a front camera and a rear camera. Generally, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth camera, a wide-angle camera, and a telephoto camera respectively, to implement functions such as the fusion of the main camera and the depth camera to achieve the background blurring function, the fusion of the main camera and the wide-angle camera to achieve panoramic shooting and VR (Virtual Reality) shooting functions or other fusion shooting functions. In some embodiments, the camera assembly 905 can also include a flash. The flash can be a single-color temperature flash or a two-color temperature flash. The two-color temperature flash refers to the combination of a warm light flash and a cold light flash, which can be used for light compensation under different color temperatures.
[0147] The audio circuit 906 may include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for input to the processor 901 for processing. For the purpose of stereo collection or noise reduction, there may be multiple microphones, which are respectively arranged at different parts of the terminal 900. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert the electrical signals from the processor 901 into sound waves. The speaker may be a traditional thin-film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for uses such as ranging. In some embodiments, the audio circuit 906 may further include a headphone jack.
[0148] The power supply 907 is used to supply power to each component in the terminal 900. The power supply 907 may be alternating current, direct current, a disposable battery or a rechargeable battery. When the power supply 907 includes a rechargeable battery, the rechargeable battery may support wired charging or wireless charging. The rechargeable battery may also be used to support fast charging technology.
[0149] Those skilled in the art can understand that Figure 9 the structure shown in
[0150] Taking the computer device as a server as an example, Figure 10 FIG. is a schematic structural diagram of a server provided by an embodiment of the present application. The server 1000 may vary greatly due to different configurations or performances, and may include one or more processors (Central Processing Units, CPUs) 1001 and one or more memories 1002. Among them, at least one computer program is stored in the one or more memories 1002, and the at least one computer program is loaded and executed by the one or more processors 1001 to implement the above-mentioned convolutional data processing method. Of course, the server 1000 may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server 1000 may further include other components for implementing the functions of the device, which will not be elaborated here.
[0151] Embodiments of the present application further provide a computer-readable storage medium. The computer-readable storage medium includes a stored computer program. When the computer program runs, it controls the device where the computer-readable storage medium is located to execute the method described above. Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, optical data storage device, etc.
[0152] Those of ordinary skill in the art can understand that all or part of the steps to implement the above embodiments can be completed by hardware or by a program instructing relevant hardware. The program can be stored in a computer-readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk, an optical disc, etc.
[0153] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.
Claims
1. A convolution data processing method, characterized in that: include: Reading a first preset number of pixels and a second preset number of convolution kernel elements; The first predetermined number of pixels are part of an input feature map; The second preset number of convolution kernel elements is a part of a convolution kernel; When the first parameter is less than the input channel parallelism, merging the first preset number of pixels to obtain first merged data; Writing the first merged data into a first cache unit; The first parameter indicates the product of the first preset number and the number of input channels; When the second parameter is less than the input channel parallelism, merging the second preset number of convolution kernel elements to obtain second merged data; Writing the second merged data into a second cache unit; The second parameter indicates the product of the second preset number and the number of input channels; For one pixel in the first cache unit, read the pixel from the first cache unit, and read the second combined data from the second cache unit; Obtain the second preset number of pixels, perform convolution calculation on the second preset number of pixels and the second merged data, and obtain a convolution value; the convolution value is a part of the output feature map.
2. The method according to claim 1, characterized in that Writing the first merged data into the first cache unit includes: When the first parameter is less than the input channel parallelism, the first filling data is written into the first cache unit; the sum of the number of filling channel data included in the first filling data and the number of pixel channel data included in the first merged data is equal to the input channel parallelism; each of the pixels includes a plurality of pixel channel data; the first filling data includes a plurality of filling channel data.
3. The method according to claim 2, characterized in that The method further comprises: When the first parameter is equal to the input channel parallelism, the number of the pixel channel data included in the first merged data is equal to the input channel parallelism.
4. The method according to claim 2, characterized in that: The method further comprises: When the first parameter is greater than the input channel parallelism, the first preset number of pixels are divided into a plurality of groups; each group includes a sub-preset number of pixels; Merging each group of the sub-preset number of pixels to obtain a plurality of sub-merged data; Each of the sub-merged data and one sub-filling data is written into the first cache unit; the sum of the number of the pixel channel data included in each of the sub-merged data and the number of the filling channel data included in one of the sub-filling data is equal to the input channel parallelism.
5. The method according to claim 2, characterized in that: Writing the second merged data into the second cache unit includes: When the second parameter is less than the input channel parallelism, the second filling data is written into the second cache unit; the sum of the number of the filling channel data included in the second filling data and the number of the convolution kernel channel data included in the second merged data is equal to the input channel parallelism; each of the convolution kernels includes a plurality of the convolution kernel channel data.
6. The method according to claim 2, characterized in that The method further comprises: When the second parameter is equal to the input channel parallelism, the number of the convolution kernel channel data included in the second merged data is equal to the input channel parallelism.
7. The method according to claim 1, characterized in that The second preset number indicates the width of the convolution kernel.
8. The method according to claim 5, characterized in that Each of the pixels includes three of the pixel channel data; each of the convolution kernel elements includes three of the convolution kernel channel data.
9. The method according to claim 5, characterized in that The method further comprises: A first quantity of the first merged data and the first padding data is read from the first cache unit; the first quantity is (pixel parallelism / the first preset quantity); and k lines of the second merged data and the second padding data are read from the second cache unit; each line of the second merged data includes a second quantity of the second merged data and the second padding data; the second quantity indicates the output channel parallelism; k is an integer greater than or equal to 1 and less than or equal to the height; the height is the height of the convolution kernel; Copy the i-th pixel in the first combined data to obtain the i-th copied data; the i-th copied data includes the i-th pixel of the second preset number; i is an integer greater than or equal to 1 and less than or equal to the first preset number; The i-th copy data and the third padding data are used as the i-th arrangement data; the sum of the number of the pixel channel data included in the i-th copy data and the number of the padding channel data included in the third padding data is equal to the input channel parallelism; Convolution calculation is performed on the i-th arranged data, each of the second merged data in the k-th row, and the second padding data to obtain the output feature map.
10. A convolution data processing device, characterized in that: include: A first reading module, used for reading a first preset number of pixels and a second preset number of convolution kernel elements; The first predetermined number of pixels are part of an input feature map; The second preset number of convolution kernel elements is a part of a convolution kernel; A first shaping module, configured to merge the first preset number of pixels to obtain first merged data when the first parameter is less than the input channel parallelism; Writing the first merged data into a first cache unit; The first parameter indicates the product of the first preset number and the number of input channels; A second shaping module, configured to merge the second preset number of convolution kernel elements to obtain second merged data when a second parameter is less than the input channel parallelism; Writing the second merged data into a second cache unit; The second parameter indicates the product of the second preset number and the number of input channels; A second reading module, configured to read the pixel from the first cache unit and read the second combined data from the second cache unit for one pixel in the first cache unit; A calculation module is used to obtain the second preset number of pixels, perform convolution calculation on the second preset number of pixels and the second merged data, and obtain a convolution value; the convolution value is part of the output feature map.
11. A computer device, characterized in that: The computer device includes a processor and a memory, the memory is used to store at least one program, and the at least one program is loaded by the processor and executes the convolution data processing method according to any one of claims 1 to 9.
12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one program, and the at least one program is loaded and executed by the processor to implement the convolution data processing method according to any one of claims 1 to 9.