Data processing method and device, processor, electronic equipment and storage medium

Abstract

The invention discloses a data processing method and device, a processor, electronic equipment and a non-instantaneous computer readable storage medium. The data processing method comprises the following steps: receiving a first convolution kernel and carrying out first rearrangement transformation to obtain a second convolution kernel; receiving a first input feature map, and performing second rearrangement transformation on the first input feature map to obtain a second input feature map; and performing convolution processing by using the second input feature map and the second convolution kernel to obtain a second output feature map, wherein the second output feature map is equivalent to a first output feature map obtained by directly performing convolution processing on the first input feature map and the first convolution kernel. According to the method, the overall performance of operators is improved, no load of partial computing resources is avoided as much as possible, and the utilization rate of hardware computing is improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a data processing method and apparatus, a processor, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] In current hardware devices, data processing integrated circuits such as graphics processing units (GPUs) and general-purpose graphics processing units (GPGPUs) can execute programs to complete various tensor-related computing tasks. As computing needs increase, hardware devices have higher and higher performance requirements for operators. Better performance means faster completion of calculations and reduced computing overhead. Evenly mapping computing tasks to hardware devices and maximizing the parallel execution efficiency of hardware devices have become the key to performance improvement.

[0003] In artificial intelligence models for image processing, convolution is an operator that can compress the information features of original images and is widely used in most common image processing tasks (such as visual tasks). In addition, large-size convolutions are often applied to the input layer of visual models to perform initial information compression for subsequent network layer processing. Summary of the invention

[0004] At least one embodiment of the present disclosure provides a data processing method, including: receiving a first convolution kernel and performing a first rearrangement transformation to obtain a second convolution kernel, wherein the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, and the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; receiving a first input feature map, wherein the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; the first input feature map is subjected to a second rearrangement transformation to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), where roundup(×) is a rounding-up function; performing convolution processing using the second input feature map and the second convolution kernel to obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by directly performing the convolution processing on the first input feature map and the first convolution kernel.

[0005] For example, the data processing method provided by at least one embodiment of the present disclosure also includes: storing the obtained second convolution kernel for use in the convolution processing.

[0006] For example, in the data processing method provided by at least one embodiment of the present disclosure, the first rearrangement transformation is performed to obtain the second convolution kernel, including: in response to R1 or S1 being a prime number, the first convolution kernel is padded to obtain an intermediate convolution kernel, and the intermediate convolution kernel is split and rearranged to obtain the second convolution kernel, wherein the size of the intermediate convolution kernel is expressed as (M, C1, R', S'), M represents the number of the intermediate convolution kernels, C1 represents the number of channels of the intermediate convolution kernel, R' represents the matrix height of the intermediate convolution kernel, S' represents the matrix width of the intermediate convolution kernel, R'=R2×roundup(R1 / R2), S'= S2× roundup(S1 / S2); in response to R1 and S1 being not prime numbers, the first convolution kernel is directly split and rearranged to obtain the second convolution kernel.

[0007] For example, in the data processing method provided by at least one embodiment of the present disclosure, the split and rearrangement transformation includes: for each kernel matrix in the first convolution kernel or the intermediate convolution kernel, splitting the kernel matrix according to a first order to obtain k second kernel matrices corresponding to the kernel matrix, and superimposing the k second kernel matrices in the k channels of the second convolution kernel according to the first order to obtain the second convolution kernel, where k is a positive integer.

[0008] For example, in the data processing method provided by at least one embodiment of the present disclosure, the first rearrangement transformation is performed offline to obtain the second convolution kernel from the first convolution kernel.

[0009] For example, in the data processing method provided by at least one embodiment of the present disclosure, the first input feature map is subjected to a second rearrangement transformation to obtain a second input feature map, including: performing padding processing on the first input feature map to obtain an intermediate input feature map, wherein the size of the intermediate input feature map is expressed as (N, C1, H', W'), N represents the batch of the intermediate input feature map, C1 represents the number of channels of the intermediate input feature map, H' represents the matrix height of the intermediate input feature map, W' represents the matrix width of the intermediate input feature map, H'=H1+2P, W'=W1+2P, P is the number of padding elements, and then performing the second rearrangement transformation on the intermediate input feature map to obtain the second input feature map; or, directly performing the second rearrangement transformation on the first input feature map to obtain the second input feature map.

[0010] For example, in the data processing method provided by at least one embodiment of the present disclosure, the second rearrangement transformation is performed on the first input feature map or the intermediate input feature map to obtain the second input feature map, including: for each first input matrix in the first input feature map or the intermediate input feature map, determining the kernel matrix corresponding to the first input matrix, taking out the elements of the first input matrix corresponding to the kernel matrix for convolution calculation, rearranging to obtain k second input matrices corresponding one-to-one to the k second kernel matrices, and superimposing the k second input matrices in the k channels of the second input feature map in the first order to obtain the second input feature map.

[0011] For example, in the data processing method provided by at least one embodiment of the present disclosure, the second input feature map is obtained by performing the second rearrangement transformation on the first input feature map online.

[0012] For example, the data processing method provided by at least one embodiment of the present disclosure also includes: using a convolution processing device to perform the data processing method, wherein the convolution processing device is configured to process Cm channels in parallel, Cm is greater than or equal to C2, and the convolution processing device is configured so that the maximum convolution kernel size for a single processing is Km, the maximum value between R2 and S2 is less than or equal to Km, the minimum value between R1 and S1 is greater than Km, and Km and Cm are positive integers.

[0013] At least one embodiment of the present disclosure provides a data processing device, including: a first rearrangement unit, configured to receive a first convolution kernel and perform a first rearrangement transformation to obtain a second convolution kernel, wherein the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, and the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; a receiving unit, configured to receive a first input feature map, wherein the size of the first input feature map is expressed as (N, C1 , H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; the second rearrangement unit is configured to perform a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), where roundup(×) is a rounding-up function; a processing unit, configured to perform convolution processing using the second input feature map and the second convolution kernel and obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by performing the convolution processing on the first input feature map and the first convolution kernel.

[0014] At least one embodiment of the present disclosure provides a data processing method, comprising: receiving a convolution processing instruction, wherein the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 is used to represent the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; after parsing the convolution processing instruction, using a convolution processing unit to execute the convolution processing instruction to obtain a second output feature map, wherein the second output feature map is the same as the output feature map obtained by the convolution processing unit. The first input feature map and the first convolution kernel are directly convolved to obtain a first output feature map equivalent; wherein, using the convolution processing unit to execute the convolution processing instruction includes: performing a first rearrangement transformation on the first convolution kernel to obtain a second convolution kernel, wherein the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), where roundup(×) is a round-up function; convolution processing is performed using the second input feature map and the second convolution kernel to obtain the second output feature map.

[0015] At least one embodiment of the present disclosure provides a processor, including an instruction parsing unit and a convolution processing unit, wherein the instruction parsing unit is used to receive and parse a convolution processing instruction, wherein the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; the convolution processing unit executes the data processing method described in any embodiment of the present disclosure after the instruction parsing unit parses the convolution processing instruction.

[0016] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transitorily stores computer-executable instructions; and a processor, which is configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the data processing method described in any embodiment of the present disclosure.

[0017] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the data processing method described in any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, but are not intended to limit the present disclosure.

[0019] Figure 1 A schematic diagram of the structure of a general-purpose graphics processing unit (GPGPU);

[0020] Figure 2 It is a schematic diagram of the structure of tensor data;

[0021] Figure 3A Schematic diagram of a convolutional layer performing multi-channel convolution operations;

[0022] Figure 3B A schematic diagram of using implicit GEMM to implement convolution;

[0023] Figure 4 A schematic flow chart of a data processing method provided for at least one embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of a segmentation and rearrangement transformation provided by at least one embodiment of the present disclosure;

[0025] Fig. 6A and Figure 6B A schematic diagram of a second rearrangement transformation provided for at least one embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of a data processing method provided by at least one embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of performing convolution processing on a first input feature map and a first convolution kernel in at least one embodiment of the present disclosure;

[0028] Fig. 9 A schematic diagram of a data processing device provided by at least one embodiment of the present disclosure;

[0029] Fig.10 A schematic flow chart of a data processing method provided for at least one embodiment of the present disclosure;

[0030] Fig.11 A schematic structural diagram of a processor provided for at least one embodiment of the present disclosure;

[0031] Fig.12 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure; and

[0032] Fig.13 A schematic block diagram of an electronic device provided according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0034] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing in front of the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits the detailed description of some known functions and known components.

[0035] A neural-network processing unit (NPU) or artificial intelligence chip is a type of microprocessor or computing system dedicated to hardware acceleration of artificial intelligence (especially artificial neural networks, machine vision, machine learning, etc.), sometimes also called an artificial intelligence accelerator (AI Accelerator). For example, it can be a graphics processor or a general-purpose graphics processor.

[0036] Figure 1 A schematic structural diagram of a general-purpose graphics processing unit (GPGPU) is shown.

[0037] like Figure 1 As shown, the general graphics processor includes an array of programmable multiprocessors. For example, the programmable multiprocessor may be a streaming processor cluster (SPC), for example, including Figure 1 The stream processor clusters 1, ..., and stream processor cluster M are shown, where M is a positive integer greater than 1. In a general-purpose graphics processor, one stream processor cluster processes one computing task, or multiple stream processor clusters process one computing task. Multiple stream processor clusters share data through a global cache or a global memory.

[0038] like Figure 1 As shown, taking stream processor cluster 1 as an example, one stream processor cluster includes multiple computing units, such as Figure 1In the figure, there are computing units 1, 2, ..., N, where N is a positive integer. Each computing unit (CU) is used to perform arithmetic and logical operations, such as accumulation, reduction, conventional addition, subtraction, multiplication, and division. A computing unit includes multiple cores (also called computing cores or computing cores), each of which includes an arithmetic logic unit (ALU), a floating-point computing unit, etc. The computing core is used to perform specific computing tasks. In addition, the computing unit also includes registers (e.g. Figure 1 The register file in the computing unit and the shared memory are used to hierarchically store source data and destination data related to computing tasks. The shared memory in a computing unit is used to share data between the cores of the computing unit.

[0039] In parallel computing, computing tasks are generally performed by multiple threads. These threads are divided into multiple thread blocks before being executed in a general-purpose graphics processor (or parallel computing processor), and then distributed through the thread block distribution module ( Figure 1 The CPU (not shown) distributes multiple thread blocks to each computing unit. All threads in a thread block must be assigned to the same computing unit for execution. At the same time, the thread block is split into minimum execution thread bundles (or simply thread bundles, warps), each of which contains a fixed number of threads (or less than this fixed number), for example, 32 threads. Multiple thread blocks can be executed in the same computing unit or in different computing units.

[0040] In each compute unit, the warp scheduling / dispatching module ( Figure 1 The thread warps are scheduled and allocated (not shown) so that multiple computing cores of the computing unit can run the thread warps. Depending on the number of computing cores in the computing unit, multiple thread warps in a thread block can be executed simultaneously or in time-sharing. Multiple threads in each thread warp will execute the same instruction. The memory execution instruction will be emitted to the shared memory in the computing unit or further emitted to the intermediate level cache or global cache or global memory for read and write operations, etc.

[0041] The tensor data used in the network model inference process is a three-dimensional tensor, including N dimension, H dimension and W dimension. The shape of the tensor data is specifically expressed as [n, h, w], where n, h, and w are positive integers. Here, the N dimension represents the batch size n of the tensor data, that is, the number of data samples captured in one training. The H dimension represents the height h of each batch, and the W dimension represents the width w of each batch.

[0042] For example, if the tensor data is an image, the specific value n of the N dimension is 1, the value h of the H dimension represents the height of the image, and the value w of the W dimension represents the width of the image.

[0043] Figure 2 is a schematic diagram of the structure of tensor data. Figure 2 As shown, there are N batches of data in the network model reasoning process, such as Figure 2 There are multiple large rectangular boxes in the figure: batch 0 (batch0), batch 1 (batch1), ... batch n-1 (batchn-1). A batch includes H×W elements. Figure 2 A small grid in a batch represents an element, for example, an element can have a data width of 8 bits, 16 bits, etc. Here, N, H, and W are positive integers.

[0044] The data in the network model reasoning process is usually divided into resources using a unified resource mapping logic to distribute the data reasoning tasks to different levels of computing resources for execution. Neural network processors can be used for reasoning of network models. For example, the network model can be a large language model, a visual language model, a multimodal model, etc.

[0045] Convolution operation is the core component of Convolutional Neural Network (CNN). Convolution operation extracts local features and generates output feature maps by sliding convolution kernel on input feature maps. Convolutional neural networks are further combined with activation functions and pooling operations, and are widely used in tasks such as image classification, object detection, and semantic segmentation. Multi-channel convolution refers to the convolution operation performed when processing data with multiple input channels in a convolutional neural network. This operation is commonly used in image processing because images usually have multiple color channels (such as RGB images), but it is also applicable to other multi-feature data.

[0046] Figure 3A Figure 1 shows a schematic diagram of a convolutional layer performing multi-channel convolution operations. Figure 3A As shown in the figure, M groups of R×S (“height×width”, the same below) convolution kernels with C channels are used to perform convolution operations on N batches of H×W input images (or input feature maps) with C channels, and N groups of E×F output feature maps with M channels are obtained respectively. Therefore, the output feature map generally includes multiple dimensions of F / E / M. Convolution operations have the characteristics of high parallelism and high data reuse. High parallelism is reflected in the fact that multiple convolution kernels can operate on multiple input feature maps simultaneously.

[0047] As mentioned above, for the convolution operation (convolution processing), the input feature map can be expressed as a tensor (N, C, H, W), where N represents the batch, C represents the number of channels, H represents the height of the input matrix, and W represents the width of the input matrix. For example, the input feature map corresponds to N RGB images, with 3 channels (C) corresponding to red, green, and blue respectively, and a single image has a height (H) and a width (W). The convolution kernel is one or more two-dimensional matrices, whose size is usually much smaller than the size of the input feature map. The convolution kernel slides on the input feature map with a certain stride (Stride) and performs dot product operations with local areas of the input data. For example, a stride of 1 means sliding one matrix element (such as pixel) each time, and a stride of 2 means sliding two matrix elements (such as pixels) each time. The convolution kernel can also be expressed as a tensor (M, C, R, S), where M represents the number of groups of convolution kernels, C represents the number of channels in each group of convolution kernels, R represents the height of the two-dimensional matrix in the convolution kernel, and S represents the width of the two-dimensional matrix in the convolution kernel. The result of the convolution processing is a new two-dimensional array, called the output feature map. Each group of convolution kernels is used to generate an output feature map, and multiple groups of convolution kernels are used to generate multi-channel output feature maps.

[0048] In convolution processing, padding is an optional processing method used to deal with the boundary problem of the input feature map; the convolution kernel slides on the input feature map that has been expanded by padding, calculates the dot product at each position, and generates an output feature map. Padding achieves a variety of purposes by adding additional values ​​(usually zero) to the edges of the input feature map (in the height and width directions). For example, by properly selecting the number of padded elements (P), the size of the output feature map can be made the same as the size of the input feature map; by padding, it can be ensured that the information on the edge can also be processed by the convolution; using padding can make the convolution neural model more flexible and maintain a consistent output size under different input sizes, thereby simplifying the design of the neural network.

[0049] For example, in the design of neural network processors, computationally intensive operations that account for a large proportion of the model, such as matrix multiplication and convolution, are usually assigned to hardware units such as tensor cores that are specifically used to accelerate calculations. However, tensor cores may not be able to natively support convolution calculations of large convolution kernels, so optimizing the convolution performance of large convolution kernels in the model is critical to improving the end-to-end performance of the model. The traditional method is to use general matrix multiplication (GEMM) or implicit general matrix multiplication (implicit GEMM). Implicit GEMM is a variant of GEMM. It still performs matrix multiplication (matmul) operations according to the output results processed by the im2col function, but there is no need to explicitly generate the output results after im2col to implement convolution operations under large convolution kernels. Large convolution kernels refer to larger-sized convolution kernels used in convolutional neural networks, such as convolution kernels with a kernel matrix size greater than 7, such as 16×16, 25×25, etc., which are usually used for tasks that need to capture a larger range of features in the input data, such as image classification, object detection, semantic segmentation, and other tasks. However, large convolution kernel calculations require more multiplication and addition operations, so the computational cost is higher.

[0050] Figure 3B Figure 2 shows a schematic diagram of using implicit GEMM to implement convolution. Figure 3B As shown in the figure, the corresponding convolution operation is to perform sliding window dot product calculation on the input N input feature maps of size C×H×W with M convolution kernels of size C×R×S, and obtain N output feature maps of size M×E×F. The implicit GEMM algorithm regards the convolution kernel as a two-dimensional matrix of size M×(C×R×S), and the input feature map is transformed into a two-dimensional matrix of size (C×R×S)×(N×H×W) according to the position of the convolution kernel sliding window, and then the two matrices are calculated by matrix multiplication to obtain the output feature map.

[0051] Generally, a neural network processor supports multi-channel convolution calculations. For example, a neural network processor is designed to support a maximum of 64 channels of parallel calculations, so that even if the actual number of channels of the processor is 3, the efficiency is the same as that of 64 channels. However, in actual convolution calculations, the 64 channels are usually not fully utilized. For example, even for 3-channel convolution calculations, the hardware of the neural network processor will be occupied during the operation, but 61 channels are actually idle. Therefore, in this case, the neural network processor is not fully utilized, which increases the cost of use.

[0052] In response to the above problems, the inventors of the present invention provide a data processing method and device, which utilizes the transformation of large convolution kernel convolution operations into small convolution kernel convolution operations. This can not only enable neural network processors that cannot natively support large convolution kernels to support large convolution kernels, but also increase the channel parallelism of the neural network processor and improve hardware utilization.

[0053] At least one embodiment of the present disclosure provides a data processing method and device, a processor, an electronic device, and a non-transitory computer-readable storage medium. The data processing method includes: receiving a first convolution kernel and performing a first rearrangement transformation to obtain a second convolution kernel, wherein the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, and the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; receiving a first input feature map, wherein the size of the first input feature map is expressed as (N, C1, H1, W1), N represents a batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), roundup(×) is a rounding function; performing convolution processing using the second input feature map and the second convolution kernel and obtaining a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by directly performing the convolution processing on the first input feature map and the first convolution kernel.

[0054] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0055] Figure 4 A schematic flowchart of a data processing method provided for at least one embodiment of the present disclosure.

[0056] like Figure 4 As shown, the data processing method provided by at least one embodiment of the present disclosure includes at least steps S10-S40.

[0057] S10: Receive the first convolution kernel and perform a first rearrangement transformation to obtain a second convolution kernel.

[0058] For example, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M is used to represent the number of the first convolution kernel, that is, the number of output channels, C1 is used to represent the number of channels of the first convolution kernel, that is, the number of input channels, R1 is used to represent the matrix height of the first convolution kernel, S1 is used to represent the matrix width of the first convolution kernel, that is, it represents M R1×S1 convolution kernels with C1 channels, and M, C1, R1 and S1 are all positive integers; the size of the second convolution kernel is expressed as (M, C2, R2, S2), M is used to represent the number of the second convolution kernel, C2 is used to represent the number of channels of the second convolution kernel, R2 is used to represent the matrix height of the second convolution kernel, S2 is used to represent the matrix width of the second convolution kernel, that is, it represents M R2×S2 convolution kernels with C2 channels, and M, C2, R2 and S2 are all positive integers.

[0059] For example, in some embodiments, step S10 may include: when R1 or S1 in the size of the first convolution kernel is a prime number, the first convolution kernel of size (M, C1, R1, S1) may be padded to obtain an intermediate convolution kernel of size (M, C1, R', S'), where M is used to represent the number of intermediate convolution kernels, C1 is used to represent the number of channels of the intermediate convolution kernel, R' is used to represent the matrix height of the intermediate convolution kernel, S' is used to represent the matrix width of the intermediate convolution kernel, and M, C1, R' and S' are all positive integers; the intermediate convolution kernel is split and rearranged to obtain a second convolution kernel of size (M, C2, R2, S2), R'=R2×roundup(R1 / R2), S'= S2× roundup(S1 / S2).

[0060] For example, when the matrix size in the first convolution kernel is 11×11, that is, R1 and S1 are equal and both are prime numbers, they can be aligned to 12×12 by padding processing (such as zero padding), and then each 12×12 kernel matrix is ​​divided into 9 4×4 second kernel matrices or 16 3×3 second kernel matrices to obtain a second convolution kernel with R2=S2=4 or R2=S2=3. For example, when the matrix size in the first convolution kernel is 13×11, that is, R1 and S1 are not equal and both are prime numbers, they can be aligned to 15×12 by padding processing (such as zero padding), and then each 15×12 kernel matrix is ​​divided into 9 5×4 second kernel matrices to obtain a second convolution kernel with R2=5 and S2=4. For example, when the matrix size in the first convolution kernel is 13×12, that is, R1 and S1 are not equal and R1 or S1 is a prime number, it can be aligned to 15×12 by padding processing (such as zero padding), and then each 15×12 kernel matrix is ​​divided into 9 5×4 second kernel matrices to obtain a second convolution kernel with R2=5 and S2=4. For example, step S10 may include: when R1 and S1 in the size of the first convolution kernel are not prime numbers, the first convolution kernel with a size of (M, C1, R1, S1) is directly divided and rearranged to obtain a second convolution kernel with a size of (M, C2, R2, S2).

[0061] For example, when R1=S1=25, that is, R1 and S1 are not prime numbers, the first convolution kernel consists of a 25×25 two-dimensional matrix, and each 25×25 kernel matrix can be directly divided into 25 5×5 second kernel matrices to obtain a second convolution kernel with R2=S2=5. For example, when R1=16, S2=9, that is, R1 and S1 are not prime numbers, the first convolution kernel consists of a 16×9 two-dimensional matrix, and each 16×9 kernel matrix can be directly divided into 12 4×3 second kernel matrices to obtain a second convolution kernel with R2=4, S2=3.

[0062] For example, the split and rearrangement transformation in step S10 may include: for each kernel matrix in the first convolution kernel or the intermediate convolution kernel, splitting the kernel matrix in a first order to obtain k second kernel matrices corresponding to the kernel matrix, and superimposing the k second kernel matrices in the k channels of the second convolution kernel in the first order to obtain the second convolution kernel, where k is a positive integer.

[0063] For example, for each kernel matrix in the first convolution kernel or the intermediate convolution kernel, determine the k second kernel matrices corresponding to the kernel matrix obtained by segmenting the kernel matrix; arrange the k second kernel matrices corresponding to each kernel matrix in the first convolution kernel or the intermediate convolution kernel according to the order of each kernel matrix to obtain the second convolution kernel, and arrange the k second kernel matrices corresponding to each kernel matrix according to the first order.

[0064] For example, a kernel matrix is ​​a two-dimensional matrix, and a single or multiple kernel matrices are combined into a convolution kernel on a single or multiple channels.

[0065] For example, the first order may be the sliding order of the convolution kernel in the convolution operation.

[0066] Figure 5 A schematic diagram of a segmentation and rearrangement transformation provided by at least one embodiment of the present disclosure. Figure 5 As shown, the left side shows a kernel matrix (R1, S1) in the first convolution kernel, and the kernel matrix is ​​divided into k second kernel matrices in the first order, for example Figure 5 The rectangular boxes numbered 1, 2, 3, ..., 16 in Figure 5 In the example, k=16, but the present disclosure is not limited thereto. Figure 5 The k second kernel matrices (R2, S2) on the right side of the convolution kernel are sequentially superimposed on the k channels of the second convolution kernel according to the first order.

[0067] Each kernel matrix (R1, S1) in the first convolution kernel can be passed through Figure 5 The split rearrangement transformation in obtains the second convolution kernel, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)).

[0068] For example, the first rearrangement transformation can be performed offline. Since different input feature maps can use the first convolution kernel for convolution operations, in each data processing, it is only necessary to perform the first rearrangement transformation on the first convolution kernel once to obtain the second convolution kernel. After that, the second convolution kernel can be used all the time to save computing resources.

[0069] For example, the second convolution kernel obtained after the first rearrangement transformation can be stored for convolution processing, without having to perform the first rearrangement transformation again each time a convolution operation is performed, thereby saving computing resources.

[0070] For example, the weight update in model training is based on the weight update of the second convolution kernel obtained after the first rearrangement transformation. After the model training is completed, the second convolution kernel needs to be subjected to an inverse first rearrangement transformation. The inverse first rearrangement transformation is the reverse process of the first rearrangement transformation, that is, the second convolution kernel is transformed to obtain the first convolution kernel again through the inverse first rearrangement transformation.

[0071] For example, the inverse first rearrangement transformation includes concatenating multiple second kernel matrices into a kernel matrix in the first order, and further arranging the kernel matrices in the order between them to obtain the first convolution kernel again.

[0072] S20: Receive a first input feature map.

[0073] For example, the size of the first input feature map is expressed as (N, C1, H1, W1), where N is used to represent the batch of the first input feature map, C1 is used to represent the number of channels of the first input feature map, H1 is used to represent the matrix height of the first input feature map, and W1 is used to represent the matrix width of the first input feature map. The number of channels of the first input feature map is equal to the number of channels of the first convolution kernel, and N, C1, H1 and W1 are all positive integers.

[0074] S30: Perform a second rearrangement transformation on the first input feature map to obtain a second input feature map.

[0075] For example, the size of the second input feature map is expressed as (N, C2, H2, W2), where N is used to represent the batch of the second input feature map, C2 is used to represent the number of channels of the second input feature map, H2 is used to represent the matrix height of the second input feature map, and W2 is used to represent the matrix width of the second input feature map. N, C2, H2, and W2 are all positive integers; C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), and roundup(×) is a round-up function.

[0076] For example, in some embodiments, the input feature map needs to be padded, and step S30 may include: padding the first input feature map of size (N, C1, H1, W1) to obtain an intermediate input feature map of size (N, C1, H', W'), where N is used to represent the batch of the intermediate input feature map, C1 is used to represent the number of channels of the intermediate input feature map, H' is used to represent the matrix height of the intermediate input feature map, and W' is used to represent the matrix width of the intermediate input feature map, and N, C1, H' and W' are all positive integers; H'=H1+2P, W'=W1+2P, P is the number of padding elements, and then performing a second rearrangement transformation on the intermediate input feature map to obtain a second input feature map of size (N, C2, H2, W2).

[0077] For example, in some other embodiments, there is no need to perform padding processing on the first input feature map. For example, step S30 may include: directly performing a second rearrangement transformation on the first input feature map to obtain a second input feature map.

[0078] For example, the second rearrangement transformation can be: for each first input matrix in the first input feature map or the intermediate input feature map, determine the kernel matrix corresponding to the first input matrix, take out the elements of the convolution calculation of the k second kernel matrices corresponding to the kernel matrix in the first input matrix, rearrange to obtain k second input matrices corresponding one-to-one to the k second kernel matrices, and superimpose the k second input matrices in the k channels of the second input feature map in the first order to obtain the second input feature map.

[0079] For example, the kernel matrix corresponding to the first input matrix refers to the kernel matrix that performs convolution calculation with the first input matrix.

[0080] For example, the first input matrix is ​​one of the two-dimensional matrices constituting the first input feature map, and the second input matrix is ​​the elements corresponding to the convolution operation between the first input matrix and a second kernel matrix.

[0081] Fig. 6A and Figure 6B A second rearrangement transformation schematic diagram provided for at least one embodiment of the present disclosure. Fig. 6A and Figure 6B In , the first input matrix is ​​a (H1, W1) two-dimensional matrix in a channel of a batch in the first input feature map.

[0082] Fig. 6A The second input matrix 1 is shown Fig. 6A The second input matrix 1 shown in the bold box) and the second kernel matrix 1 ( Fig. 6A The kernel matrix in the first convolution kernel is divided into the second kernel matrix located in the upper left corner). The second input matrix 1 is the element corresponding to the convolution operation of the first input matrix and the second kernel matrix 1. Specifically, the element is the element used to perform the convolution operation with the second kernel matrix 1. The upper left corner element of the second input matrix corresponding to different second kernel matrices has a different position in the first input matrix, and the position is related to the position of the second kernel matrix in the kernel matrix of the first convolution kernel.

[0083] Figure 6B The second input matrix 2 is shown Figure 6B The second input matrix 2 shown in the bold box) and the second kernel matrix 2 ( Figure 6B The second input matrix 2 is the element corresponding to the convolution operation between the first input matrix and the second kernel matrix 2.

[0084] according to Fig. 6A and Figure 6B In this way, the elements corresponding to the convolution calculation of the first input matrix and each second kernel matrix are taken out, and 16 second input matrices are rearranged to obtain the 16 second input matrices, and the 16 second input matrices are superimposed on the k channels of the second input feature map according to the first order to complete the second rearrangement transformation. For example, the second input feature map is obtained by performing the second rearrangement transformation on the first input feature map online, and each first input matrix of the first input feature map needs to undergo a second rearrangement transformation.

[0085] S40: Perform convolution processing using the second input feature map and the second convolution kernel to obtain a second output feature map.

[0086] For example, the second output feature map is equivalent to the first output feature map obtained by directly performing convolution processing on the first input feature map and the first convolution kernel. Direct convolution processing refers to performing conventional convolution calculations, and equivalence includes numerical equivalence between the first output feature map and the second output map, and numerical equivalence refers to that the pixel values ​​of the two feature maps at each position are exactly the same.

[0087] It should be noted that the present disclosure does not limit the sequence of S10 - S40 .

[0088] Figure 7 This is a schematic diagram of a data processing method provided by an embodiment of the present disclosure. Figure 7 right Figure 4 S10-S40 are further explained.

[0089] like Figure 7 As shown, in step S10, a first rearrangement transformation is performed on the first convolution kernel to obtain a second convolution kernel.

[0090] because Figure 7 The first convolution kernel in (32, 3, 16, 16) is a non-prime number, so there is no need to fill the first convolution kernel, and it can be directly split and rearranged. Figure 5 The method shown in FIG. 1 divides the 16×16 kernel matrix in the first convolution kernel into 16 4×4 second kernel matrices according to the first order, and superimposes the 16 second kernel matrices on the 16 channels of the second convolution kernel according to the first order, for example, inserting the 16 second kernel matrices into the 16 channels of the second convolution kernel according to the first order. Thus, each kernel matrix in the first convolution kernel is Figure 5 The segmentation and rearrangement transformation shown can obtain the second convolution kernel (32, 48, 4, 4).

[0091] In steps S20 and S30, a first input feature map is received, and a second rearrangement transformation is performed on the first input feature map to obtain a second input feature map.

[0092] First, the first input feature map (16, 3, 512, 512) is padded with p=7 (i.e., 7×7 padding) to obtain an intermediate input feature map (16, 3, 526, 526), ​​which is composed of the first input matrix of 526×526.

[0093] Next, follow Fig. 6A and Figure 6BThe method shown performs a second rearrangement transformation, that is, sequentially taking out the corresponding elements of each first input matrix that are convolved with each second kernel matrix, and rearranging them to obtain 16 second input matrices, and the above 16 second input matrices are superimposed on the 16 channels of the second input feature map in the first order, for example, the 16 second input matrices are inserted into the 16 channels of the second input feature map in the first order; in addition, the 16 second input matrices corresponding to the first input matrices of the three channels are arranged in channel order, thereby obtaining the second input feature map (16, 48, 514, 514).

[0094] In step S40, convolution processing is performed using the second input feature map and the second convolution kernel to obtain a second output feature map.

[0095] Convolution is performed using the second input feature map (16, 48, 514, 514) and the second convolution kernel (32, 48, 4, 4) to obtain a second output feature map (16, 32, 512, 512).

[0096] Figure 8 A schematic diagram of performing convolution processing on a first input feature map and a first convolution kernel in at least one embodiment of the present disclosure.

[0097] like Figure 8 As shown in the figure, the first input feature map (16, 32, 512, 512) and the first convolution kernel (32, 3, 16, 16) are convolved after padding 7×7 (not shown in the figure) to obtain the output result of (16, 32, 512, 512). It can be seen that the result is similar to Figure 7 The second output feature map obtained in S40 is equivalent.

[0098] The disclosed embodiment also includes using a convolution processing device to perform the data processing method provided in the above embodiment. The convolution processing device is configured to process Cm channels in parallel, Cm is greater than or equal to C2; the convolution processing device is configured to process a maximum convolution kernel size Km for a single time, the maximum value between R2 and S2 is less than or equal to Km, and the minimum value between R1 and S1 is greater than Km; it is used to increase the channel dimension and improve the utilization rate of hardware calculation, thereby improving the overall performance of the operator and speeding up the calculation speed.

[0099] For example, the convolution processing device includes a hardware unit such as a tensor core that is specifically used for accelerating calculations, and the channel dimension of the tensor core varies with different GPU architectures. For example, the number of channels of the first input feature map can be subjected to different second rearrangement transformations according to different tensor core channel dimensions (expressed as Cm) to obtain the number of channels of the corresponding second input feature map (expressed as C2), as long as Cm is greater than or equal to C2; for example, the size of the first convolution kernel (expressed as (R1, S1)) can be subjected to different first rearrangement transformations according to the maximum convolution kernel size that can be processed at a time by different tensor cores (expressed as Km) to obtain the corresponding second convolution kernel size (expressed as (R2, S2)), as long as the maximum value between R2 and S2 is less than or equal to Km, and the minimum value between R1 and S1 is greater than Km.

[0100] The data processing method provided by at least one embodiment of the present disclosure improves the overall performance of the operator and speeds up the calculation by converting large convolution kernels into small convolution kernels; and increases the number of channels of convolution operations with a small number of input layer channels through transformation, thereby avoiding idleness of some computing resources as much as possible and improving the utilization rate of hardware computing.

[0101] Fig. 9 A schematic block diagram of a data processing device provided by at least one embodiment of the present disclosure. Fig. 9 As shown, the data processing device 10 provided in the embodiment of the present disclosure includes a first rearrangement unit 11, a receiving unit 12, a second rearrangement unit 13 and a processing unit 14.

[0102] The first rearrangement unit 11 is configured to receive the first convolution kernel and perform a first rearrangement transformation to obtain a second convolution kernel. The size of the first convolution kernel is expressed as (M, C1, R1, S1), where M is used to represent the number of the first convolution kernel, C1 is used to represent the number of channels of the first convolution kernel, R1 is used to represent the matrix height of the first convolution kernel, and S1 is used to represent the matrix width of the first convolution kernel. The size of the second convolution kernel is expressed as (M, C2, R2, S2), where M is used to represent the number of the second convolution kernel, C2 is used to represent the number of channels of the second convolution kernel, R2 is used to represent the matrix height of the second convolution kernel, and S2 is used to represent the matrix width of the second convolution kernel. The receiving unit 12 is configured to receive a first input feature map, the size of the first input feature map is expressed as (N, C1, H1, W1), N is used to represent the batch of the first input feature map, C1 is used to represent the number of channels of the first input feature map, H1 is used to represent the matrix height of the first input feature map, W1 is used to represent the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; The second rearrangement unit 13 is configured to perform a second rearrangement transformation on the first input feature map to obtain a second input feature map, the size of the second input feature map is expressed as (N, C2, H2, W2), N is used to represent the batch of the second input feature map, C2 is used to represent the number of channels of the second input feature map, H2 is used to represent the matrix height of the second input feature map, W2 is used to represent the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))×(roundup(S1 / S2)), roundup(×) is a round-up function; The processing unit 14 is configured to perform convolution processing using the second input feature map and the second convolution kernel and obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by performing the convolution processing on the first input feature map and the first convolution kernel.

[0103] For the relevant contents of the first rearrangement unit 11, the receiving unit 12, the second rearrangement unit 13 and the processing unit 14, reference may be made to the embodiments of the above-mentioned data processing method. Figure 4 The description of steps S10-S40 is omitted here.

[0104] Above Fig. 9 The various modules shown may be implemented, for example, by software, hardware, firmware or any combination thereof.

[0105] It should be noted that the first input feature map, the first convolution kernel, the second input feature map, and the second convolution kernel in the data processing method or data processing device provided in at least one embodiment of the present disclosure may have different specific physical meanings according to different application scenarios. For example, the data processing method provided in at least one embodiment of the present disclosure may be applied in the fields of speech processing, image processing, text processing, video processing, etc.

[0106] For example, in the field of speech processing, the first input feature map, the first convolution kernel, the second input feature map, and the second convolution kernel can be parameters involved in convolution processing in tasks such as feature extraction, speech enhancement, and speech recognition.

[0107] For example, in the field of image processing, the first input feature map, the first convolution kernel, the second input feature map, and the second convolution kernel can be parameters involved in convolution processing in tasks such as image recognition, feature extraction, image segmentation, target detection, image classification, and scene reconstruction.

[0108] For example, in the field of text processing, the first input feature map, the first convolution kernel, the second input feature map, and the second convolution kernel can be parameters involved in convolution processing in tasks such as text classification, sentiment analysis, and text generation.

[0109] For example, in the field of video processing, the first input feature map, the first convolution kernel, the second input feature map and the second convolution kernel can be the relevant parameters in the image processing field as described above, or the parameters involved in convolution processing in the video processing field such as video classification and action recognition.

[0110] Of course, the present disclosure is not limited to this, and for other application scenarios or fields, as long as convolution processing is required, the data processing method described in at least one embodiment of the present disclosure can be applied, and will not be described one by one here.

[0111] At least one embodiment of the present disclosure also provides a data processing method. Fig.10 A schematic flowchart of a data processing method provided for at least one embodiment of the present disclosure.

[0112] like Fig.10 As shown, the data processing method at least includes steps S50-S60.

[0113] In step S50, a convolution processing instruction is received.

[0114] For example, a convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, H1 represents the matrix height of the first input feature map, W1 is used to represent the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel.

[0115] In step S60, after parsing the convolution processing instruction, the convolution processing unit is used to execute the convolution processing instruction to obtain a second output feature map.

[0116] For example, the second output feature map is equivalent to the first output feature map obtained by directly performing convolution processing on the first input feature map and the first convolution kernel.

[0117] For example, in step S60, using the convolution processing unit to execute the convolution processing instruction includes: performing a first rearrangement transformation on the first convolution kernel to obtain a second convolution kernel, wherein the size of the second convolution kernel is expressed as (M, C2, R2, S2), C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), roundup(×) is a round-up function; using the second input feature map and the second convolution kernel to perform convolution processing and obtain a second output feature map.

[0118] For example, the data processing method provided by at least one embodiment of the present disclosure can be applied to Fig.11 Processor shown.

[0119] For example, in the data processing method provided by at least one embodiment of the present disclosure, a convolution processing instruction is provided, and the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters.

[0120] For example, after receiving a convolution processing instruction, the processor parses the convolution processing instruction, such as decoding the convolution processing instruction, generates a microinstruction and sends the microinstruction to an instruction allocation unit; the instruction allocation unit sends the microinstruction to a corresponding scheduling queue according to the microinstruction category; in response to the microinstruction, when the input parameters are prepared, the convolution processing unit reads and executes the related operations of the convolution processing instruction.

[0121] Regarding the specific process of using the convolution processing unit to execute the convolution processing instruction, reference may be made to steps S10-S40 in the data processing method described above, and the repeated parts will not be repeated here.

[0122] The data processing method provided by at least one embodiment of the present disclosure can achieve a technical effect similar to the data processing method described above, and will not be described in detail here.

[0123] Fig.11 A schematic structural diagram of a processor provided in at least one embodiment of the present disclosure. Fig.11 As shown, the processor 400 includes an instruction parsing unit 401 and a convolution processing unit 402 .

[0124] For example, the instruction parsing unit 401 is used to receive and parse a convolution processing instruction, wherein the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel.

[0125] For example, after the instruction parsing unit 401 parses the convolution processing instruction, the convolution processing unit 402 executes the data processing method described in any embodiment of the present disclosure.

[0126] Specifically, when the upper-layer software based on the processor (for example, AI applications, HPC applications, and scientific computing applications, etc.) can send a convolution processing instruction for computing processing to the processor (for example, CPU or GPU) through a uniformly encapsulated function library, the convolution processing instruction can carry the first convolution kernel and the first input feature map as input parameters; when the processor 400 receives the convolution processing instruction, the instruction parsing unit 401 parses the convolution processing instruction to obtain the input parameters, and the processor 400 schedules the convolution processing unit 402 to perform the data processing task of the input parameters. For example, after parsing the convolution processing instruction, the processor can store the input parameters in the convolution processing instruction into a register or memory, so that when the convolution processing unit 402 performs computing processing, it can obtain the input parameters from the register or memory.

[0127] Regarding the specific process of using the convolution processing unit 402 to execute the convolution processing instruction, reference may be made to steps S10-S40 in the data processing method described above, and the repeated parts will not be repeated here.

[0128] Fig.12 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Fig.12 As shown, the storage medium 200 may be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 201 may be non-transitory stored on the storage medium 200. For example, when the computer-readable instructions 201 are executed by a processor, one or more steps in the data processing method described above may be executed.

[0129] For example, the storage medium 200 may be applied in an electronic device. For example, the storage medium 200 may include a memory in the electronic device.

[0130] For example, the storage device may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory, etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. One or more computer-readable instructions may be stored on the computer-readable storage medium, and the data processor may run the computer-readable instructions to implement various functions of the data processor. Various applications and various data may also be stored in the storage medium.

[0131] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other applicable storage media.

[0132] Fig.13 The following is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Fig.13 As shown, the electronic device 300 is suitable for implementing the data processing method provided by the embodiment of the present disclosure. It should be noted that Fig.13 The components of the electronic device 300 shown are merely exemplary and non-limiting. The electronic device 300 may also have other components according to actual application requirements.

[0133] For example, Fig.13 As shown, in some examples, the electronic device 300 includes a processing device (e.g., a graphics processor, a general-purpose graphics processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303 to achieve various functions, such as executing the data processing method provided by at least one embodiment of the present disclosure.

[0134] For example, when the computer readable instructions are executed by the processing device 301, one or more steps in the data processing method described in any of the above embodiments may be executed. It should be noted that for a detailed description of the processing process of the data processing method, reference may be made to the relevant description in the above data processing method embodiment, and the repetitive parts will not be repeated.

[0135] Various programs and data required for the operation of the computer system are also stored in the RAM 303 . The processing device 301 , the ROM 302 , and the RAM 303 are connected to each other via a bus 304 . An input / output (I / O) interface 305 is also connected to the bus 304 .

[0136] For example, the memory may include any combination of one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory (cache), etc. For example, computer-readable instructions may be loaded from storage device 308 into random access memory (RAM) 303 to run computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, hard disk, erasable programmable read-only memory (EPROM), portable compact disk read-only memory (CD-ROM), USB memory, flash memory, etc. Various applications and various data, such as style images, and various data used and / or generated by applications, etc., may also be stored in the computer-readable storage medium.

[0137] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, a flash memory, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other electronic devices wirelessly or by wire to exchange data. Although Fig.13An electronic device 300 having various devices is shown, but it should be understood that it is not required to implement or have all the devices shown, and the electronic device 300 may be implemented or have more or fewer devices instead. For example, the processor 301 can control other components in the electronic device 300 to perform the desired functions. The processing device 301 can be a device with data processing capabilities and / or program execution capabilities such as a central processing unit (CPU), a tensor processing unit (TPU) or a graphics processing unit (GPU). The central processing unit can be an X86 or ARM architecture, etc. The graphics processor can be directly integrated into the motherboard separately, or built into the north bridge chip of the motherboard. The graphics processor can also be built into the central processing unit.

[0138] For example, the electronic device 300 may further include a peripheral interface (not shown in the figure), etc. The peripheral interface may be various types of interfaces, such as a USB interface, a lightning interface, etc. The communication device 309 may communicate with a network and other devices through wireless communication, such as the Internet, an intranet and / or a wireless network such as a cellular phone network, a wireless local area network (LAN) and / or a metropolitan area network (MAN). Wireless communication may use any of a variety of communication standards, protocols, and techniques, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0139] For example, the electronic device 300 may be any device such as a mobile phone, a tablet computer, a laptop computer, an e-book, a game console, a television, a digital photo frame, a navigator, etc., or may be any combination of electronic devices and hardware, which is not limited in the embodiments of the present disclosure.

[0140] For example, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned data processing method defined in the method of the embodiment of the present disclosure is executed.

[0141] It should be noted that the computer-readable medium of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In an embodiment of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, device or device. In an embodiment of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than computer readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0142] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0143] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0144] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0145] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0146] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0147] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

[0148] There are a few points to note about this disclosure: (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to the general design.

[0149] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to obtain new embodiments.

[0150] The above description is only a specific implementation of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A data processing method, comprising: Receive a first convolution kernel and perform a first rearrangement transformation to obtain a second convolution kernel, wherein the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, and the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; Receive a first input feature map, wherein the size of the first input feature map is expressed as (N, C1, H1, W1), N represents a batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; Performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), and roundup(×) is a round-up function; Convolution processing is performed using the second input feature map and the second convolution kernel to obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by directly performing the convolution processing on the first input feature map and the first convolution kernel.

2. The data processing method according to claim 1, further comprising: The obtained second convolution kernel is stored for use in the convolution process.

3. The data processing method according to claim 1, wherein: Performing the first rearrangement transformation to obtain the second convolution kernel includes: In response to R1 or S1 being a prime number, padding the first convolution kernel to obtain an intermediate convolution kernel, and performing a segmentation and rearrangement transformation on the intermediate convolution kernel to obtain the second convolution kernel, wherein the size of the intermediate convolution kernel is expressed as (M, C1, R', S'), M represents the number of the intermediate convolution kernels, C1 represents the number of channels of the intermediate convolution kernel, R' represents the matrix height of the intermediate convolution kernel, S' represents the matrix width of the intermediate convolution kernel, R'=R2× roundup(R1 / R2), S'= S2× roundup(S1 / S2); In response to the fact that both R1 and S1 are not prime numbers, the first convolution kernel is directly subjected to the splitting and rearrangement transformation to obtain the second convolution kernel.

4. The data processing method according to claim 3, wherein: The segmentation and rearrangement transformation comprises: For each kernel matrix in the first convolution kernel or the intermediate convolution kernel, the kernel matrix is ​​divided according to a first order to obtain k second kernel matrices corresponding to the kernel matrix, and the k second kernel matrices are superimposed on the k channels of the second convolution kernel according to the first order, thereby obtaining the second convolution kernel, where k is a positive integer.

5. The data processing method according to any one of claims 2 to 4, wherein: The first rearrangement transformation is performed offline to obtain the second convolution kernel from the first convolution kernel.

6. The data processing method according to claim 4, wherein: Performing a second rearrangement transformation on the first input feature map to obtain a second input feature map includes: Perform padding processing on the first input feature map to obtain an intermediate input feature map, wherein the size of the intermediate input feature map is expressed as (N, C1, H', W'), N represents the batch of the intermediate input feature map, C1 represents the number of channels of the intermediate input feature map, H' represents the matrix height of the intermediate input feature map, W' represents the matrix width of the intermediate input feature map, H'=H1+2P, W'=W1+2P, P is the number of padding elements, and then perform the second rearrangement transformation on the intermediate input feature map to obtain the second input feature map; or, The second input feature map is directly subjected to the second rearrangement transformation to obtain the second input feature map.

7. The data processing method according to claim 6, wherein: Performing the second rearrangement transformation on the first input feature map or the intermediate input feature map to obtain the second input feature map includes: For each first input matrix in the first input feature map or the intermediate input feature map, determine the kernel matrix corresponding to the first input matrix, take out the elements of the first input matrix corresponding to the kernel matrix for convolution calculation, rearrange them to obtain k second input matrices corresponding one-to-one to the k second kernel matrices, and superimpose the k second input matrices in the k channels of the second input feature map in the first order to obtain the second input feature map.

8. The data processing method according to claim 6, wherein: The second rearrangement transformation is performed online on the first input feature map to obtain the second input feature map.

9. The data processing method according to claim 1, further comprising: A convolution processing device is used to perform the data processing method, wherein the convolution processing device is configured to process Cm channels in parallel, Cm is greater than or equal to C2, the convolution processing device is configured so that the maximum convolution kernel size for a single processing is Km, the maximum value between R2 and S2 is less than or equal to Km, the minimum value between R1 and S1 is greater than Km, and Km and Cm are positive integers.

10. A data processing device comprising The first rearrangement unit is configured to receive the first convolution kernel and perform a first rearrangement transformation to obtain a second convolution kernel, wherein: The size of the first convolution kernel is expressed as (M, C1, R1, S1), where M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, and S1 represents the matrix width of the first convolution kernel. The size of the second convolution kernel is expressed as (M, C2, R2, S2), where M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel. A receiving unit is configured to receive a first input feature map, wherein the size of the first input feature map is expressed as (N, C1, H1, W1), N represents a batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; A second rearrangement unit is configured to perform a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents a batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), and roundup(×) is a round-up function; A processing unit is configured to perform convolution processing using the second input feature map and the second convolution kernel and obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by performing the convolution processing on the first input feature map and the first convolution kernel.

11. A data processing method, comprising: Receive a convolution processing instruction, wherein the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 is used to represent the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; After parsing the convolution processing instruction, executing the convolution processing instruction using a convolution processing unit to obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by directly performing convolution processing on the first input feature map and the first convolution kernel; Wherein, using the convolution processing unit to execute the convolution processing instruction includes: Performing a first rearrangement transformation on the first convolution kernel to obtain a second convolution kernel, wherein the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; Performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), and roundup(×) is a round-up function; Convolution processing is performed using the second input feature map and the second convolution kernel to obtain the second output feature map.

12. A processor comprising an instruction parsing unit and a convolution processing unit, wherein: The instruction parsing unit is used to receive and parse a convolution processing instruction, wherein the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; The convolution processing unit executes the data processing method according to any one of claims 1 to 9 after the instruction parsing unit parses the convolution processing instruction.

13. An electronic device comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions, Wherein, when the computer executable instructions are executed by the processor, the data processing method according to any one of claims 1 to 9 or the data processing method according to claim 11 is implemented.

14. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, When the computer executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 9 or the data processing method according to claim 11 is implemented.

Images