Data processing method and device, processor, electronic device, and storage medium

Abstract

A data processing method and apparatus, processor, electronic device, and non-transitory computer-readable storage medium. The data processing method comprises: receiving a first convolution kernel and performing a first rearrangement transformation to obtain a second convolution kernel; receiving a first input feature map, performing a second rearrangement transformation on the first input feature map to obtain a second input feature map; and performing convolution processing on the second input feature map and a second convolution kernel to obtain a second output feature map, the second output feature map being equivalent to the first output feature map obtained by directly performing convolution processing on the first input feature map and the first convolution kernel. This method improves the overall performance of the operator and minimizes idleness of some computing resources, thereby improving hardware computing utilization.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to a data processing method and apparatus, a processor, an electronic device, and a non-transitory computer-readable storage medium. Background Art

[0002] In current hardware, data processing integrated circuits (ICs) such as graphics processing units (GPUs) and general-purpose graphics processing units (GPGPUs) can execute programs to complete various tensor-related computational tasks. As computing demands increase, hardware devices have increasingly demanding performance from operators. Better performance means faster computations and reduced computational overhead. Balancing computational tasks across hardware devices and maximizing their parallel execution efficiency are key to improving performance.

[0003] In AI models used for image processing, convolution is an operator that compresses raw image information and features and is widely used in most common image processing tasks (such as vision tasks). Furthermore, large-scale convolutions are often applied to the input layer of vision models to perform initial information compression for processing in subsequent network layers. Summary of the Invention

[0004] At least one embodiment of the present disclosure provides a data processing method, including: receiving a first convolution kernel and performing a first rearrangement transformation to obtain a second convolution kernel, wherein the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, and the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; receiving a first input feature map, wherein the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; the first input feature map is subjected to a second rearrangement transformation to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), where roundup(×) is a rounding-up function; performing convolution processing on the second input feature map and the second convolution kernel to obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by directly performing the convolution processing on the first input feature map and the first convolution kernel.

[0005] For example, the data processing method provided by at least one embodiment of the present disclosure further includes: storing the obtained second convolution kernel for use in the convolution processing.

[0006] For example, in the data processing method provided in at least one embodiment of the present disclosure, the first rearrangement transformation is performed to obtain the second convolution kernel, including: in response to R1 or S1 being a prime number, padding the first convolution kernel to obtain an intermediate convolution kernel, and performing a splitting and rearrangement transformation on the intermediate convolution kernel to obtain the second convolution kernel, wherein the size of the intermediate convolution kernel is expressed as (M, C1, R', S'), M represents the number of the intermediate convolution kernels, C1 represents the number of channels of the intermediate convolution kernel, R' represents the matrix height of the intermediate convolution kernel, S' represents the matrix width of the intermediate convolution kernel, R'=R2×roundup(R1 / R2), S'= S2× roundup(S1 / S2); in response to R1 and S1 being not prime numbers, directly performing the splitting and rearrangement transformation on the first convolution kernel to obtain the second convolution kernel.

[0007] For example, in the data processing method provided by at least one embodiment of the present disclosure, the split and rearrangement transformation includes: for each kernel matrix in the first convolution kernel or the intermediate convolution kernel, splitting the kernel matrix according to a first order to obtain k second kernel matrices corresponding to the kernel matrix, and superimposing the k second kernel matrices in the k channels of the second convolution kernel according to the first order, thereby obtaining the second convolution kernel, where k is a positive integer.

[0008] For example, in the data processing method provided by at least one embodiment of the present disclosure, the first rearrangement transformation is performed offline to obtain the second convolution kernel from the first convolution kernel.

[0009] For example, in the data processing method provided by at least one embodiment of the present disclosure, the first input feature map is subjected to a second rearrangement transformation to obtain a second input feature map, including: performing padding processing on the first input feature map to obtain an intermediate input feature map, wherein the size of the intermediate input feature map is expressed as (N, C1, H', W'), N represents the batch of the intermediate input feature map, C1 represents the number of channels of the intermediate input feature map, H' represents the matrix height of the intermediate input feature map, W' represents the matrix width of the intermediate input feature map, H'=H1+2P, W'=W1+2P, P is the number of padding elements, and then performing the second rearrangement transformation on the intermediate input feature map to obtain the second input feature map; or, directly performing the second rearrangement transformation on the first input feature map to obtain the second input feature map.

[0010] For example, in the data processing method provided by at least one embodiment of the present disclosure, the second rearrangement transformation is performed on the first input feature map or the intermediate input feature map to obtain the second input feature map, including: for each first input matrix in the first input feature map or the intermediate input feature map, determining the kernel matrix corresponding to the first input matrix, taking out the elements of the first input matrix that perform convolution calculation on the k second kernel matrices corresponding to the kernel matrix, rearranging to obtain k second input matrices that correspond one-to-one to the k second kernel matrices, and superimposing the k second input matrices in the k channels of the second input feature map in the first order to obtain the second input feature map.

[0011] For example, in the data processing method provided in at least one embodiment of the present disclosure, the second input feature map is obtained by performing the second rearrangement transformation on the first input feature map online.

[0012] For example, the data processing method provided by at least one embodiment of the present disclosure also includes: using a convolution processing device to perform the data processing method, wherein the convolution processing device is configured to process Cm channels in parallel, Cm is greater than or equal to C2, and the convolution processing device is configured so that the maximum convolution kernel size of a single processing is Km, the maximum value between R2 and S2 is less than or equal to Km, the minimum value between R1 and S1 is greater than Km, and Km and Cm are positive integers.

[0013] At least one embodiment of the present disclosure provides a data processing device, comprising: a first rearrangement unit, configured to receive a first convolution kernel and perform a first rearrangement transformation to obtain a second convolution kernel, wherein the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, and the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; a receiving unit, configured to receive a first input feature map, wherein the size of the first input feature map is expressed as (N, C1 , H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; the second rearrangement unit is configured to perform a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), roundup(×) is a rounding-up function; a processing unit is configured to perform convolution processing using the second input feature map and the second convolution kernel and obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by performing the convolution processing on the first input feature map and the first convolution kernel.

[0014] At least one embodiment of the present disclosure provides a data processing method, comprising: receiving a convolution processing instruction, wherein the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 is used to represent the matrix width of the first input feature map, the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; after parsing the convolution processing instruction, using a convolution processing unit to execute the convolution processing instruction to obtain a second output feature map, wherein the second output feature map is the same as the output feature map obtained by the convolution processing unit. The first output feature map obtained by directly performing convolution processing on the first input feature map and the first convolution kernel is equivalent; wherein, using the convolution processing unit to execute the convolution processing instruction includes: performing a first rearrangement transformation on the first convolution kernel to obtain a second convolution kernel, wherein the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), where roundup(×) is a round-up function; convolution processing is performed using the second input feature map and the second convolution kernel to obtain the second output feature map.

[0015] At least one embodiment of the present disclosure provides a processor, including an instruction parsing unit and a convolution processing unit, wherein the instruction parsing unit is used to receive and parse a convolution processing instruction, wherein the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; after the instruction parsing unit parses the convolution processing instruction, the convolution processing unit executes the data processing method described in any embodiment of the present disclosure.

[0016] At least one embodiment of the present disclosure provides an electronic device, comprising: a memory, which non-transiently stores computer-executable instructions; and a processor, configured to execute the computer-executable instructions, wherein the computer-executable instructions, when executed by the processor, implement the data processing method described in any embodiment of the present disclosure.

[0017] At least one embodiment of the present disclosure provides a non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, the data processing method described in any embodiment of the present disclosure is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0019] Figure 1 A schematic diagram of the structure of a general-purpose graphics processing unit (GPGPU);

[0020] Figure 2 A schematic diagram of the structure of tensor data;

[0021] Figure 3A Schematic diagram of a convolutional layer performing multi-channel convolution operations;

[0022] Figure 3B Schematic diagram of using implicit GEMM to implement convolution;

[0023] Figure 4 A schematic flow chart of a data processing method provided in at least one embodiment of the present disclosure;

[0024] Figure 5 A schematic diagram of a split-rearrangement transformation provided by at least one embodiment of the present disclosure;

[0025] Figure 6A and Figure 6B A schematic diagram of a second rearrangement transformation provided for at least one embodiment of the present disclosure;

[0026] Figure 7 A schematic diagram of a data processing method provided by at least one embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of performing convolution processing on a first input feature map and a first convolution kernel in at least one embodiment of the present disclosure;

[0028] Figure 9 A schematic diagram of a data processing device provided in at least one embodiment of the present disclosure;

[0029] Figure 10 A schematic flow chart of a data processing method provided in at least one embodiment of the present disclosure;

[0030] Figure 11 A schematic structural diagram of a processor provided for at least one embodiment of the present disclosure;

[0031] Figure 12 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure; and

[0032] Figure 13 A schematic block diagram of an electronic device provided in accordance with an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0034] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the described object changes, the relative positional relationship may also change accordingly. In order to keep the following description of the embodiments of the present disclosure clear and concise, the present disclosure omits detailed descriptions of some known functions and known components.

[0035] A neural-network processing unit (NPU) or artificial intelligence chip is a type of microprocessor or computing system dedicated to hardware acceleration of artificial intelligence (especially artificial neural networks, machine vision, machine learning, etc.), sometimes also called an artificial intelligence accelerator. For example, it can be a graphics processor or a general-purpose graphics processor.

[0036] Figure 1 A schematic structural diagram of a general-purpose graphics processing unit (GPGPU) is shown.

[0037] like Figure 1 As shown, the general graphics processor includes an array of programmable multiprocessors, for example, the programmable multiprocessor can be a streaming processor cluster (SPC), for example including Figure 1 Streaming processor clusters 1, ..., and M are shown, where M is a positive integer greater than 1. In a general-purpose graphics processor, one streaming processor cluster processes one computing task, or multiple streaming processor clusters process one computing task. Multiple streaming processor clusters share data through a global cache or global memory.

[0038] like Figure 1 As shown, taking stream processor cluster 1 as an example, a stream processor cluster includes multiple computing units, such as Figure 1In the CU, there are CU 1, CU 2, ..., CU N, where N is a positive integer. Each CU is used to perform arithmetic and logical operations, such as accumulation, reduction, conventional addition, subtraction, multiplication, and division. A CU includes multiple cores (also called computing cores or computing cores), each of which includes an arithmetic logic unit (ALU), a floating-point computing unit, etc. The computing core is used to perform specific computing tasks. In addition, the CU also includes registers (e.g. Figure 1 The register file in the computing unit and the shared memory are used to hierarchically store source data and destination data related to computing tasks. The shared memory in a computing unit is used to share data between the cores of the computing unit.

[0039] In parallel computing, computing tasks are generally executed by multiple threads. These threads are divided into multiple thread blocks before being executed in a general-purpose graphics processor (or parallel computing processor), and then distributed through the thread block distribution module ( Figure 1 (Not shown) Multiple thread blocks are distributed to various CUs. All threads in a thread block must be assigned to the same CU for execution. Thread blocks are also split into minimum execution warps (or simply warps). Each warp contains a fixed number (or less) of threads, for example, 32 threads. Multiple thread blocks can execute in the same CU or in different CUs.

[0040] In each computational unit, the warp scheduling / dispatching module ( Figure 1 (not shown) schedules and allocates thread warps so that the multiple compute cores of the compute unit can execute the warps. Depending on the number of compute cores in the compute unit, multiple warps in a thread block can execute simultaneously or in a time-sharing manner. Multiple threads in each warp execute the same instruction. Memory execution instructions are sent to the shared memory within the compute unit or further to the mid-level cache, global cache, or global memory for read and write operations.

[0041] The tensor data used in the network model inference process is a three-dimensional tensor, including N, H, and W dimensions. The shape of the tensor data is specifically expressed as [n, h, w], where n, h, and w are positive integers. Here, the N dimension represents the batch size n of the tensor data, that is, the number of data samples captured in one training session, the H dimension represents the height h of each batch, and the W dimension represents the width w of each batch.

[0042] For example, if the tensor data is an image, the specific value n of the N dimension is 1, the value h of the H dimension represents the height of the image, and the value w of the W dimension represents the width of the image.

[0043] Figure 2 This is a schematic diagram of the structure of tensor data. Figure 2 As shown, there are N batches of data in the network model inference process, such as Figure 2 Multiple large rectangular boxes are shown in the figure: batch 0 (batch0), batch 1 (batch1), ... batch n-1 (batchn-1). A batch consists of H×W elements. Figure 2 Each small grid in a batch represents an element, for example, an element can have a data width of 8 bits, 16 bits, etc. Here, N, H, and W are positive integers.

[0044] Data used in network model inference is typically partitioned using a unified resource mapping logic to distribute data inference tasks across different layers of computing resources. Neural network processors can be used for inference on network models, such as large language models, visual language models, and multimodal models.

[0045] The convolution operation is a core component of a convolutional neural network (CNN). Convolution extracts local features by sliding a convolution kernel across an input feature map and generating an output feature map. CNNs, further combined with activation functions and pooling operations, are widely used in tasks such as image classification, object detection, and semantic segmentation. Multi-channel convolution refers to the convolution operation performed when processing data with multiple input channels within a CNN. This operation is commonly used in image processing, as images typically have multiple color channels (such as RGB images), but it is also applicable to other multi-feature data scenarios.

[0046] Figure 3A Figure 2 shows a schematic diagram of a convolutional layer performing multi-channel convolution operations. Figure 3A As shown in the figure, M sets of R×S (height×width, the same applies below) convolution kernels with C channels are used to perform convolution operations on N batches of H×W input images (or input feature maps) with C channels. N sets of E×F output feature maps with M channels are obtained. Therefore, the output feature maps have multiple dimensions F / E / M. Convolution operations are characterized by high parallelism and high data reuse. This high parallelism is reflected in the fact that multiple convolution kernels can operate on multiple input feature maps simultaneously.

[0047] As mentioned above, for a convolution operation (convolution), the input feature map can be represented as a tensor (N, C, H, W), where N represents the batch, C represents the number of channels, H represents the height of the input matrix, and W represents the width of the input matrix. For example, the input feature map corresponds to N RGB images, with three channels (C) corresponding to red, green, and blue, respectively. A single image has a height (H) and width (W). The convolution kernel is one or more two-dimensional matrices, typically much smaller than the input feature map. The convolution kernel slides over the input feature map with a specific stride, performing a dot product operation with a local region of the input data. For example, a stride of 1 means sliding one matrix element (e.g., pixel) at a time, while a stride of 2 means sliding two matrix elements (e.g., pixels) at a time. The convolution kernel can also be represented as a tensor (M, C, R, S), where M represents the number of kernel groups, C represents the number of channels in each kernel group, R represents the height of the two-dimensional matrix in the kernel, and S represents the width of the two-dimensional matrix in the kernel. The result of the convolution operation is a new two-dimensional array, called the output feature map. Each group of convolution kernels is used to generate an output feature map, and multiple groups of convolution kernels are used to generate multi-channel output feature maps.

[0048] In convolution processing, padding is an optional processing method used to deal with the boundary problem of the input feature map; the convolution kernel slides on the input feature map that has been expanded by padding, calculates the dot product at each position, and generates an output feature map. Padding achieves various purposes by adding extra values ​​(usually zero) to the edges of the input feature map (in the height and width directions). For example, by properly selecting the number of padded elements (P), the size of the output feature map can be made the same as the size of the input feature map; by padding, it can be ensured that the information at the edge can also be processed by convolution; using padding can make the convolution neural network model more flexible and able to maintain a consistent output size under different input sizes, thereby simplifying the design of the neural network.

[0049] For example, in neural network processor design, compute-intensive operations such as matrix multiplication and convolution, which account for a high proportion of the model, are often offloaded to dedicated hardware units such as tensor cores for accelerated computation. However, tensor cores may not natively support convolutions with large kernels. Therefore, optimizing the convolution performance of large kernels in the model is critical for improving end-to-end performance. Traditional approaches employ generalized matrix multiplication (GEMM) or implicit generalized matrix multiplication (implicit GEMM). Implicit GEMM is a variant of GEMM that still performs matrix multiplication (matmul) on the output of the im2col function, but does not explicitly generate the im2col output to implement convolution operations with large kernels. Large convolution kernels refer to larger convolution kernels used in convolutional neural networks. For example, kernels with kernel matrices larger than 7, such as 16×16 and 25×25, are typically used for tasks that require capturing a wide range of features in the input data, such as image classification, object detection, and semantic segmentation. However, large convolution kernel calculations require more multiplication and addition operations, so the computational cost is higher.

[0050] Figure 3B Figure 2 shows a schematic diagram of using implicit GEMM to implement convolution. Figure 3B As shown in the figure, the corresponding convolution operation is to perform sliding window dot product calculations on N input feature maps of size C×H×W with M convolution kernels of size C×R×S, and output N output feature maps of size M×E×F. The implicit GEMM algorithm treats the convolution kernel as a two-dimensional matrix of size M×(C×R×S), and the input feature map is transformed into a two-dimensional matrix of size (C×R×S)×(N×H×W) based on the position of the convolution kernel sliding window. The two matrices are then multiplied to obtain the output feature map.

[0051] Typically, neural network processors support multi-channel convolution calculations. For example, a certain neural network processor is designed to support a maximum of 64 channels of parallel calculations. This means that even if the processor actually has 3 channels, it is as efficient as a 64-channel processor. However, in actual convolution calculations, these 64 channels are usually not fully utilized. For example, even for a 3-channel convolution calculation, the neural network processor's hardware is occupied during the operation, but 61 channels are actually idle. Therefore, in this case, the neural network processor is not fully utilized, increasing its cost.

[0052] In response to the above problems, the inventors of the present invention provide a data processing method and device. By transforming the convolution operation of a large convolution kernel into a convolution operation of a small convolution kernel, not only can a neural network processor that does not natively support large convolution kernels be used to support large convolution kernels, but the channel parallelism of the neural network processor can also be increased, thereby improving hardware utilization.

[0053] At least one embodiment of the present disclosure provides a data processing method and device, a processor, an electronic device, and a non-transitory computer-readable storage medium. The data processing method includes: receiving a first convolution kernel and performing a first rearrangement transformation to obtain a second convolution kernel, wherein the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, and the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; receiving a first input feature map, wherein the size of the first input feature map is expressed as (N, C1, H1, W1), N represents The batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))×(roundup(S1 / S2)), roundup(×) is a ceiling rounding function; using the second input feature map and the second convolution kernel to perform convolution processing and obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by directly performing the convolution processing on the first input feature map and the first convolution kernel.

[0054] The embodiments of the present disclosure are described in detail below with reference to the accompanying drawings, but the present disclosure is not limited to these specific embodiments.

[0055] Figure 4 A schematic flowchart of a data processing method provided in at least one embodiment of the present disclosure.

[0056] like Figure 4 As shown, the data processing method provided by at least one embodiment of the present disclosure includes at least steps S10-S40.

[0057] S10: Receive the first convolution kernel and perform a first rearrangement transformation to obtain a second convolution kernel.

[0058] For example, the size of the first convolution kernel is expressed as (M, C1, R1, S1), where M is used to represent the number of the first convolution kernel, that is, the number of output channels, C1 is used to represent the number of channels of the first convolution kernel, that is, the number of input channels, R1 is used to represent the matrix height of the first convolution kernel, and S1 is used to represent the matrix width of the first convolution kernel, that is, it represents M R1×S1 convolution kernels with C1 channels, and M, C1, R1 and S1 are all positive integers; the size of the second convolution kernel is expressed as (M, C2, R2, S2), where M is used to represent the number of the second convolution kernel, C2 is used to represent the number of channels of the second convolution kernel, R2 is used to represent the matrix height of the second convolution kernel, and S2 is used to represent the matrix width of the second convolution kernel, that is, it represents M R2×S2 convolution kernels with C2 channels, and M, C2, R2 and S2 are all positive integers.

[0059] For example, in some embodiments, step S10 may include: when R1 or S1 in the size of the first convolution kernel is a prime number, the first convolution kernel of size (M, C1, R1, S1) may be padded to obtain an intermediate convolution kernel of size (M, C1, R', S'), where M is used to represent the number of intermediate convolution kernels, C1 is used to represent the number of channels of the intermediate convolution kernel, R' is used to represent the matrix height of the intermediate convolution kernel, and S' is used to represent the matrix width of the intermediate convolution kernel, and M, C1, R' and S' are all positive integers; the intermediate convolution kernel is split and rearranged to obtain a second convolution kernel of size (M, C2, R2, S2), where R'=R2×roundup(R1 / R2) and S'=S2×roundup(S1 / S2).

[0060] For example, when the matrix size in the first convolution kernel is 11×11, that is, R1 and S1 are equal and both are prime numbers, they can be aligned to 12×12 through padding (for example, zero padding), and then each 12×12 kernel matrix is ​​divided into 9 4×4 second kernel matrices or 16 3×3 second kernel matrices to obtain a second convolution kernel with R2=S2=4 or R2=S2=3. For example, when the matrix size in the first convolution kernel is 13×11, that is, R1 and S1 are not equal and both are prime numbers, they can be aligned to 15×12 through padding (for example, zero padding), and then each 15×12 kernel matrix is ​​divided into 9 5×4 second kernel matrices to obtain a second convolution kernel with R2=5 and S2=4. For example, when the matrix size in the first convolution kernel is 13×12, that is, R1 and S1 are not equal and R1 or S1 is a prime number, it can be aligned to 15×12 through padding processing (e.g., zero padding), and then each 15×12 kernel matrix is ​​divided into 9 5×4 second kernel matrices to obtain a second convolution kernel with R2=5 and S2=4. For example, step S10 may include: when R1 and S1 in the size of the first convolution kernel are not prime numbers, the first convolution kernel with a size of (M, C1, R1, S1) is directly split and rearranged to obtain a second convolution kernel with a size of (M, C2, R2, S2).

[0061] For example, when R1=S1=25, that is, neither R1 nor S1 is a prime number, the first convolution kernel consists of a 25×25 two-dimensional matrix. Each 25×25 kernel matrix can be directly split into 25 5×5 second kernel matrices to obtain a second convolution kernel with R2=S2=5. For example, when R1=16 and S2=9, that is, neither R1 nor S1 is a prime number, the first convolution kernel consists of a 16×9 two-dimensional matrix. Each 16×9 kernel matrix can be directly split into 12 4×3 second kernel matrices to obtain a second convolution kernel with R2=4 and S2=3.

[0062] For example, the split and rearrangement transformation in step S10 may include: for each kernel matrix in the first convolution kernel or the intermediate convolution kernel, splitting the kernel matrix in a first order to obtain k second kernel matrices corresponding to the kernel matrix, and superimposing the k second kernel matrices in the k channels of the second convolution kernel in a first order to obtain a second convolution kernel, where k is a positive integer.

[0063] For example, for each kernel matrix in the first convolution kernel or the intermediate convolution kernel, determine the k second kernel matrices corresponding to the kernel matrix obtained by segmenting the kernel matrix; arrange the k second kernel matrices corresponding to each kernel matrix in the first convolution kernel or the intermediate convolution kernel according to the order relationship of each kernel matrix to obtain the second convolution kernel, and arrange the k second kernel matrices corresponding to each kernel matrix in the first order.

[0064] For example, the kernel matrix is ​​a two-dimensional matrix, and a single or multiple kernel matrices are combined into a convolution kernel on a single or multiple channels.

[0065] For example, the first order may be the sliding order of the convolution kernel in the convolution operation.

[0066] Figure 5 Schematic diagram of the segmentation and rearrangement transformation provided by at least one embodiment of the present disclosure. Figure 5 As shown, the left side shows a kernel matrix (R1, S1) in the first convolution kernel, and the kernel matrix is ​​divided into k second kernel matrices in the first order, for example Figure 5 The rectangular boxes numbered 1, 2, 3, ..., 16 in Figure 5 In the example, k=16, of course, the present disclosure is not limited thereto. Figure 5 The k second kernel matrices (R2, S2) on the right side of the convolution kernel are sequentially superimposed on the k channels of the second convolution kernel according to the first order mentioned above.

[0067] Each kernel matrix (R1, S1) in the first convolution kernel can be passed through Figure 5 The split rearrangement transformation in the second convolution kernel is obtained, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)).

[0068] For example, the first rearrangement transformation can be performed offline. Since different input feature maps can use the first convolution kernel for convolution operations, in each data processing, it is only necessary to perform the first rearrangement transformation on the first convolution kernel once to obtain the second convolution kernel. After that, the second convolution kernel can be used all the time to save computing resources.

[0069] For example, the second convolution kernel obtained by the first rearrangement transformation can be stored for convolution processing, without having to perform the first rearrangement transformation again each time a convolution operation is performed, thus saving computing resources.

[0070] For example, the weight update during model training is based on the weight of the second convolution kernel obtained after the first permutation transformation. After the model training is completed, the second convolution kernel needs to undergo an inverse first permutation transformation. The inverse first permutation transformation is the inverse of the first permutation transformation, that is, the second convolution kernel is transformed back to the first convolution kernel through the inverse first permutation transformation.

[0071] For example, the inverse first rearrangement transformation includes concatenating multiple second kernel matrices into a kernel matrix in the first order, and further arranging the kernel matrices in the order between them to obtain the first convolution kernel again.

[0072] S20: Receive a first input feature map.

[0073] For example, the size of the first input feature map is expressed as (N, C1, H1, W1), where N is used to represent the batch of the first input feature map, C1 is used to represent the number of channels of the first input feature map, H1 is used to represent the matrix height of the first input feature map, and W1 is used to represent the matrix width of the first input feature map. The number of channels of the first input feature map is equal to the number of channels of the first convolution kernel, and N, C1, H1, and W1 are all positive integers.

[0074] S30: Perform a second rearrangement transformation on the first input feature map to obtain a second input feature map.

[0075] For example, the size of the second input feature map is expressed as (N, C2, H2, W2), where N is used to represent the batch of the second input feature map, C2 is used to represent the number of channels of the second input feature map, H2 is used to represent the matrix height of the second input feature map, and W2 is used to represent the matrix width of the second input feature map. N, C2, H2, and W2 are all positive integers; C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))× (roundup(S1 / S2)), and roundup(×) is a rounding-up function.

[0076] For example, in some embodiments, the input feature map needs to be padded, and step S30 may include: padding the first input feature map of size (N, C1, H1, W1) to obtain an intermediate input feature map of size (N, C1, H', W'), where N is used to represent the batch of the intermediate input feature map, C1 is used to represent the number of channels of the intermediate input feature map, H' is used to represent the matrix height of the intermediate input feature map, and W' is used to represent the matrix width of the intermediate input feature map. N, C1, H' and W' are all positive integers; H'=H1+2P, W'=W1+2P, and P is the number of padding elements. Then, a second rearrangement transformation is performed on the intermediate input feature map to obtain a second input feature map of size (N, C2, H2, W2).

[0077] For example, in some other embodiments, there is no need to perform padding processing on the first input feature map. For example, step S30 may include: directly performing a second rearrangement transformation on the first input feature map to obtain a second input feature map.

[0078] For example, the second rearrangement transformation can be: for each first input matrix in the first input feature map or the intermediate input feature map, determine the kernel matrix corresponding to the first input matrix, take out the elements of the convolution calculation of the k second kernel matrices corresponding to the kernel matrix in the first input matrix, rearrange to obtain k second input matrices corresponding one to one to the k second kernel matrices, and superimpose the k second input matrices in the k channels of the second input feature map in the first order to obtain the second input feature map.

[0079] For example, the kernel matrix corresponding to the first input matrix refers to the kernel matrix that performs convolution calculation with the first input matrix.

[0080] For example, the first input matrix is ​​one of the two-dimensional matrices constituting the first input feature map, and the second input matrix is ​​the elements corresponding to the convolution operation between the first input matrix and a second kernel matrix.

[0081] Figure 6A and Figure 6B A second rearrangement transformation diagram is provided for at least one embodiment of the present disclosure. Figure 6A and Figure 6B In , the first input matrix is ​​the (H1, W1) two-dimensional matrix within a channel in a batch in the first input feature map.

[0082] Figure 6A The second input matrix 1 is shown ( Figure 6A The second input matrix 1 is shown in the bold box) and the second kernel matrix 1 ( Figure 6A The kernel matrix in the first convolution kernel is segmented to form the second kernel matrix located in the upper left corner. The second input matrix 1 is the element corresponding to the convolution operation between the first input matrix and the second kernel matrix 1. Specifically, this element is the element used for the convolution operation with the second kernel matrix 1. The upper left corner element of the second input matrix corresponding to different second kernel matrices has a different position in the first input matrix, which is related to the position of the second kernel matrix in the kernel matrix of the first convolution kernel.

[0083] Figure 6B The second input matrix 2 is shown ( Figure 6B The second input matrix 2 shown in the bold box) and the second kernel matrix 2 ( Figure 6B The second input matrix 2 is the element corresponding to the convolution operation between the first input matrix and the second kernel matrix 2.

[0084] according to Figure 6A and Figure 6B In this manner, the elements corresponding to the convolution calculations of the first input matrix and each second kernel matrix are extracted, and 16 second input matrices are rearranged to obtain them. The 16 second input matrices are then superimposed on the k channels of the second input feature map in the first order to complete the second rearrangement transformation. For example, the second rearrangement transformation is performed online on the first input feature map to obtain the second input feature map. Each first input matrix of the first input feature map needs to undergo the second rearrangement transformation once.

[0085] S40: Perform convolution processing using the second input feature map and the second convolution kernel to obtain a second output feature map.

[0086] For example, the second output feature map is equivalent to the first output feature map obtained by directly convolving the first input feature map with the first convolution kernel. Direct convolution refers to performing a conventional convolution calculation. Equivalence includes numerical equivalence between the first and second output feature maps. Numerical equivalence means that the pixel values ​​at every position in the two feature maps are exactly the same.

[0087] It should be noted that the present disclosure does not limit the order of S10 - S40 .

[0088] Figure 7 This is a schematic diagram of a data processing method provided by an embodiment of the present disclosure. Figure 7 right Figure 4 S10-S40 are further explained.

[0089] like Figure 7 As shown, in step S10, a first rearrangement transformation is performed on the first convolution kernel to obtain a second convolution kernel.

[0090] because Figure 7 The first convolution kernel in (32, 3, 16, 16), 16 is a non-prime number, so there is no need to fill the first convolution kernel, and it can be directly split and rearranged. Figure 5 The method shown in FIG1 is to divide the 16×16 kernel matrix in the first convolution kernel into 16 4×4 second kernel matrices according to the first order, and superimpose the above 16 second kernel matrices on the 16 channels of the second convolution kernel according to the first order, for example, insert the 16 second kernel matrices into the 16 channels of the second convolution kernel according to the first order. Thus, each kernel matrix in the first convolution kernel is Figure 5 The split rearrangement transformation shown can obtain the second convolution kernel (32, 48, 4, 4).

[0091] In steps S20 and S30, a first input feature map is received, and a second rearrangement transformation is performed on the first input feature map to obtain a second input feature map.

[0092] First, the first input feature map (16, 3, 512, 512) is padded with p=7 (i.e., 7×7 padding) to obtain an intermediate input feature map (16, 3, 526, 526). The intermediate input feature map is composed of the first input matrix of 526×526.

[0093] Then, follow Figure 6A and Figure 6BThe method shown performs a second rearrangement transformation, that is, sequentially extracts the corresponding elements of each first input matrix that are convolved with each second kernel matrix, and rearranges them to obtain 16 second input matrices, and the above 16 second input matrices are superimposed on the 16 channels of the second input feature map in the first order, for example, the 16 second input matrices are inserted into the 16 channels of the second input feature map in the first order; in addition, the 16 second input matrices corresponding to the first input matrices of the three channels are arranged in channel order, thereby obtaining the second input feature map (16, 48, 514, 514).

[0094] In step S40, convolution processing is performed using the second input feature map and the second convolution kernel to obtain a second output feature map.

[0095] Convolution is performed using the second input feature map (16, 48, 514, 514) and the second convolution kernel (32, 48, 4, 4) to obtain the second output feature map (16, 32, 512, 512).

[0096] Figure 8 This is a schematic diagram of performing convolution processing on a first input feature map and a first convolution kernel in at least one embodiment of the present disclosure.

[0097] like Figure 8 As shown in the figure, the first input feature map (16, 32, 512, 512) and the first convolution kernel (32, 3, 16, 16) are convolved after padding 7×7 (not shown in the figure) to obtain the output result of (16, 32, 512, 512). It can be seen that the result is the same as Figure 7 The second output feature map obtained in S40 is equivalent.

[0098] The disclosed embodiments also include using a convolution processing device to perform the data processing method provided in the above embodiments. The convolution processing device is configured to process Cm channels in parallel, where Cm is greater than or equal to C2; the convolution processing device is configured to process a maximum convolution kernel size Km for a single operation, with the maximum value between R2 and S2 being less than or equal to Km, and the minimum value between R1 and S1 being greater than Km; this is used to increase the channel dimension and improve the utilization of hardware computing, thereby improving the overall performance of the operator and accelerating the computing speed.

[0099] For example, convolution processing devices include hardware units specifically designed for accelerated computing, such as tensor cores. The channel dimensions of tensor cores vary depending on different GPU architectures. For example, different second permutation transformations can be performed on the number of channels of the first input feature map based on different tensor core channel dimensions (denoted as Cm) to obtain the corresponding number of channels of the second input feature map (denoted as C2), as long as Cm is greater than or equal to C2. For example, different first permutation transformations can be performed on the size of the first convolution kernel (denoted as (R1, S1)) to obtain the corresponding second convolution kernel size (denoted as (R2, S2)) based on the maximum convolution kernel size that can be processed at a time by different tensor cores (denoted as Km), as long as the maximum value between R2 and S2 is less than or equal to Km, and the minimum value between R1 and S1 is greater than Km.

[0100] The data processing method provided by at least one embodiment of the present disclosure improves the overall performance of the operator and speeds up the calculation speed by converting large convolution kernels into small convolution kernels; and increases the number of channels of convolution operations with a small number of input layer channels through transformation, thereby avoiding idleness of some computing resources as much as possible and improving the utilization rate of hardware calculations.

[0101] Figure 9 A schematic block diagram of a data processing device provided by at least one embodiment of the present disclosure. Figure 9 As shown, the data processing device 10 provided by the embodiment of the present disclosure includes a first rearrangement unit 11 , a receiving unit 12 , a second rearrangement unit 13 and a processing unit 14 .

[0102] The first rearrangement unit 11 is configured to receive the first convolution kernel and perform a first rearrangement transformation to obtain a second convolution kernel. The size of the first convolution kernel is expressed as (M, C1, R1, S1), where M is used to represent the number of first convolution kernels, C1 is used to represent the number of channels of the first convolution kernel, R1 is used to represent the matrix height of the first convolution kernel, and S1 is used to represent the matrix width of the first convolution kernel. The size of the second convolution kernel is expressed as (M, C2, R2, S2), where M is used to represent the number of second convolution kernels, C2 is used to represent the number of channels of the second convolution kernel, R2 is used to represent the matrix height of the second convolution kernel, and S2 is used to represent the matrix width of the second convolution kernel.

[0103] The receiving unit 12 is configured to receive a first input feature map, where the size of the first input feature map is expressed as (N, C1, H1, W1), where N is used to represent the batch of the first input feature map, C1 is used to represent the number of channels of the first input feature map, H1 is used to represent the matrix height of the first input feature map, and W1 is used to represent the matrix width of the first input feature map. The number of channels of the first input feature map is equal to the number of channels of the first convolution kernel;

[0104] The second rearrangement unit 13 is configured to perform a second rearrangement transformation on the first input feature map to obtain a second input feature map, where the size of the second input feature map is expressed as (N, C2, H2, W2), where N is used to represent the batch of the second input feature map, C2 is used to represent the number of channels of the second input feature map, H2 is used to represent the matrix height of the second input feature map, and W2 is used to represent the matrix width of the second input feature map. C2=k×C1, where k is an integer greater than 1 and k=(roundup(R1 / R2))×(roundup(S1 / S2)), where roundup(×) is a ceiling rounding function.

[0105] The processing unit 14 is configured to perform convolution processing using the second input feature map and the second convolution kernel and obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by performing the convolution processing on the first input feature map and the first convolution kernel.

[0106] For details about the first rearrangement unit 11, the receiving unit 12, the second rearrangement unit 13 and the processing unit 14, please refer to the embodiment of the above data processing method. Figure 4 The description of steps S10-S40 is omitted here.

[0107] above Figure 9 The various modules shown may be implemented, for example, by software, hardware, firmware, or any combination thereof.

[0108] It should be noted that the specific physical meanings of the first input feature map, first convolution kernel, second input feature map, and second convolution kernel in the data processing method or data processing device provided in at least one embodiment of the present disclosure may vary depending on the application scenario. For example, the data processing method provided in at least one embodiment of the present disclosure may be applied in fields such as speech processing, image processing, text processing, and video processing.

[0109] For example, in the field of speech processing, the first input feature map, the first convolution kernel, the second input feature map, and the second convolution kernel can be parameters involved in convolution processing in tasks such as feature extraction, speech enhancement, and speech recognition.

[0110] For example, in the field of image processing, the first input feature map, the first convolution kernel, the second input feature map, and the second convolution kernel can be parameters involved in convolution processing in tasks such as image recognition, feature extraction, image segmentation, target detection, image classification, and scene reconstruction.

[0111] For example, in the field of text processing, the first input feature map, the first convolution kernel, the second input feature map, and the second convolution kernel can be parameters involved in convolution processing in tasks such as text classification, sentiment analysis, and text generation.

[0112] For example, in the field of video processing, the first input feature map, the first convolution kernel, the second input feature map, and the second convolution kernel can be relevant parameters in the image processing field as described above, or parameters involving convolution processing in the video processing field (such as video classification, action recognition, etc.).

[0113] Of course, the present disclosure is not limited to this. For other application scenarios or fields, as long as convolution processing is required, the data processing method described in at least one embodiment of the present disclosure can be applied, and no further details will be given here.

[0114] At least one embodiment of the present disclosure further provides a data processing method. Figure 10 A schematic flowchart of a data processing method provided in at least one embodiment of the present disclosure.

[0115] like Figure 10 As shown, the data processing method at least includes steps S50-S60.

[0116] In step S50, a convolution processing instruction is received.

[0117] For example, a convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, H1 represents the matrix height of the first input feature map, W1 is used to represent the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel.

[0118] In step S60 , after parsing the convolution processing instruction, the convolution processing unit is used to execute the convolution processing instruction to obtain a second output feature map.

[0119] For example, the second output feature map is equivalent to the first output feature map obtained by directly convolving the first input feature map with the first convolution kernel.

[0120] For example, in step S60, using the convolution processing unit to execute the convolution processing instruction includes: performing a first rearrangement transformation on the first convolution kernel to obtain a second convolution kernel, wherein the size of the second convolution kernel is expressed as (M, C2, R2, S2), C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))×(roundup(S1 / S2)), roundup(×) is a rounding-up function; using the second input feature map and the second convolution kernel to perform convolution processing and obtain a second output feature map.

[0121] For example, the data processing method provided by at least one embodiment of the present disclosure can be applied to Figure 11 Processor shown.

[0122] For example, in the data processing method provided in at least one embodiment of the present disclosure, a convolution processing instruction is provided, and the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters.

[0123] For example, after receiving a convolution processing instruction, the processor parses the convolution processing instruction, such as decoding the convolution processing instruction, generates a microinstruction, and sends the microinstruction to the instruction dispatch unit; the instruction dispatch unit sends the microinstruction to the corresponding scheduling queue according to the microinstruction category; in response to the microinstruction, when the input parameters are prepared, the convolution processing unit reads and executes the relevant operations of the convolution processing instruction.

[0124] Regarding the specific process of using the convolution processing unit to execute the convolution processing instruction, reference may be made to steps S10-S40 in the aforementioned data processing method, and repeated descriptions will be omitted.

[0125] The data processing method provided in at least one embodiment of the present disclosure can achieve technical effects similar to those of the aforementioned data processing method, and will not be described in detail here.

[0126] Figure 11 This is a schematic structural diagram of a processor provided in at least one embodiment of the present disclosure. Figure 11 As shown, the processor 400 includes an instruction parsing unit 401 and a convolution processing unit 402 .

[0127] For example, the instruction parsing unit 401 is used to receive and parse a convolution processing instruction, wherein the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel.

[0128] For example, after the instruction parsing unit 401 parses the convolution processing instruction, the convolution processing unit 402 executes the data processing method according to any embodiment of the present disclosure.

[0129] Specifically, when processor-based upper-layer software (e.g., AI applications, HPC applications, and scientific computing applications) can send a convolution processing instruction for computational processing to a processor (e.g., a CPU or GPU) through a uniformly packaged function library, the convolution processing instruction can carry a first convolution kernel and a first input feature map as input parameters; when the processor 400 receives the convolution processing instruction, the instruction parsing unit 401 parses the convolution processing instruction to obtain the input parameters, and the processor 400 schedules the convolution processing unit 402 to perform data processing tasks on the input parameters. For example, after parsing the convolution processing instruction, the processor can store the input parameters in the convolution processing instruction into a register or memory, so that when the convolution processing unit 402 performs computational processing, it can obtain the input parameters from the register or memory.

[0130] Regarding the specific process of using the convolution processing unit 402 to execute the convolution processing instruction, reference may be made to steps S10-S40 in the aforementioned data processing method, and repeated descriptions will be omitted.

[0131] Figure 12 A schematic diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Figure 12 As shown, the storage medium 200 may be a non-transitory computer-readable storage medium, and one or more computer-readable instructions 201 may be non-transitory stored on the storage medium 200. For example, when the computer-readable instructions 201 are executed by a processor, one or more steps in the data processing method described above may be performed.

[0132] For example, the storage medium 200 may be applied to an electronic device. For example, the storage medium 200 may include a memory in the electronic device.

[0133] For example, the storage device may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, a flash memory, etc. The computer-readable storage medium may store one or more computer-readable instructions, which the data processor may execute to implement various functions of the data processor. The storage medium may also store various application programs and various data.

[0134] For example, the storage medium may include a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a flash memory, or any combination of the above storage media, or other applicable storage media.

[0135] Figure 13 This is a schematic block diagram of an electronic device provided by an embodiment of the present disclosure. Figure 13 As shown, the electronic device 300 is suitable for implementing the data processing method provided by the embodiment of the present disclosure. It should be noted that Figure 13 The components of the electronic device 300 shown are merely exemplary and non-limiting. The electronic device 300 may also have other components according to actual application requirements.

[0136] For example, Figure 13 As shown, in some examples, the electronic device 300 includes a processing device (e.g., a graphics processor, a general-purpose graphics processor, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 to a random access memory (RAM) 303 to achieve various functions, such as executing the data processing method provided by at least one embodiment of the present disclosure.

[0137] For example, when the computer-readable instructions are executed by the processing device 301, one or more steps of the data processing method described in any of the above embodiments may be executed. It should be noted that for a detailed description of the processing process of the data processing method, reference can be made to the relevant description in the above embodiments of the data processing method, and any repetitions will not be repeated.

[0138] Various programs and data required for the operation of the computer system are also stored in the RAM 303 . The processing device 301 , the ROM 302 , and the RAM 303 are connected to each other via a bus 304 . An input / output (I / O) interface 305 is also connected to the bus 304 .

[0139] For example, the memory may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) 303 and / or cache memory. For example, computer-readable instructions may be loaded from storage device 308 into RAM 303 to execute the computer-readable instructions. Non-volatile memory may include, for example, read-only memory (ROM) 302, a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, a flash memory, and the like. Various applications and various data, such as style images and various data used and / or generated by the applications, may also be stored in the computer-readable storage media.

[0140] Typically, the following devices may be connected to the I / O interface 305: an input device 306 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 308 including, for example, a magnetic tape, a hard disk, a flash memory, etc.; and a communication device 309. The communication device 309 may allow the electronic device 300 to communicate with other electronic devices wirelessly or by wire to exchange data. Figure 13While an electronic device 300 is shown with various devices, it should be understood that implementation or presence of all of the devices shown is not required, and the electronic device 300 may alternatively implement or possess more or fewer devices. For example, the processor 301 may control other components in the electronic device 300 to perform desired functions. The processing device 301 may be a device with data processing and / or program execution capabilities, such as a central processing unit (CPU), a tensor processing unit (TPU), or a graphics processing unit (GPU). The CPU may be based on an X86 or ARM architecture. The graphics processing unit may be separately integrated directly onto the motherboard or built into the motherboard's northbridge chip. The graphics processor may also be built into the CPU.

[0141] For example, the electronic device 300 may further include a peripheral interface (not shown in the figure), etc. The peripheral interface may be various types of interfaces, such as a USB interface, a lightning interface, etc. The communication device 309 may communicate with a network and other devices through wireless communication, such as the Internet, an intranet, and / or a wireless network such as a cellular telephone network, a wireless local area network (LAN), and / or a metropolitan area network (MAN). Wireless communications may use any of a variety of communication standards, protocols, and technologies, including, but not limited to, Global System for Mobile Communications (GSM), Enhanced Data GSM Environment (EDGE), Wideband Code Division Multiple Access (W-CDMA), Code Division Multiple Access (CDMA), Time Division Multiple Access (TDMA), Bluetooth, Wi-Fi (e.g., based on IEEE 802.11a, IEEE 802.11b, IEEE 802.11g, and / or IEEE 802.11n standards), Voice over Internet Protocol (VoIP), Wi-MAX, protocols for email, instant messaging, and / or Short Message Service (SMS), or any other suitable communication protocol.

[0142] For example, the electronic device 300 can be any device such as a mobile phone, tablet computer, laptop computer, e-book, game console, television, digital photo frame, navigator, etc., or a combination of any electronic devices and hardware, which is not limited in the embodiments of the present disclosure.

[0143] For example, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned data processing method defined in the method of the embodiment of the present disclosure is executed.

[0144] It should be noted that the computer-readable medium described above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. Computer-readable storage media may include, for example, but not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or components, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wire, optical cable, RF (radio frequency), or any suitable combination thereof.

[0145] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0146] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0147] The units involved in the embodiments described in this disclosure may be implemented in software or hardware, wherein the name of a unit does not necessarily limit the unit itself.

[0148] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the present disclosure is not limited to technical solutions formed by specific combinations of the aforementioned technical features. It also encompasses other technical solutions formed by any combination of the aforementioned technical features or their equivalents, without departing from the scope of the above disclosure. For example, a technical solution formed by replacing the aforementioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0149] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0150] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

[0151] Regarding this disclosure, the following points need to be explained:

[0152] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure. Other structures may refer to conventional designs.

[0153] (2) In the absence of conflict, the embodiments of the present disclosure and the features therein may be combined with each other to form new embodiments.

[0154] The above description is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. The protection scope of the present disclosure shall be based on the protection scope of the claims.

Claims

1. A data processing method, comprising: Receive the first convolution kernel and perform a first rearrangement transformation to obtain a second convolution kernel, wherein the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, and the size of the second convolution kernel is expressed as (M, C2, R2, S2), M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; Receive a first input feature map, where the size of the first input feature map is expressed as (N, C1, H1, W1), where N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, and W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; Performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))×(roundup(S1 / S2)), and roundup(*) is a rounding-up function; Performing convolution processing using the second input feature map and the second convolution kernel to obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by directly performing the convolution processing on the first input feature map and the first convolution kernel, wherein the equivalence includes numerical equivalence; The data processing method further includes: A convolution processing device is used to perform the data processing method, wherein the convolution processing device is configured to process Cm channels in parallel, Cm is greater than or equal to C2, the convolution processing device is configured so that the maximum convolution kernel size of a single processing is Km, the maximum value between R2 and S2 is less than or equal to Km, the minimum value between R1 and S1 is greater than Km, and Km and Cm are positive integers.

2. The data processing method according to claim 1, further comprising: The obtained second convolution kernel is stored for use in the convolution process.

3. The data processing method according to claim 1, wherein: Performing the first rearrangement transformation to obtain the second convolution kernel includes: In response to R1 or S1 being a prime number, padding the first convolution kernel to obtain an intermediate convolution kernel, and performing a split rearrangement transformation on the intermediate convolution kernel to obtain the second convolution kernel, wherein the size of the intermediate convolution kernel is expressed as (M, C1, R', S'), M represents the number of the intermediate convolution kernels, C1 represents the number of channels of the intermediate convolution kernel, R' represents the matrix height of the intermediate convolution kernel, S' represents the matrix width of the intermediate convolution kernel, R'=R2×roundup(R1 / R2), S'=S2×roundup(S1 / S2); In response to the fact that neither R1 nor S1 is a prime number, the first convolution kernel is directly subjected to the splitting and rearranging transformation to obtain the second convolution kernel.

4. The data processing method according to claim 3, wherein: The split rearrangement transformation includes: For each kernel matrix in the first convolution kernel or the intermediate convolution kernel, the kernel matrix is ​​divided according to the first order to obtain k second kernel matrices corresponding to the kernel matrix, and the k second kernel matrices are superimposed on the k channels of the second convolution kernel according to the first order, thereby obtaining the second convolution kernel, where k is a positive integer.

5. The data processing method according to any one of claims 2 to 4, wherein: The first rearrangement transformation is performed offline to obtain the second convolution kernel from the first convolution kernel.

6. The data processing method according to claim 4, wherein: Performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, comprising: Performing padding processing on the first input feature map to obtain an intermediate input feature map, wherein the size of the intermediate input feature map is expressed as (N, C1, H', W'), N represents the batch of the intermediate input feature map, C1 represents the number of channels of the intermediate input feature map, H' represents the matrix height of the intermediate input feature map, W' represents the matrix width of the intermediate input feature map, H'=H1+2P, W'=W1+2P, P is the number of padding elements, and then performing the second rearrangement transformation on the intermediate input feature map to obtain the second input feature map; or, The second input feature map is directly subjected to the second rearrangement transformation to obtain the second input feature map.

7. The data processing method according to claim 6, wherein: Performing the second rearrangement transformation on the first input feature map or the intermediate input feature map to obtain the second input feature map includes: For each first input matrix in the first input feature map or the intermediate input feature map, determine the kernel matrix corresponding to the first input matrix, take out the elements of the first input matrix that are convolution-calculated with the k second kernel matrices corresponding to the kernel matrix, rearrange them to obtain k second input matrices that correspond one-to-one to the k second kernel matrices, and superimpose the k second input matrices on the k channels of the second input feature map in the first order to obtain the second input feature map.

8. The data processing method according to claim 6, wherein: Perform the second rearrangement transformation on the first input feature map online to obtain the second input feature map.

9. A data processing device comprising The first rearrangement unit is configured to receive the first convolution kernel and perform a first rearrangement transformation to obtain a second convolution kernel, wherein: The size of the first convolution kernel is expressed as (M, C1, R1, S1), where M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, and S1 represents the matrix width of the first convolution kernel. The size of the second convolution kernel is expressed as (M, C2, R2, S2), where M represents the number of the second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel. A receiving unit is configured to receive a first input feature map, wherein the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; a second rearrangement unit, configured to perform a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein a size of the second input feature map is expressed as (N, C2, H2, W2), N represents a batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))×(roundup(S1 / S2)), and roundup(*) is a rounding-up function; a processing unit configured to perform convolution processing using the second input feature map and the second convolution kernel to obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by performing the convolution processing on the first input feature map and the first convolution kernel, wherein the equivalence includes numerical equivalence; In which, the data processing device is configured to process Cm channels in parallel, Cm is greater than or equal to C2, the data processing device is configured to have a maximum convolution kernel size of Km for single processing, the maximum value between R2 and S2 is less than or equal to Km, the minimum value between R1 and S1 is greater than Km, and Km and Cm are positive integers.

10. A data processing method, comprising: Receive a convolution processing instruction, wherein the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 is used to represent the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; After parsing the convolution processing instruction, executing the convolution processing instruction using a convolution processing unit to obtain a second output feature map, wherein the second output feature map is equivalent to the first output feature map obtained by directly convolving the first input feature map with the first convolution kernel, wherein the equivalence includes numerical equivalence; The step of executing the convolution processing instruction using the convolution processing unit includes: Performing a first rearrangement transformation on the first convolution kernel to obtain a second convolution kernel, where the size of the second convolution kernel is expressed as (M, C2, R2, S2), where M represents the number of second convolution kernels, C2 represents the number of channels of the second convolution kernel, R2 represents the matrix height of the second convolution kernel, and S2 represents the matrix width of the second convolution kernel; Performing a second rearrangement transformation on the first input feature map to obtain a second input feature map, wherein the size of the second input feature map is expressed as (N, C2, H2, W2), N represents the batch of the second input feature map, C2 represents the number of channels of the second input feature map, H2 represents the matrix height of the second input feature map, W2 represents the matrix width of the second input feature map, C2=k×C1, k is an integer greater than 1 and k=(roundup(R1 / R2))×(roundup(S1 / S2)), and roundup(*) is a rounding-up function; Performing convolution processing using the second input feature map and the second convolution kernel to obtain the second output feature map; In which, the convolution processing unit is configured to process Cm channels in parallel, Cm is greater than or equal to C2, the convolution processing unit is configured so that the maximum convolution kernel size of a single processing is Km, the maximum value between R2 and S2 is less than or equal to Km, the minimum value between R1 and S1 is greater than Km, and Km and Cm are positive integers.

11. A processor comprising an instruction parsing unit and a convolution processing unit, wherein: The instruction parsing unit is used to receive and parse a convolution processing instruction, wherein the convolution processing instruction includes a first convolution kernel and a first input feature map as input parameters, the size of the first convolution kernel is expressed as (M, C1, R1, S1), M represents the number of the first convolution kernels, C1 represents the number of channels of the first convolution kernel, R1 represents the matrix height of the first convolution kernel, S1 represents the matrix width of the first convolution kernel, the size of the first input feature map is expressed as (N, C1, H1, W1), N represents the batch of the first input feature map, C1 represents the number of channels of the first input feature map, H1 represents the matrix height of the first input feature map, W1 represents the matrix width of the first input feature map, and the number of channels of the first input feature map is equal to the number of channels of the first convolution kernel; The convolution processing unit executes the data processing method according to any one of claims 1 to 9 after the instruction parsing unit parses the convolution processing instruction; In which, the convolution processing unit is configured to process Cm channels in parallel, Cm is greater than or equal to C2, the convolution processing unit is configured so that the maximum convolution kernel size of a single processing is Km, the maximum value between R2 and S2 is less than or equal to Km, the minimum value between R1 and S1 is greater than Km, and Km and Cm are positive integers.

12. An electronic device comprising: a memory that non-transitorily stores computer-executable instructions; a processor configured to execute the computer-executable instructions, Wherein, when the computer executable instructions are executed by the processor, the data processing method according to any one of claims 1 to 8 or the data processing method according to claim 10 is implemented.

13. A non-transitory computer-readable storage medium, wherein: The non-transitory computer-readable storage medium stores computer-executable instructions, When the computer-executable instructions are executed by a processor, the data processing method according to any one of claims 1 to 8 or the data processing method according to claim 10 is implemented.

Images