Convolution operation method and device, convolution processing method, equipment and storage medium

Abstract

A convolution operation method and apparatus, a convolution processing method, an electronic device, and a storage medium. The convolution operation method includes: obtaining a first matrix corresponding to at least part of the input data and a weight matrix corresponding to the convolution kernel; dividing the weight matrix into N sub-weight matrices; setting a Kth offset along a first direction corresponding to the first matrix for the Kth sub-weight matrix of the N sub-weight matrices; based on the Kth offset, performing a multi-step convolution operation on the first matrix using the Kth sub-weight matrix; accumulating the first to Nth sub-operation results corresponding to each convolution step in the multi-step convolution operation to obtain a first operation result corresponding to each convolution step; and using the multiple first operation results corresponding to the multi-step convolution operation as at least part of the output data and outputting the multiple first operation results. This data processing method can adapt to the hardware characteristics of the chip, fully utilize the chip computing power, and improve computing performance.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to a convolution operation method and apparatus, a convolution processing method, a device and a storage medium. Background Art

[0002] The field of deep learning in artificial intelligence (AI) involves a large number of convolution operations. In current computing devices, data processing integrated circuits such as central processing units (CPUs), graphics processing units (GPUs), and general-purpose computing on graphics processing units (GPGPUs) can execute programs to perform various functions such as convolutional neural network (CNN) operations.

[0003] GPGPUs are GPUs used for general-purpose computing, leveraging the high concurrent computing capabilities of graphics processing units (GPUs) to perform general-purpose computing tasks previously handled by central processing units (CPUs). With the rapid development of the internet, the emergence of artificial intelligence, and the subsequent innovations in traditional industries like aerospace, weather forecasting, and surveillance and security, using GPGPUs to replace CPUs for big data processing has become a major trend. Summary of the Invention

[0004] At least one embodiment of the present disclosure provides a convolution operation method, which includes: obtaining a first matrix corresponding to at least part of the data in the input data and a weight matrix corresponding to the convolution kernel, wherein the first matrix includes multiple elements arranged in two dimensions; dividing the weight matrix into N sub-weight matrices, wherein N is a positive integer; setting a K-th offset along a first direction corresponding to the first matrix for the K-th sub-weight matrix in the N sub-weight matrices, wherein 1≤K≤N and K is an integer; based on the K-th offset, using the K-th sub-weight matrix to perform a multi-step convolution operation on the first matrix to obtain multiple K-th sub-operation results corresponding to the multi-step convolution operations respectively; accumulating the 1st to Nth sub-operation results corresponding to each step of the multi-step convolution operation to obtain the first operation result corresponding to each step of the convolution operation; using the multiple first operation results corresponding to the multi-step convolution operation as at least part of the output data and outputting the multiple first operation results.

[0005] For example, the convolution operation method provided by at least one embodiment of the present disclosure further includes: filling the convolution kernel to obtain the weight matrix.

[0006] For example, the convolution operation method provided by at least one embodiment of the present disclosure further includes: filling at least part of the input data to obtain the first matrix.

[0007] For example, in the convolution operation method provided in at least one embodiment of the present disclosure, the weight matrix includes a one-dimensional weight vector.

[0008] For example, in the convolution operation method provided by at least one embodiment of the present disclosure, each sub-weight matrix of the N sub-weight matrices includes at least one element, the N sub-weight matrices include at least one first sub-weight matrix and at least one second sub-weight matrix, and the number of elements contained in each first sub-weight matrix in the at least one first sub-weight matrix is ​​the same as or different from the number of elements contained in each second sub-weight matrix in the at least one second sub-weight matrix.

[0009] For example, in the convolution operation method provided in at least one embodiment of the present disclosure, the number of elements contained in each first sub-weight matrix is ​​3, and the number of elements contained in each second sub-weight matrix is ​​5.

[0010] For example, in the convolution operation method provided by at least one embodiment of the present disclosure, the Kth offset corresponding to the Kth sub-weight matrix when it is the first sub-weight matrix is ​​different from the Kth offset corresponding to the Kth sub-weight matrix when it is the second sub-weight matrix.

[0011] For example, in the convolution operation method provided by at least one embodiment of the present disclosure, in response to the K-th sub-weight matrix being the first sub-weight matrix, the K-th offset is 0; in response to the K-th sub-weight matrix being the second sub-weight matrix, the K-th offset is equal to the number of elements contained in each first sub-weight matrix.

[0012] For example, the convolution operation method provided by at least one embodiment of the present disclosure also includes: dividing the first matrix into M first sub-matrices, wherein each first sub-matrix of the M first sub-matrices includes a plurality of elements arranged as r rows and s columns, M is a positive integer, r is a positive integer and s is a positive integer.

[0013] For example, in the convolution operation method provided in at least one embodiment of the present disclosure, the multi-step convolution operation includes a p-step convolution operation, where p is a positive integer, and based on the K-th offset, the K-th sub-weight matrix is ​​used to perform a multi-step convolution operation on the first matrix to obtain multiple K-th sub-operation results corresponding to the multi-step convolution operation, including: based on the K-th offset, the K-th sub-weight matrix is ​​used to perform p-step convolution operations with the M first sub-matrices respectively to obtain p K-th sub-operation results corresponding to the p-step convolution operation.

[0014] For example, in the convolution operation method provided in at least one embodiment of the present disclosure, the multi-step convolution operation includes a p-step convolution operation, where p is a positive integer, and the taking of multiple first operation results corresponding to the multi-step convolution operation as at least part of the output data and outputting the multiple first operation results includes: taking the p first operation results corresponding to the p-step convolution operation as a second sub-matrix corresponding to the at least part of the output data and outputting the second sub-matrix.

[0015] For example, in the convolution operation method provided in at least one embodiment of the present disclosure, obtaining the first matrix corresponding to at least part of the input data and the weight matrix corresponding to the convolution kernel includes: using a first scheduling unit to load the weight matrix; using a second scheduling unit to load the M first sub-matrices sequentially along the first direction.

[0016] For example, in the convolution operation method provided in at least one embodiment of the present disclosure, the multi-step convolution operation includes a p-step convolution operation, where p is a positive integer, and based on the K-th offset, the K-th sub-weight matrix is ​​used to perform a multi-step convolution operation on the first matrix to obtain multiple K-th sub-operation results corresponding to the multi-step convolution operation, including: based on the K-th offset, using the third scheduling unit to perform a p-step convolution operation on the K-th sub-weight matrix and the M first sub-matrices to obtain p K-th sub-operation results corresponding to the p-step convolution operation.

[0017] For example, in the convolution operation method provided in at least one embodiment of the present disclosure, the input data is divided into multiple parts, each of the multiple parts corresponds to a first sub-matrix, and the method further includes: using the second scheduling unit to load the M+1th first sub-matrix along the first direction, wherein the M+1th first sub-matrix includes multiple elements arranged as r rows and s columns, r is a positive integer and s is a positive integer; based on the Kth offset, using the third scheduling unit to perform a q-step convolution operation on the Kth sub-weight matrix and the M+1th first sub-matrix to obtain multiple Kth sub-operation results corresponding to the q-step convolution operations, wherein q is a positive integer.

[0018] For example, in the convolution operation method provided in at least one embodiment of the present disclosure, the first direction is the column direction of the first matrix.

[0019] For example, in the convolution operation method provided in at least one embodiment of the present disclosure, the step size of the multi-step convolution operation is 1.

[0020] At least one embodiment of the present disclosure also provides a convolution processing method, which includes: obtaining input data; dividing the input data into multiple parts, wherein each of the multiple parts corresponds to a first matrix; for the first matrix corresponding to each part of the input data, performing the multi-step convolution operation using the convolution operation method provided according to any embodiment of the present disclosure, obtaining the at least partial output data corresponding to the multi-step convolution operation and outputting the at least partial output data; obtaining complete output data based on the at least partial output data corresponding to each part of the input data.

[0021] At least one embodiment of the present disclosure further provides a convolution operation device, which includes: an acquisition module configured to acquire a first matrix corresponding to at least part of the input data and a weight matrix corresponding to the convolution kernel, wherein the first matrix includes a plurality of elements arranged in two dimensions; a division module configured to divide the weight matrix into N sub-weight matrices, wherein N is a positive integer; a processing module configured to set a K-th offset along a first direction corresponding to the first matrix for a K-th sub-weight matrix of the N sub-weight matrices, wherein 1≤K≤N and K is an integer; a calculation module configured to perform a multi-step convolution operation on the first matrix using the K-th sub-weight matrix based on the K-th offset to obtain a plurality of K-th sub-operation results corresponding to the multi-step convolution operations respectively; an accumulation module configured to accumulate the first to N-th sub-operation results corresponding to each step of the multi-step convolution operation to obtain a first operation result corresponding to each step of the convolution operation; and an output module configured to use the plurality of first operation results corresponding to the multi-step convolution operation as at least part of the output data and output the plurality of first operation results.

[0022] At least one embodiment of the present disclosure further provides an electronic device. The electronic device includes: a processor; and a memory including one or more computer program modules; wherein the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the convolution operation method or the convolution processing method provided in any embodiment of the present disclosure.

[0023] At least one embodiment of the present disclosure further provides a storage medium storing non-transitory computer-readable instructions, which, when executed by a computer, implements the convolution operation method provided by any embodiment of the present disclosure or the convolution processing method provided by any embodiment of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings in the following description only relate to some embodiments of the present disclosure, rather than limiting the present disclosure.

[0025] Figure 1 A schematic diagram of the process of performing a convolution operation on a large-size convolution kernel and input data;

[0026] Figure 2 An exemplary flow chart of a convolution operation method provided for at least one embodiment of the present disclosure;

[0027] Figure 3A A schematic diagram of an example of weight matrix partitioning provided by at least one embodiment of the present disclosure;

[0028] Figure 3B A schematic diagram of an example of a convolution operation method provided by at least one embodiment of the present disclosure;

[0029] Figure 4A A schematic diagram of an example of first matrix division provided in at least one embodiment of the present disclosure;

[0030] Figure 4B A schematic diagram of another example of a convolution operation method provided by at least one embodiment of the present disclosure;

[0031] Figure 4C for Figure 4B A schematic diagram of an example of weight matrix partitioning in ;

[0032] Figure 5 A schematic diagram of yet another example of a convolution operation method provided in at least one embodiment of the present disclosure;

[0033] Figure 6 A schematic diagram of yet another example of a convolution operation method provided in at least one embodiment of the present disclosure;

[0034] Figure 7 Another exemplary flow chart of a convolution operation method provided for at least one embodiment of the present disclosure;

[0035] Figure 8 An exemplary flow chart of a convolution processing method provided for at least one embodiment of the present disclosure;

[0036] Figure 9A schematic block diagram of a convolution operation device provided for at least one embodiment of the present disclosure;

[0037] Figure 10 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure;

[0038] Figure 11 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure; and

[0039] Figure 12 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure. DETAILED DESCRIPTION

[0040] To make the purpose, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0041] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The words "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "an" or "the" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0042] The present disclosure is described below using several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numeral in each drawing.

[0043] By utilizing the computing units of GPGPU chips, the convolution computing capabilities can be greatly improved. These computing units have become the distinctive feature of the GPGPU architecture that is different from the CPU. For example, in GPGPU, there are computing units such as the Compute Unified Device Architecture (CUDA) core unit (CUDA core), Special Function Unit (SFU), Double Precision Unit (DPU) and Tensor Core Unit (tensorcore). For example, GPGPU uses programmable multiprocessors as the division granularity, grouping and organizing various types of computing units in a certain proportion, thereby supporting a variety of data processing needs in various scenarios such as general computing, scientific computing and neural network computing.

[0044] For example, for convolution kernels of regular sizes, calculations can be performed directly using specific computing units; for convolution kernels of large sizes, calculations often cannot be performed directly on specific computing units due to the limitations of chip hardware characteristics. For example, taking the tensor core unit as an example, when the height of the convolution kernel is less than or equal to 7 and the width is less than or equal to 7, the tensor core unit can be directly used for calculations; when the height of the convolution kernel is greater than 7 or the width is greater than 7, the tensor core unit cannot be directly used for calculations.

[0045] Figure 1 A diagram showing the process of performing a convolution operation on a large-size convolution kernel and input data. Figure 1 As shown, the large-size convolution kernel is a one-dimensional convolution kernel with a height of 31 (for example, the size of the one-dimensional convolution kernel is recorded as weight(31×1)); the input data is two-dimensionally arranged data with a height of 40 and a width of 16 (for example, the size of the input data is recorded as input(40×16)); the output data obtained by the convolution operation is also two-dimensionally arranged data with a height of 40 and a width of 16 (for example, the size of the output data is recorded as output(40×16)). For example, in the operation unit, in order to perform parallel computing, the input data can be two-dimensionally arranged input data, or it can be two-dimensional input data obtained by reshaping the one-dimensional input data.

[0046] For example, Figure 1As shown in the figure, the convolution kernel weight (31×1) slides along the row and column directions of the input data to complete the two-dimensional convolution operation and obtain two-dimensional output data; in each convolution step, the 31 elements on the convolution kernel are multiplied by the input data to obtain 31 sub-operation results. These 31 sub-operation results need to be accumulated to obtain a convolution operation result on the two-dimensional output data. However, due to the limitation of the computing power of the operation unit, this accumulation process cannot be performed directly on a specific operation unit (for example, the tensor core). Instead, multiple sub-operation results need to be input into the memory first, then accumulated in the memory, and then the accumulated results are transferred back to the operation unit. During the above convolution operation process, the frequent data transmission between the operation unit and the memory causes a certain degree of performance loss of the chip.

[0047] At least one embodiment of the present disclosure provides a convolution operation method, which includes: obtaining a first matrix corresponding to at least part of the data in the input data and a weight matrix corresponding to the convolution kernel, wherein the first matrix includes multiple elements arranged in two dimensions; dividing the weight matrix into N sub-weight matrices, wherein N is a positive integer; setting a K-th offset along a first direction corresponding to the first matrix for the K-th sub-weight matrix in the N sub-weight matrices, wherein 1≤K≤N and K is an integer; based on the K-th offset, using the K-th sub-weight matrix to perform a multi-step convolution operation on the first matrix to obtain multiple K-th sub-operation results corresponding to the multi-step convolution operations respectively; accumulating the 1st to Nth sub-operation results corresponding to each step of the multi-step convolution operation to obtain a first operation result corresponding to each step of the convolution operation; using the multiple first operation results corresponding to the multi-step convolution operation as at least part of the output data and outputting the multiple first operation results.

[0048] At least one embodiment of the present disclosure further provides a convolution processing method, a convolution operation device, an electronic device, and a storage medium.

[0049] The method, apparatus, device, and storage medium provided in at least one embodiment of the present disclosure decompose a large-size convolution kernel into multiple weight matrices equivalently based on the storage capacity, computing power, and other characteristics of the chip. The weight matrix can directly implement accelerated computing on the chip's computing unit, and can adapt to the chip's hardware characteristics, fully utilize the chip's computing power, and improve computing performance.

[0050] Hereinafter, at least one embodiment of the present disclosure will be described in detail with reference to the accompanying drawings. It should be noted that the same reference numerals in different drawings will be used to refer to the same elements described above.

[0051] Figure 2 An exemplary flow chart of a convolution operation method provided for at least one embodiment of the present disclosure.

[0052] For example, Figure 2 As shown, at least one embodiment of the present disclosure provides a convolution operation method, which includes the following steps S110 to S160.

[0053] Step S110: Obtain a first matrix corresponding to at least part of the input data and a weight matrix corresponding to the convolution kernel;

[0054] Step S120: Divide the weight matrix into N sub-weight matrices, where N is a positive integer;

[0055] Step S130: setting a K-th offset along the first direction corresponding to the first matrix for the K-th sub-weight matrix among the N sub-weight matrices, where 1≤K≤N and K is an integer;

[0056] Step S140: Based on the K-th offset, use the K-th sub-weight matrix to perform a multi-step convolution operation on the first matrix to obtain a plurality of K-th sub-operation results corresponding to the multi-step convolution operations;

[0057] Step S150: Accumulating the first to Nth sub-operation results corresponding to each convolution step in the multi-step convolution operation to obtain a first operation result corresponding to each convolution step;

[0058] Step S160: using the multiple first operation results corresponding to the multi-step convolution operation as at least part of the output data and outputting the multiple first operation results.

[0059] For example, in step S110, the input data is data to be convolved (activation), such as a feature map. To enable parallel computation within the computation unit, the input data can be two-dimensionally arranged input data or two-dimensional input data obtained by transforming one-dimensional input data, although this is not a limitation of the present disclosure. For example, in step S110, at least a portion of the input data is selected as a first matrix, where the first matrix includes a plurality of elements arranged in two dimensions.

[0060] For example, in some examples, to obtain at least a portion of output data of a specific size, the convolution operation method provided in at least one embodiment of the present disclosure further includes padding at least a portion of the input data to obtain a first matrix. For example, the padding operation is a padding operation, and the padded portion includes elements with values ​​of 0, thereby not affecting the convolution operation result.

[0061] For example, when at least part of the output data obtained by performing a convolution operation on at least part of the unfilled input data does not meet a specific size requirement, it is necessary to first fill at least part of the data in the input data to obtain a first matrix, and then perform a convolution operation on the first matrix; when at least part of the output data obtained by performing a convolution operation on at least part of the unfilled input data meets a specific size requirement, at least part of the data in the input data may not be filled; whether to fill at least part of the data in the input data can also be selected according to other actual needs, and the embodiments of the present disclosure are not limited to this.

[0062] For example, in step S110, the convolution kernel is a convolution kernel used to perform a convolution operation on the input data, and the matrix corresponding to the convolution kernel is used as the weight matrix. For example, when the convolution kernel is a one-dimensional convolution kernel, the weight matrix includes a one-dimensional weight vector; when the convolution kernel is a two-dimensional convolution kernel, the weight matrix can be a matrix of elements arranged in a two-dimensional manner. The dimension of the convolution kernel can be selected according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0063] For example, in some examples, before step S120, the convolution operation method provided by at least one embodiment of the present disclosure further includes: padding the convolution kernel to obtain a weight matrix. For example, the padding operation is a padding operation, and the element values ​​included in the padded part are 0, so it does not affect the convolution operation result. For example, when the convolution kernel size does not match the first matrix size corresponding to the input data or does not meet the division requirements, it is necessary to first pad the convolution kernel to obtain the weight matrix, and then divide the weight matrix; when the convolution kernel size matches the first matrix size or meets the division requirements, it is not necessary to pad the convolution kernel, but directly use the convolution kernel as the weight matrix; whether to pad the convolution kernel can also be selected according to other actual needs, and the embodiments of the present disclosure do not limit this.

[0064] For example, in step S120, the weight matrix is ​​divided into N sub-weight matrices according to the computing power requirements of the computing unit in the chip; the size of each sub-weight matrix meets the computing power requirements of the computing unit, so that calculations can be performed directly on a specific computing unit.

[0065] For example, each of the N sub-weight matrices includes at least one element; the N sub-weight matrices include at least one first sub-weight matrix and at least one second sub-weight matrix, and the number of elements contained in each first sub-weight matrix is ​​the same as or different from the number of elements contained in each second sub-weight matrix. For example, in some examples, the number of elements contained in the first sub-weight matrix may be 3, and the number of elements contained in the second sub-weight matrix may be 5; in other examples, the number of elements contained in the first sub-weight matrix may be 4, and the number of elements contained in the second sub-weight matrix may be 4.

[0066] It should be noted that as long as the sum of the sizes of the multiple sub-weight matrices into which the weight matrix is ​​divided does not exceed the hardware limitation of the operation unit (for example, the maximum convolution kernel size that the operation unit can support can be 7×7, 8×8, etc.), the appropriate size of the sub-weight matrix can be selected according to actual needs, and the embodiments of the present disclosure do not impose any restrictions on this. For example, in some examples, the size of the sub-weight matrix can be an odd number (for example, 3×1, 5×1, or 7×1, etc.); in other examples, the size of the sub-weight matrix can be an even number (for example, 2×1, 4×1, or 6×1, etc.).

[0067] For example, based on the size requirements of the above-mentioned sub-weight matrices, the number of elements contained in the first sub-weight matrix and the second sub-weight matrix can be the same or different (can be an odd number or an even number); for example, the size combination of the first sub-weight matrix / the second sub-weight matrix can be 1×1 / 7×1, 3×1 / 5×1, 4×1 / 4×1, etc.; the specific selection can be made according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0068] Figure 3A A schematic diagram of an example of weight matrix partitioning provided in at least one embodiment of the present disclosure.

[0069] For example, Figure 3A As shown, taking a large-size one-dimensional convolution kernel as an example, its height is e (for example, the size of the one-dimensional convolution kernel is recorded as weight(e×1)), where e is a positive integer and e is an odd number. For example, the one-dimensional convolution kernel includes multiple elements w1, w2, ..., we.

[0070] For example, first, the one-dimensional convolution kernel is padded with one element w0 (w0=0) in front, expanding it into a weight matrix weight((e+1)×1) including e+1 elements; then, the weight matrix weight((e+1)×1) is divided into N / 2 blocks of weight(8×1) matrices; each block of weight(8×1) is further divided into a first sub-weight matrix sub_weight(3×1) and a second sub-weight matrix sub_weight(5×1), where N is a positive integer and N is an even number. That is, the weight matrix weight((e+1)×1) is divided into N sub-weight matrices, the N sub-weight matrices including the first sub-weight matrix sub_weight(3×1) and the second sub-weight matrix sub_weight(5×1), the number of elements contained in the first sub-weight matrix is ​​3, and the number of elements contained in the second sub-weight matrix is ​​5.

[0071] For example, Figure 3AAs shown, the N sub-weight matrices include N / 2 first sub-weight matrices (ww0, ww2, ..., ww(N-2)) with a height of 3 and a width of 1 and N / 2 second sub-weight matrices (ww1, ww3, ..., ww(N-1)) with a height of 5 and a width of 1, thereby meeting the requirements of the computing power of the operation unit for the convolution kernel size (for example, when calculating in the tensor core, the height and width of the sub-weight matrix are required to be less than or equal to 7).

[0072] It should be noted that Figure 3A The weight matrix division method shown is only an example. In other examples (for example, when the weight matrix is ​​a two-dimensional matrix), the weight matrix can also choose a different division method; or, the weight matrix can also choose other division methods according to actual needs, and the embodiments of the present disclosure do not limit this.

[0073] It should be noted that the convolution operation method provided in the embodiment of the present disclosure is only for convolution kernels with large sizes. When the size of the convolution kernel does not meet the computing power of the operation unit, the weight matrix corresponding to the convolution kernel is divided and processed; when the convolution kernel is a convolution kernel of regular size, it is not necessary to implement the division processing in the embodiment of the present disclosure on the weight matrix corresponding to the convolution kernel, but it can be calculated directly on a specific operation unit.

[0074] For example, after the weight matrix is ​​divided into the first sub-weight matrix and the second sub-weight matrix, the operation unit defaults to calculating the first sub-weight matrix from the elements of the 1st row and 1st column of the first matrix when performing the convolution operation, and at the same time defaults to calculating the second sub-weight matrix from the elements of the 1st row and 1st column of the first matrix. This will cause the convolution operation results obtained by the first sub-weight matrix and the second sub-weight matrix to be different from the convolution operation results of the undivided weight matrix after the accumulation, which will cause the convolution operation after the weight matrix is ​​divided to be inequivalent (that is, an incorrect convolution operation result will be obtained).

[0075] For example, to solve the above problem, in step S130, an offset is set for each sub-weight matrix along the first direction to ensure that the starting operation position of the first sub-weight matrix and the starting operation position of the second sub-weight matrix correspond to their starting operation positions in the unpartitioned weight matrix, so that a correct convolution operation result can be obtained. For example, the first direction can be the column direction of the first matrix or the row direction of the first matrix, and the embodiments of the present disclosure are not limited to this.

[0076] For example, for the Kth sub-weight matrix among the N sub-weight matrices (where 1≤K≤N and K is an integer), the Kth offset corresponding to the Kth sub-weight matrix when it is the first sub-weight matrix is ​​different from the Kth offset corresponding to the Kth sub-weight matrix when it is the second sub-weight matrix. For example, in some examples, in response to the Kth sub-weight matrix being the first sub-weight matrix, the Kth offset is 0; in response to the Kth sub-weight matrix being the second sub-weight matrix, the Kth offset is equal to the number of elements contained in each first sub-weight matrix. For example, Figure 3A Taking the multiple sub-weight matrices in as an example, in response to the K-th sub-weight matrix being the first sub-weight matrix sub_weight(3×1), the K-th offset is 0; in response to the K-th sub-weight matrix being the second sub-weight matrix sub_weight(5×1), the K-th offset is equal to the number of elements contained in the first sub-weight matrix sub_weight(3×1), that is, the K-th offset is 3.

[0077] For example, for the Kth sub-weight matrix among the N sub-weight matrices, a Kth offset relative to the first matrix is ​​set along the first direction, so that in step S140, based on the Kth offset, a multi-step convolution operation can be performed on the Kth sub-weight matrix and the elements at the corresponding positions of the first matrix, thereby making the multi-step convolution operation of the N sub-weight matrices completely equivalent to the convolution operation of the weight matrix that has not been divided.

[0078] For example, in some examples, the step size of the multi-step convolution operation is 1; the step size of the multi-step convolution operation can also select other step size values ​​according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0079] For example, in step S150, the 1st to Nth sub-operation results corresponding to each step of the multi-step convolution operation obtained in step S140 are accumulated to obtain the first operation result corresponding to each step of the convolution operation.

[0080] For example, the first operation result obtained by the N sub-weight matrices in each step of the convolution operation is the same as the convolution operation result obtained by the weight matrix that has not been divided in each step of the convolution operation, thereby achieving the complete equivalence of the N convolution operations of the N sub-weight matrices and the one convolution operation of the weight matrix that has not been divided in each step of the convolution operation.

[0081] For example, since each convolution operation in step S150 corresponds to an accumulated first operation result, multiple first operation results can be obtained after multiple convolution operations; therefore, in step S160, the multiple first operation results can be used as at least part of the output data and output. Here, the multiple first operation results output are operation results of the portion of the input data corresponding to the first matrix.

[0082] Figure 3B A schematic diagram of an example of a convolution operation method provided by at least one embodiment of the present disclosure. The convolution operation method is, for example, Figure 2 The convolution operation method shown.

[0083] For example, Figure 3B As shown, in step S110, a first matrix corresponding to at least part of the input data and a weight matrix corresponding to the convolution kernel are obtained. For example, the convolution kernel is Figure 3A In step S120, according to the large one-dimensional convolution kernel in Figure 3A The division method in , the weight matrix weight((e+1)×1) corresponding to the one-dimensional convolution kernel is divided into N sub-weight matrices (ww0, ww1, ..., ww(N-1)); Figure 3B As shown, the N sub-weight matrices include a first sub-weight matrix (ww0, ww2, ..., ww(N-2)) and a second sub-weight matrix (ww1, ww3, ..., ww(N-1)), the number of elements contained in the first sub-weight matrix is ​​3, and the number of elements contained in the second sub-weight matrix is ​​5.

[0084] In step S130, for the Kth sub-weight matrix ww(K-1) among the N sub-weight matrices, a Kth offset (offset_y) relative to the first matrix is ​​set along a first direction (for example, the first direction is the column direction y of the first matrix).

[0085] For example, the Kth offset corresponding to the Kth sub-weight matrix ww(K-1) when it is the first sub-weight matrix is ​​different from the Kth offset corresponding to the Kth sub-weight matrix ww(K-1) when it is the second sub-weight matrix. For example, in response to the Kth sub-weight matrix ww(K-1) being the first sub-weight matrix, the Kth offset is 0; in response to the Kth sub-weight matrix ww(K-1) being the second sub-weight matrix, the Kth offset is equal to the number of elements contained in each first sub-weight matrix, that is, the Kth offset of the second sub-weight matrix is ​​3.

[0086] For example, Figure 3B As shown, the sub-weight matrix ww0 is the first sub-weight matrix, and the first offset corresponding to ww0 is 0 (offset_y=0); the sub-weight matrix ww1 is the second sub-weight matrix, and the second offset corresponding to ww1 is 3 (offset_y=3); ...; the sub-weight matrix ww(N-2) is the first sub-weight matrix, and the (N-1)th offset corresponding to ww(N-2) is 0 (offset_y=0); the sub-weight matrix ww(N-1) is the second sub-weight matrix, and the Nth offset corresponding to ww(N-1) is 3 (offset_y=3).

[0087] For example, in step S140, since the first sub-weight matrix offset_y=0, the first sub-weight matrix can directly perform multi-step convolution operation with the first matrix starting from the default starting operation position; since the second sub-weight matrix offset_y=3, it is necessary to first move the first matrix upward by 3 elements so that the second sub-weight matrix corresponds to the correct starting operation position, and then perform multi-step convolution operation on the second sub-weight matrix and the first matrix.

[0088] For example, Figure 3B As shown, in step S140, based on the above-mentioned K-th offset, a multi-step convolution operation is performed on the K-th sub-weight matrix ww(K-1) and the elements at the corresponding positions of the first matrix to obtain multiple K-th sub-operation results corresponding to the multi-step convolution operation of the sub-weight matrix ww(K-1); for example, the multi-step convolution operation of the 1st sub-weight matrix ww0 corresponds to multiple 1st sub-operation results, the multi-step convolution operation of the 2nd sub-weight matrix ww1 corresponds to multiple 2nd sub-operation results,..., the multi-step convolution operation of the N-th sub-weight matrix ww(N-1) corresponds to multiple N-th sub-operation results.

[0089] For example, in step S150, taking one of the multi-step convolution operations as an example (eg Figure 3B The first convolution operation in the first convolution operation is performed), and the 1st sub-operation result to the Nth sub-operation result corresponding to the first convolution operation are accumulated to obtain the first operation result corresponding to the first convolution operation.

[0090] For example, Figure 3B As shown, the first operation results obtained by the N sub-weight matrices (ww0, ww1, ..., ww(N-1)) in the first convolution operation are the same as the convolution operation results obtained by the unpartitioned weight matrix weight ((e+1)×1) in the first convolution operation, thereby achieving the complete equivalence of the N convolution operations of the N sub-weight matrices in the first convolution operation and the one convolution operation of the unpartitioned weight matrix.

[0091] For example, Figure 3B As shown, since multiple first operation results can be obtained through multi-step convolution operations in step S150, in step S160, the multiple first operation results can be used as at least part of the output data and the above multiple first operation results can be output.

[0092] For example, in some examples, the convolution operation method provided by at least one embodiment of the present disclosure further includes: dividing the first matrix into M first sub-matrices. For example, each of the M first sub-matrices includes a plurality of elements arranged in r rows and s columns, where r is a positive integer and s is a positive integer.

[0093] Figure 4A A schematic diagram of an example of first matrix division provided in at least one embodiment of the present disclosure.

[0094] For example, Figure 4A As shown, the first matrix includes multiple elements arranged as M×r rows and s columns, and the first matrix is ​​divided into M first sub-matrices (B0, B1, ..., B(M-1)), each of the M first sub-matrices includes multiple elements arranged as r rows and s columns, where r is a positive integer and s is a positive integer.

[0095] It should be noted that Figure 4A The first matrix division method shown is only an example. The first matrix may also be divided in other different ways according to actual needs, and the embodiments of the present disclosure are not limited to this.

[0096] For example, in Figure 2 When performing a multi-step convolution operation on the first matrix using each sub-weight matrix in step S140, the multi-step convolution operation includes a p-step convolution operation, where p is a positive integer. For example, for M first sub-matrices (B0, B1, ..., B(M-1)), Figure 2 Step S140 may further include: based on the K-th offset, using the K-th sub-weight matrix to perform p-step convolution operations with the M first sub-matrices respectively, to obtain p K-th sub-operation results corresponding to the p-step convolution operations.

[0097] For example, based on the p-step convolution operation obtained in step S140 above, the p-th K-th sub-operation results corresponding to each of the p-step convolution operations are further Figure 2 In step S150, the first sub-operation result to the Nth sub-operation result corresponding to each convolution operation in the p-step convolution operation are accumulated to obtain the first operation result corresponding to each convolution operation; Figure 2 Step S160 may further include: using the p first operation results corresponding to the p-step convolution operation as a second sub-matrix corresponding to at least part of the output data and outputting the second sub-matrix.

[0098] Figure 4B A schematic diagram of another example of a convolution operation method provided by at least one embodiment of the present disclosure; Figure 4C for Figure 4B A schematic diagram of an example of weight matrix partitioning in . The convolution operation method is, for example, Figure 2 The convolution operation method shown, and the step size of the convolution operation is 1, for example.

[0099] For example, Figure 4BAs shown, the first matrix includes multiple elements arranged in 4×8 rows and 8 columns (i.e., M=4, r=s=8), and the first matrix is ​​divided into four first sub-matrices (B0, B1, B2, B3), each of which includes multiple elements arranged in 8 rows and 8 columns. For example, the weight matrix is ​​a one-dimensional weight vector with 32 rows and 1 column (for example, the size of the weight matrix is ​​recorded as weight(32×1)), and the weight matrix includes 32 elements w0, w1, ..., w31. For example, the weight matrix can be expanded by padding the front of a one-dimensional convolution kernel with 31 rows and 1 column with one element w0 (w0=0).

[0100] For example, Figure 4C As shown, in step S120, according to Figure 3A In the division method, the weight matrix weight (32×1) is divided into 8 sub-weight matrices (i.e., N=8). The 8 sub-weight matrices include 4 first sub-weight matrices (W00, W10, W20, W30) and 4 second sub-weight matrices (W01, W11, W21, W31). The number of elements contained in the first sub-weight matrix is ​​3, and the number of elements contained in the second sub-weight matrix is ​​5. For example, the element distribution in the 8 sub-weight matrices is shown in Table 1 below:

[0101] Table 1 Element distribution in the 8 sub-weight matrices

[0102]

[0103] For example, in step S130, for the K-th sub-weight matrix among the 8 sub-weight matrices, a K-th offset (offset_y) relative to the first matrix is ​​set along a first direction (e.g., the first direction is the column direction y of the first matrix). For example, in response to the K-th sub-weight matrix being the first sub-weight matrix, the K-th offset is 0; in response to the K-th sub-weight matrix being the second sub-weight matrix, the K-th offset is equal to the number of elements included in each first sub-weight matrix, that is, the K-th offset of the second sub-weight matrix is ​​3.

[0104] For example, Figure 4B As shown, the sub-weight matrix W00 is the first sub-weight matrix, and the first offset corresponding to W00 is 0 (offset_y=0); the sub-weight matrix W01 is the second sub-weight matrix, and the second offset corresponding to W01 is 3 (offset_y=3); ...; the sub-weight matrix W30 is the first sub-weight matrix, and the seventh offset corresponding to W30 is 0 (offset_y=0); the sub-weight matrix W31 is the second sub-weight matrix, and the eighth offset corresponding to W31 is 3 (offset_y=3).

[0105] For example, in step S140, each sub-weight matrix is ​​used to perform a multi-step convolution operation on the four first sub-matrices (B0, B1, B2, B3), and the multi-step convolution operation includes a p-step convolution operation, where p is a positive integer. Figure 4B As shown, since the first sub-weight matrix offset_y=0, the first sub-weight matrix can directly perform p-step convolution operation with the four first sub-matrices starting from the default starting operation position (in Figure 4B In the example, the default starting position is the position of the first row and first column element of the first sub-matrix B0); since the second sub-weight matrix offset_y = 3, it is necessary to first move the first matrix upward by 3 elements so that the second sub-weight matrix corresponds to the correct starting position, and then perform a p-step convolution operation on the second sub-weight matrix and the four first sub-matrices (in Figure 4B In the example of , the correct starting operation position corresponding to the second sub-weight matrix is ​​the position of the element in the 4th row and 1st column of the first sub-matrix B0).

[0106] For example, specifically, Figure 4B In the example, if the first matrix does not move, the second sub-weight matrix W01 starts the convolution operation from the position of the 1st row and 1st column element of the first sub-matrix B0 by default, the second sub-weight matrix W11 starts the convolution operation from the position of the 1st row and 1st column element of the first sub-matrix B1 by default, the second sub-weight matrix W21 starts the convolution operation from the position of the 1st row and 1st column element of the first sub-matrix B2 by default, and the second sub-weight matrix W31 starts the convolution operation from the position of the 1st row and 1st column element of the first sub-matrix B3 by default; however, after moving the first matrix up by 3 elements, the second sub-weight matrix W01 starts the convolution operation from the position of the 4th row and 1st column element of the first sub-matrix B0, the second sub-weight matrix W11 starts the convolution operation from the position of the 4th row and 1st column element of the first sub-matrix B1, the second sub-weight matrix W21 starts the convolution operation from the position of the 4th row and 1st column element of the first sub-matrix B2, and the second sub-weight matrix W31 starts the convolution operation from the position of the 4th row and 1st column element of the first sub-matrix B3. Therefore, by setting the Kth offset for the Kth sub-weight matrix, it is ensured that the starting operation positions of the first sub-weight matrix and the second sub-weight matrix correspond to their starting operation positions in the undivided weight matrix, so that the correct convolution operation result can be obtained.

[0107] For example, in step S140, for the Kth sub-weight matrix, based on the 1st offset to the 8th offset determined by the first sub-weight matrix and the second sub-weight matrix, the 1st sub-weight matrix W00 to the 8th sub-weight matrix W31 are used to perform p-step convolution operations with the 4 first sub-matrices respectively.

[0108] For example, Figure 4B As shown in the figure, the solid arrows represent the process of performing p-step convolution operations on the first sub-weight matrix (W00, W10, W20, W30) with the four first sub-matrices (B0, B1, B2, B3); the dotted arrows represent the process of performing p-step convolution operations on the second sub-weight matrix (W01, W11, W21, W31) with the four first sub-matrices (B0, B1, B2, B3). For example, after performing p-step convolution operations on the eight sub-weight matrices with the four first sub-matrices, the first to eighth sub-operation results corresponding to each step of the p-step convolution operation are obtained.

[0109] For example, further, in step S150, the first sub-operation results to the eighth sub-operation results corresponding to each convolution operation in the p-step convolution operation of the 8 sub-weight matrices are accumulated to obtain the first operation result corresponding to each convolution operation.

[0110] For example, Figure 4B As shown, in step S160, the p first operation results corresponding to the p-step convolution operation are used as the second sub-matrix C2 corresponding to at least part of the output data, and the second sub-matrix C2 is output. That is, the 8 sub-weight matrices and the 4 first sub-matrices (B0, B1, B2, B3) are subjected to the p-step convolution operation to obtain the second sub-matrix C2.

[0111] For example, the first eight convolution operations of the eight sub-weight matrices and the four first sub-matrices (B0, B1, B2, B3) can only obtain the first row elements of the second sub-matrix C2; therefore, in order to obtain the second sub-matrix C2, before step S140, at least part of the input data may be padded to obtain the first matrix. Figure 4B As shown, a first submatrix B4 is padded after the first submatrix B3. The element values ​​of the padded first submatrix B4 are 0, so it does not affect the convolution result. For example, the 8 sub-weight matrices are convolved with the 4 first submatrices (B0, B1, B2, B3) and the padded first submatrix B4 (i.e., from the last row of B0 to the second to last row of B4) in the last 8 steps to obtain the last row of elements of the second submatrix C2.

[0112] Figure 5 A schematic diagram of another example of a convolution operation method provided in at least one embodiment of the present disclosure. The convolution operation method is, for example, Figure 2 The convolution operation method shown, and the step size of the convolution operation is 1, for example.

[0113] For example, Figure 5As shown, the first matrix includes multiple elements arranged in 8×8 rows and 8 columns (i.e., M=8, r=s=8), and the first matrix is ​​divided into 8 first sub-matrices (B0, B1, ..., B7), each of which includes multiple elements arranged in 8 rows and 8 columns. For example, the weight matrix is ​​a one-dimensional weight vector with 32 rows and 1 column (for example, the size of the weight matrix is ​​recorded as weight(32×1)), and the weight matrix includes 32 elements w0, w1, ..., w31. For example, the weight matrix can be expanded by padding the front of a one-dimensional convolution kernel with 31 rows and 1 column with one element w0 (w0=0).

[0114] For example, in step S120, according to Figure 3A The weight matrix weight (32×1) is divided into 8 sub-weight matrices (i.e., N=8) according to the division method in . As shown in Table 1, the 8 sub-weight matrices include 4 first sub-weight matrices (W00, W10, W20, W30) and 4 second sub-weight matrices (W01, W11, W21, W31). The number of elements contained in the first sub-weight matrix is ​​3, and the number of elements contained in the second sub-weight matrix is ​​5.

[0115] For example, in step S130, for the K-th sub-weight matrix among the 8 sub-weight matrices, a K-th offset (offset_y) relative to the first matrix is ​​set along a first direction (e.g., the first direction is the column direction y of the first matrix). For example, in response to the K-th sub-weight matrix being the first sub-weight matrix, the K-th offset is 0; in response to the K-th sub-weight matrix being the second sub-weight matrix, the K-th offset is equal to the number of elements included in each of the first sub-weight matrices, that is, the K-th offset of the second sub-weight matrix is ​​3.

[0116] For example, Figure 5 As shown, the sub-weight matrix W00 is the first sub-weight matrix, and the first offset corresponding to W00 is 0 (offset_y=0); the sub-weight matrix W01 is the second sub-weight matrix, and the second offset corresponding to W01 is 3 (offset_y=3); ...; the sub-weight matrix W30 is the first sub-weight matrix, and the seventh offset corresponding to W30 is 0 (offset_y=0); the sub-weight matrix W31 is the second sub-weight matrix, and the eighth offset corresponding to W31 is 3 (offset_y=3).

[0117] For example, Figure 5As shown, the 8 sub-weight matrices and the 8 first sub-matrices (B0, B1, ..., B7) are subjected to 8×p-step convolution operations to obtain 8 second sub-matrices (C0, C1, ..., C7), where p is a positive integer; therefore, in order to obtain the above-mentioned 8 second sub-matrices, before step S140, 2 first sub-matrices B-1 and B-2 are sequentially filled before the first sub-matrix B0, and 2 first sub-matrices B8 and B9 are filled after the first sub-matrix B7, and the element values ​​included in the 4 filled first sub-matrices B-1, B-2, B8 and B9 are 0, and thus do not affect the convolution operation results.

[0118] For example, Figure 5 As shown, in step S140, 8 sub-weight matrices are used to perform 8×p-step convolution operations on the 8 first sub-matrices (B0, B1, ..., B7) and the 4 filled first sub-matrices (B-1, B-2, B8, B9) to obtain 8 second sub-matrices (C0, C1, ..., C7). For example, since the first sub-weight matrix offset_y = 0, the first sub-weight matrix can be directly convolved with multiple first sub-matrices starting from the default starting position (at Figure 5 In the example, the default starting position is the position of the first row and first column element of the first sub-matrix B-2); since the second sub-weight matrix offset_y=3, it is necessary to first move the first matrix upward by 3 elements so that the second sub-weight matrix corresponds to the correct starting position, and then perform 8×p-step convolution operations on the second sub-weight matrix and multiple first sub-matrices (in Figure 5 In the example, the correct starting operation position corresponding to the second sub-weight matrix is ​​the position of the 4th row and 1st column element of the first sub-matrix B-2).

[0119] For example, Figure 5 As shown on the right side of , the default starting position of the second sub-weight matrix is ​​also the position of the first row and first column element of the first sub-matrix B-2, that is, Figure 5 The dotted box on the right is located, which will cause the second sub-weight matrix to obtain the wrong convolution operation result; after moving the first matrix up by 3 elements, that is, Figure 5 The actual position of the first matrix on the right makes the second sub-weight matrix correspond to the correct starting operation position (that is, the position of the 4th row and 1st column element of the first sub-matrix B-2), thereby obtaining the correct convolution operation result.

[0120] For example, in step S150, the first to eighth sub-operation results corresponding to each convolution step of the 8×p-step convolution operation of the 8 sub-weight matrices are accumulated to obtain the first operation result corresponding to each convolution step; in step S160, the p first operation results corresponding to each p-step convolution operation in the 8×p-step convolution operation are used as the second sub-matrix (C0, C1, ..., or C7) corresponding to at least part of the output data, and the second sub-matrix is ​​output. That is, the 8 sub-weight matrices, the 8 first sub-matrices (B0, B1, ..., B7), and the 4 filled first sub-matrices (B-1, B-2, B8, B9) are subjected to the 8×p-step convolution operation to obtain the 8 second sub-matrices (C0, C1, ..., C7).

[0121] For example, Figure 5 As shown, the first group of solid arrows on the left are the first sub-weight matrix (W00, W10, W20, W30) performing p-step convolution operations with the first sub-matrix (B-2, B-1, B0, B1) to obtain the 1st, 3rd, 5th, and 7th sub-operation results, and the first group of dotted arrows on the right are the second sub-weight matrix (W01, W11, W21, W31) performing p-step convolution operations with the first sub-matrix (B-2, B-1, B0, B1) to obtain the 2nd, 4th, 6th, and 8th sub-operation results; the 1st to 8th sub-operation results corresponding to each step of the convolution operation are accumulated to obtain the second sub-matrix C0 corresponding to the p first operation results, and the second sub-matrix C0 is output.

[0122] For example, Figure 5 As shown, the second group of solid arrows on the left are the first sub-weight matrix (W00, W10, W20, W30) performing p-step convolution operations with the first sub-matrix (B3, B4, B5, B6) to obtain the 1st, 3rd, 5th, and 7th sub-operation results, and the second group of dotted arrows on the right are the second sub-weight matrix (W01, W11, W21, W31) performing p-step convolution operations with the first sub-matrix (B3, B4, B5, B6) to obtain the 2nd, 4th, 6th, and 8th sub-operation results; the 1st to 8th sub-operation results corresponding to each step of the convolution operation are accumulated to obtain the second sub-matrix C5 corresponding to the p first operation results, and the second sub-matrix C5 is output.

[0123] For example, in Figure 5 In the example, an example of the pseudo code of the above convolution operation process is as follows:

[0124] for i in(0,1,2,3)

[0125] Cn+=Conv(Wi0,B[n-2+i],offset_y=0)

[0126] Cn+=Conv(Wi1,B[n-2+i],offset_y=3)

[0127] Where i is one of the digits of the first sub-weight matrix, n is the number of the second sub-matrix to be obtained (i.e. Cn); the Conv instruction is used to perform the convolution operation of the sub-weight matrix and the first sub-matrix. For example, Cn+=Conv(Wi0,B[n-2+i],offset_y=0) indicates the convolution operation of the first sub-weight matrix and the first sub-matrix, and Cn+=Conv(Wi1,B[n-2+i],offset_y=3) indicates the convolution operation of the second sub-weight matrix and the first sub-matrix. For example, in Figure 5 In the example, through the for loop of the above pseudo code example, the convolution operation of the 8 sub-weight matrices and the corresponding first sub-matrix can be completed to obtain the specified second sub-matrix Cn.

[0128] It should be noted that Figures 2 to 5 The specific processing methods such as the division method of the weight matrix and the first matrix, the specific number of sub-weight matrices and the first sub-matrix, and the convolution operation are only some examples. Other processing methods can also be selected according to actual needs. The embodiments of the present disclosure do not limit this.

[0129] The convolution operation method provided by at least one embodiment of the present disclosure addresses the problem that large-sized convolution kernels cannot be directly calculated on specific computing units due to the limitations of chip hardware characteristics. The large-sized convolution kernels are equivalently decomposed into multiple weight matrices, and the weight matrices can be directly accelerated on the computing units of the chip. For example, on a specific computing unit (e.g., a tensor core), the convolution operation of the weight matrix can be completed through one (or several) instructions. Compared with the large-sized convolution kernel that needs to be transferred to the memory before the convolution operation is completed, the convolution operation method provided by at least one embodiment of the present disclosure greatly improves the operation speed, and can adapt to the hardware characteristics of the chip, give full play to the chip computing power, and improve computing performance.

[0130] Figure 6 A schematic diagram of another example of a convolution operation method provided in at least one embodiment of the present disclosure. The convolution operation method is, for example, Figure 2 The convolution operation method shown.

[0131] For example, when performing a convolution operation on an arithmetic unit, the program used to execute the convolution operation on the arithmetic unit includes multiple workgroups, each of which includes multiple threads. Threads in the same workgroup can be grouped according to scheduling units, and then scheduled to the hardware for execution in groups. Each group of scheduling units is called a scheduling unit (cwarp).

[0132] For example, Figure 6 As shown, step S110 may further include: using the first scheduling unit (cwarp0) to load the weight matrix; using the second scheduling unit (cwarp1) to load M first sub-matrices (for example, B-2, B-1, B0, B1..., where the M first sub-matrices include the filled first sub-matrices B-2 and B-1) in sequence along the first direction, where M is a positive integer.

[0133] For example, when each sub-weight matrix is ​​used to perform a multi-step convolution operation on the first matrix in step S140, the multi-step convolution operation includes a p-step convolution operation, where p is a positive integer; Figure 6 As shown, step S140 may further include: based on the K-th offset, using the third scheduling unit (cwarp2) to perform a p-step convolution operation on the K-th sub-weight matrix and the M first sub-matrices to obtain p K-th sub-operation results corresponding to the p-step convolution operation.

[0134] For example, the operation unit further includes an accumulation module (not shown in the figure) integrated therein, which is used to perform the accumulation of the 1st to Nth sub-operation results corresponding to each convolution operation in step S150, and then use the accumulated multiple first operation results as the second sub-matrix corresponding to at least part of the output data and output the second sub-matrix (for example, C0, C1, C2...) in step S160.

[0135] For example, the specific process of using the first scheduling unit (cwarp0), the second scheduling unit (cwarp1), the third scheduling unit (cwarp2) and the accumulation module to perform steps S110 to S160 is detailed in Figures 2 to 5 The description in , will not be repeated here.

[0136] In the convolution operation method provided in the embodiment of the present disclosure, the weight matrix can be directly accelerated on the operation unit, so the accumulation module integrated in the operation unit can be directly used to accumulate multiple sub-operation results of each convolution operation without the need to complete the accumulation process in the memory, thereby avoiding the chip performance loss caused by frequent data transmission between the operation unit and the memory.

[0137] Figure 7 Another exemplary flowchart of a convolution operation method provided for at least one embodiment of the present disclosure. Figure 6 The convolution operation method shown is, for example, Figure 7 An example of the convolution operation method in

[15] .

[0138] For example, the input data is divided into multiple parts, each of which corresponds to a first sub-matrix. For example, after completing the convolution operation of the first matrix corresponding to at least part of the input data (for example, the first matrix is ​​divided into M first sub-matrices), the first sub-matrix (for example, the M+1th first sub-matrix) can be continuously loaded, and the convolution operation is performed in parallel during the loading process until all the input data are calculated. For example, Figure 7 As shown, the convolution operation method provided by at least one embodiment of the present disclosure may further include the following steps S210 to S220.

[0139] Step S210: using the second scheduling unit to load the M+1th first sub-matrix along the first direction;

[0140] Step S220: Based on the K-th offset, use the third scheduling unit to perform a q-step convolution operation on the K-th sub-weight matrix and the M+1-th first sub-matrix to obtain multiple K-th sub-operation results corresponding to the q-step convolution operations, where q is a positive integer.

[0141] For example, Figure 6 For example, the first matrix is ​​divided into M first sub-matrices, for example, only including B-2, B-1, ..., B3, and the weight matrix and the first sub-matrices B-2, B-1, ..., B3 are subjected to 3×p-step convolution operations to obtain three second sub-matrices (C0, C1, C2).

[0142] For example, in step S210, the second scheduling unit (cwarp1) can be used to continue loading the M+1th first sub-matrix (for example, the M+1th first sub-matrix is ​​B4) along the first direction; then, in step S220, based on the Kth offset, the third scheduling unit (cwarp2) is used to perform a q-step convolution operation on the Kth sub-weight matrix and the M+1th first sub-matrix through the same convolution operation method as the previous M first sub-matrices to obtain multiple Kth sub-operation results corresponding to the q-step convolution operations.

[0143] For example, Figure 6 As shown, you can follow Figure 7 The convolution operation method in continues to load the first sub-matrix B5, B6, ..., and performs convolution operation in parallel during the loading process to obtain and output the second sub-matrix C4, C5, ..., until all the input data are calculated, thereby obtaining complete output data.

[0144] For example, the specific process of executing the convolution operation method provided by the embodiment of the present disclosure is detailed in Figures 2 to 5 The description in , will not be repeated here.

[0145] In the convolution operation method provided in the embodiment of the present disclosure, the parallel process of input data block loading and convolution can be implemented on the operation unit, thereby achieving accelerated calculation and improving the overall computing performance of the chip.

[0146] Figure 8 An exemplary flowchart of a convolution processing method provided for at least one embodiment of the present disclosure.

[0147] For example, at least one embodiment of the present disclosure further provides a convolution processing method for implementing a convolution operation on all input data to obtain complete output data. Figure 8 As shown, the convolution processing method may include the following steps S310 to S340.

[0148] Step S310: Obtain input data;

[0149] Step S320: Divide the input data into multiple parts;

[0150] Step S330: performing a multi-step convolution operation on the first matrix corresponding to each portion of the input data using the convolution operation method provided by any embodiment of the present disclosure, obtaining at least a portion of output data corresponding to the multi-step convolution operation and outputting at least a portion of the output data;

[0151] Step S340: Obtain complete output data according to at least part of the output data corresponding to each part of the input data.

[0152] For example, in step S310, the input data obtained includes, for example Figure 2 At least part of the input data obtained in step S110; in step S320, the input data is divided into multiple parts, each part corresponds to a first matrix; in step S330, using, for example Figure 2 The convolution operation method in the embodiment performs a multi-step convolution operation on the first matrix corresponding to each part of the input data to obtain and output at least part of the output data corresponding to each part of the input data; in step S340, the complete output data is obtained based on the at least part of the output data corresponding to each part of the input data.

[0153] For example, the specific process of performing the convolution operation method provided by the embodiment of the present disclosure on each part of the input data is detailed in Figures 2 to 5 The description in , will not be repeated here.

[0154] The convolution operation method and convolution processing method provided in at least one embodiment of the present disclosure decompose the large-size convolution kernel into multiple weight matrices equivalently based on the storage capacity, computing power and other characteristics of the chip. The weight matrix can directly implement accelerated calculation on the chip, thereby adapting to the hardware characteristics of the chip, giving full play to the chip computing power, and improving computing performance.

[0155] Figure 9 A schematic block diagram of a convolution operation device provided for at least one embodiment of the present disclosure

[0156] For example, Figure 9 As shown, the convolution operation device 200 includes an acquisition module 210, a division module 220, a processing module 230, a calculation module 240, an accumulation module 250 and an output module 260.

[0157] For example, the acquisition module 210 is configured to acquire a first matrix corresponding to at least part of the input data and a weight matrix corresponding to the convolution kernel. For example, the first matrix includes a plurality of elements arranged in two dimensions. That is, the acquisition module 210 can be configured to perform, for example Figure 2 Step S110 is shown.

[0158] For example, the partitioning module 220 is configured to partition the weight matrix into N sub-weight matrices, where N is a positive integer. That is, the partitioning module 220 can be configured to perform, for example Figure 2 Step S120 is shown.

[0159] For example, the processing module 230 is configured to set a K-th offset along the first direction corresponding to the first matrix for the K-th sub-weight matrix among the N sub-weight matrices, where 1≤K≤N and K is an integer. That is, the processing module 230 can be configured to perform, for example Figure 2 Step S130 is shown.

[0160] For example, the calculation module 240 is configured to perform a multi-step convolution operation on the first matrix using the K-th sub-weight matrix based on the K-th offset to obtain a plurality of K-th sub-operation results corresponding to the multi-step convolution operation. That is, the calculation module 240 can be configured to perform, for example Figure 2 Step S140 is shown.

[0161] For example, the computing module 240 can be integrated inside a computing unit (e.g., a tensor core). The tensor core can complete the multi-step convolution operation in step S140 through one (or several) instructions. Compared with large-size convolution kernels that need to be transferred to the memory before completing the convolution operation, the convolution operation method provided by at least one embodiment of the present disclosure greatly improves the operation speed, and can adapt to the hardware characteristics of the chip, give full play to the chip computing power, and improve computing performance.

[0162] For example, the accumulation module 250 is configured to accumulate the first to Nth sub-operation results corresponding to each convolution operation in the multi-step convolution operation to obtain the first operation result corresponding to each convolution operation. That is, the accumulation module 250 can be configured to perform, for example Figure 2 Step S150 is shown.

[0163] For example, the accumulation module 250 is integrated inside the operation unit, and can directly complete the accumulation of the 1st sub-operation result to the Nth sub-operation result corresponding to each convolution operation in step S150 inside the operation unit without the need to complete the accumulation process in the memory, thereby avoiding the chip performance loss caused by frequent data transmission between the operation unit and the memory.

[0164] For example, the output module 260 is configured to use the multiple first operation results corresponding to the multi-step convolution operation as at least part of the output data and output the multiple first operation results. That is, the output module 260 can be configured to perform, for example Figure 2 Step S160 is shown.

[0165] Since in the above description Figure 2 In the process of the convolution operation method shown in FIG, the details of the operation of the convolution operation device 200 have been introduced, so for the sake of brevity, they will not be repeated here. For relevant details, please refer to the above Figures 1 to 8 Description.

[0166] It should be noted that Figure 9 The modules in the convolution operation device 200 can be configured as software, hardware, firmware, or any combination thereof to perform specific functions. For example, these modules can correspond to dedicated integrated circuits, pure software codes, or modules that combine software and hardware. Figure 9 The device described may be a PC computer, a tablet device, a personal digital assistant, a smartphone, a web application, or other device capable of executing program instructions, but is not limited thereto.

[0167] In addition, although the convolution operation device 200 is divided into modules for performing corresponding processing in the above description, it is clear to those skilled in the art that the processing performed by each module can also be performed without any specific module division in the device or without clear boundaries between the modules. Figure 9 The described convolution operation device 200 is not limited to including the modules described above, but some other modules (for example, a writing module, a control module, etc.) may be added as needed, or the above modules may also be combined.

[0168] At least one embodiment of the present disclosure also provides an electronic device, which includes a processor and a memory; the memory includes one or more computer program modules; the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules include a method for implementing the convolution operation method or convolution processing method provided by the embodiment of the present disclosure described above.

[0169] Figure 10 A schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure

[0170] For example, Figure 10 As shown, the electronic device 300 includes a processor 310 and a memory 320. For example, the memory 320 is used to store non-transitory computer-readable instructions (e.g., one or more computer program modules). The processor 310 is used to execute non-transitory computer-readable instructions. When the non-transitory computer-readable instructions are executed by the processor 310, one or more steps of the convolution operation method or convolution processing method described above can be executed. The memory 320 and the processor 310 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0171] For example, the processor 310 may be a central processing unit (CPU), a digital signal processor (DSP), or other processing units with data processing capabilities and / or program execution capabilities, such as a field programmable gate array (FPGA). For example, the central processing unit (CPU) may be an X86 or ARM architecture. The processor 310 may be a general-purpose processor or a dedicated processor, and may control other components in the electronic device 300 to perform desired functions.

[0172] For example, the memory 320 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), a hard disk, an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, a flash memory, etc. One or more computer program modules may be stored on the computer-readable storage medium, and the processor 310 may execute one or more computer program modules to implement various functions of the electronic device 300. Various applications and various data, as well as various data used and / or generated by the applications, may also be stored in the computer-readable storage medium.

[0173] It should be noted that, in the embodiments of the present disclosure, the specific functions and technical effects of the electronic device 300 can refer to the above description of the convolution operation method and convolution processing method provided in at least one embodiment of the present disclosure, and will not be repeated here.

[0174] Figure 11 A schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0175] For example, Figure 11 As shown, the electronic device 400 is suitable for implementing the convolution operation method or convolution processing method provided by the embodiment of the present disclosure. It should be noted that, Figure 11 The electronic device 400 shown is only an example and does not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0176] For example, Figure 11 As shown, electronic device 400 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 41, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 42 or programs loaded from storage device 48 into random access memory (RAM) 43. RAM 43 also stores various programs and data required for the operation of device 400 for cache system simulation. Processing device 41, ROM 42, and RAM 43 are connected to each other via bus 44. Input / output (I / O) interface 45 is also connected to bus 44. Generally, the following devices can be connected to I / O interface 45: input device 46 including, for example, a touch screen, touchpad, keyboard, mouse, camera, microphone, accelerometer, gyroscope, etc.; output device 47 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage device 48 including, for example, a magnetic tape, hard disk, etc.; and communication device 49. Communication device 49 can allow electronic device 400 to communicate with other electronic devices wirelessly or by wire to exchange data.

[0177] Although Figure 11 The electronic device 400 is shown with various devices, but it should be understood that it is not required to implement or possess all of the devices shown, and the electronic device 400 may alternatively implement or possess more or fewer devices.

[0178] For the detailed description and technical effects of the electronic device 400, please refer to the above description of the convolution operation method or the convolution processing method, which will not be repeated here.

[0179] Figure 12 A schematic diagram of a storage medium provided for at least one embodiment of the present disclosure.

[0180] For example, Figure 12As shown, the storage medium 500 stores non-transitory computer-readable instructions 510. For example, when the non-transitory computer-readable instructions 510 are executed by a computer, one or more steps in the convolution operation method or convolution processing method described above are performed.

[0181] For example, the storage medium 500 can be applied to Figure 10 For example, the storage medium 500 may be the memory 320 in the electronic device 300. For example, the description of the storage medium 500 may refer to Figure 10 The corresponding description of the memory 320 in the electronic device 300 is not repeated here.

[0182] Regarding this disclosure, the following points need to be explained:

[0183] (1) The drawings of the embodiments of the present disclosure only relate to the structures related to the embodiments of the present disclosure, and other structures may refer to conventional designs.

[0184] (2) Unless there is any conflict, the features of the same embodiment and different embodiments of the present disclosure may be combined with each other.

[0185] The above are only specific embodiments of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this disclosure should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be based on the scope of protection of the claims.

Claims

1. A convolution operation method, comprising: Obtaining a first matrix corresponding to at least part of the input data and a weight matrix corresponding to the convolution kernel, wherein the first matrix includes a plurality of elements arranged in two dimensions; Dividing the weight matrix into N sub-weight matrices, where N is a positive integer; Setting a K-th offset along a first direction corresponding to the first matrix for a K-th sub-weight matrix among the N sub-weight matrices, where 1≤K≤N and K is an integer; Based on the K-th offset, use the K-th sub-weight matrix to perform a multi-step convolution operation on the first matrix to obtain a plurality of K-th sub-operation results corresponding to the multi-step convolution operations respectively; Accumulating the first to Nth sub-operation results corresponding to each convolution operation in the multi-step convolution operation to obtain a first operation result corresponding to each convolution operation; The plurality of first operation results corresponding to the multi-step convolution operation are used as at least part of the output data and the plurality of first operation results are output.

2. The convolution operation method according to claim 1, further comprising: The convolution kernel is padded to obtain the weight matrix.

3. The convolution operation method according to claim 1, further comprising: At least part of the input data is padded to obtain the first matrix.

4. The convolution operation method according to claim 1, wherein: The weight matrix includes a one-dimensional weight vector.

5. The convolution operation method according to claim 4, wherein: Each of the N sub-weight matrices includes at least one element. The N sub-weight matrices include at least one first sub-weight matrix and at least one second sub-weight matrix, and the number of elements contained in each first sub-weight matrix in the at least one first sub-weight matrix is ​​the same as or different from the number of elements contained in each second sub-weight matrix in the at least one second sub-weight matrix.

6. The convolution operation method according to claim 5, wherein: The number of elements contained in each first sub-weight matrix is ​​3, and the number of elements contained in each second sub-weight matrix is ​​5.

7. The convolution operation method according to claim 5, wherein: The Kth offset corresponding to the Kth sub-weight matrix when it is the first sub-weight matrix is ​​different from the Kth offset corresponding to the Kth sub-weight matrix when it is the second sub-weight matrix.

8. The convolution operation method according to claim 7, wherein: In response to the K-th sub-weight matrix being the first sub-weight matrix, the K-th offset is 0; In response to the K-th sub-weight matrix being the second sub-weight matrix, the K-th offset is equal to the number of elements included in each of the first sub-weight matrices.

9. The convolution operation method according to claim 1, further comprising: Divide the first matrix into M first sub-matrices, Each of the M first sub-matrices includes a plurality of elements arranged in r rows and s columns, M is a positive integer, r is a positive integer, and s is a positive integer.

10. The convolution operation method according to claim 9, wherein: The multi-step convolution operation includes a p-step convolution operation, where p is a positive integer. The performing a multi-step convolution operation on the first matrix using the K-th sub-weight matrix based on the K-th offset to obtain a plurality of K-th sub-operation results respectively corresponding to the multi-step convolution operations includes: Based on the K-th offset, the K-th sub-weight matrix is ​​used to perform p-step convolution operations with the M first sub-matrices respectively to obtain p K-th sub-operation results corresponding to the p-step convolution operations.

11. The convolution operation method according to claim 9, wherein: The multi-step convolution operation includes a p-step convolution operation, where p is a positive integer. The step of using the plurality of first operation results corresponding to the multi-step convolution operation as at least part of the output data and outputting the plurality of first operation results includes: The p first operation results corresponding to the p-step convolution operation are used as a second sub-matrix corresponding to the at least part of the output data, and the second sub-matrix is ​​output.

12. The convolution operation method according to claim 9, wherein: The obtaining of a first matrix corresponding to at least part of the input data and a weight matrix corresponding to the convolution kernel includes: Loading the weight matrix using a first scheduling unit; The M first sub-matrices are loaded sequentially along the first direction using a second scheduling unit.

13. The convolution operation method according to claim 12, wherein: The multi-step convolution operation includes a p-step convolution operation, where p is a positive integer. The performing a multi-step convolution operation on the first matrix using the K-th sub-weight matrix based on the K-th offset to obtain a plurality of K-th sub-operation results respectively corresponding to the multi-step convolution operations includes: Based on the K-th offset, a third scheduling unit is used to perform a p-step convolution operation on the K-th sub-weight matrix and the M first sub-matrices to obtain p K-th sub-operation results corresponding to the p-step convolution operation.

14. The convolution operation method according to claim 13, wherein: The input data is divided into a plurality of parts, each of the plurality of parts corresponds to a first sub-matrix, The method further comprises: Loading an M+1th first submatrix along the first direction using the second scheduling unit, wherein the M+1th first submatrix includes a plurality of elements arranged in r rows and s columns, where r is a positive integer and s is a positive integer; Based on the K-th offset, the third scheduling unit is used to perform a q-step convolution operation on the K-th sub-weight matrix and the M+1-th first sub-matrix to obtain multiple K-th sub-operation results corresponding to the q-step convolution operations, where q is a positive integer.

15. The convolution operation method according to claim 1, wherein: The first direction is the column direction of the first matrix.

16. The convolution operation method according to claim 1, wherein: The step size of the multi-step convolution operation is 1.

17. A convolution processing method, comprising: Get input data; dividing the input data into a plurality of parts, wherein each of the plurality of parts corresponds to a first matrix; performing the multi-step convolution operation on the first matrix corresponding to each portion of the input data using the convolution operation method according to any one of claims 1 to 16, obtaining the at least portion of output data corresponding to the multi-step convolution operation, and outputting the at least portion of the output data; Complete output data is obtained according to the at least part of the output data corresponding to each part of the input data.

18. A convolution operation device, comprising: an acquisition module configured to acquire a first matrix corresponding to at least a portion of the input data and a weight matrix corresponding to the convolution kernel, wherein the first matrix includes a plurality of elements arranged in two dimensions; a partitioning module configured to partition the weight matrix into N sub-weight matrices, where N is a positive integer; a processing module configured to set a Kth offset along a first direction corresponding to the first matrix for a Kth sub-weight matrix among the N sub-weight matrices, where 1≤K≤N and K is an integer; a calculation module configured to perform a multi-step convolution operation on the first matrix using the K-th sub-weight matrix based on the K-th offset, so as to obtain a plurality of K-th sub-operation results respectively corresponding to the multi-step convolution operations; an accumulation module configured to accumulate the first to Nth sub-operation results corresponding to each convolution operation in the multi-step convolution operation to obtain a first operation result corresponding to each convolution operation; The output module is configured to use the multiple first operation results corresponding to the multi-step convolution operation as at least part of the output data and output the multiple first operation results.

19. An electronic device comprising: processor; a memory comprising one or more computer program modules; Wherein, the one or more computer program modules are stored in the memory and configured to be executed by the processor, and the one or more computer program modules are used to implement the convolution operation method described in any one of claims 1-16 or the convolution processing method described in claim 17.

20. A storage medium storing non-transitory computer-readable instructions, which, when executed by a computer, implement the convolution operation method described in any one of claims 1 to 16 or the convolution processing method described in claim 17.

Images