Method, computing device and storage medium for implementing convolution operation

Abstract

The present disclosure provides a method, a computing device, and a computer-readable storage medium for implementing a convolution operation. The method includes, in a 3D convolver: receiving a weight input and an activation input of a low-dimensional convolution operation, wherein the weight input includes a filter value and the activation input includes a pixel value; based on a predetermined filter depth of the 3D convolver, converting the filter value of the weight input into a first filter value and a second filter value of the 3D convolver; based on a predetermined pixel data block size of the 3D convolver, converting the pixel value of the activation input into a first pixel value and a second pixel value of the 3D convolver; and performing a 3D convolution operation based on the first filter value and the second filter value of the 3D convolver and the first pixel value and the second pixel value of the 3D convolver.

Description

Technical Field

[0001] The present disclosure generally relates to the field of processors, and more specifically, to a computing device, method, and computer-readable storage medium for implementing a convolution operation. Background Art

[0002] Convolutional neural networks are a special type of artificial neural network that uses a mathematical operation called convolution instead of general matrix multiplication in at least one of its layers. General Purpose Graphics Processing Unit (GPGPU) needs to perform a large number of convolution operations when training and reasoning models. Convolution operations can be convolution operations of multiple dimensions, such as one-dimensional (1D), two-dimensional (2D) and three-dimensional (3D) convolution operations, which are widely used in speech, image and video processing. Therefore, there is a need to use a processor, such as GPGPU, to simultaneously implement convolution operations of multiple dimensions.

[0003] Currently, one way to solve the problem of coexistence of convolution operations in multiple dimensions is to design a corresponding hardware convolution device for each convolution operation in the processor. For example, a GPGPU can be designed with 1D, 2D and 3D convolution devices to implement 1D, 2D and 3D convolution operations respectively. However, this approach makes the processor design complicated and greatly increases the hardware resource overhead.

[0004] Another way to solve the problem of coexistence of convolution operations in multiple dimensions is to use software to reshape the input and output data of 1D or 2D convolution operations to meet the input and output requirements of 3D convolution operations, so that 3D convolution devices can be used for processing. However, this method requires the input and output data to be reshaped by software on the host side, which has high software complexity, and the amount of data after reshaping increases, which will occupy more storage resources. Summary of the invention

[0005] In response to the above problems, the present disclosure provides a method for using a 3D convolution device to implement low-dimensional convolution operations (such as 1D convolution operations or 2D convolution operations), which can use a set of hardware circuits to simultaneously implement 1D, 2D and 3D convolution operations without increasing hardware resources and software complexity.

[0006] According to one aspect of the present disclosure, a method for implementing a convolution operation is provided. The method includes, in a 3D convolver: receiving a weight input and an activation input of a low-dimensional convolution operation, wherein the weight input includes a filter value and the activation input includes a pixel value; based on a predetermined filter depth of the 3D convolver, converting the filter value of the weight input into a first filter value and a second filter value of the 3D convolver; based on a predetermined pixel data block size of the 3D convolver, converting the pixel value of the activation input into a first pixel value and a second pixel value of the 3D convolver; and performing a 3D convolution operation based on the first filter value and the second filter value of the 3D convolver and the first pixel value and the second pixel value of the 3D convolver.

[0007] In some implementations, converting the filter value of the weight input into the first filter value and the second filter value of the 3D convolution device includes: determining whether the filter value of the weight input is an integer multiple of the predetermined filter depth; in response to determining that the filter value of the weight input is an integer multiple of the predetermined filter depth, taking the quotient of the filter value of the weight input and the predetermined filter depth as the second filter value, and taking the predetermined filter depth as the first filter value; in response to determining that the filter value of the weight input is not an integer multiple of the predetermined filter depth, splitting the low-dimensional convolution operation into a first convolution operation and a second convolution operation to respectively determine the first filter value and the second filter value of the first convolution operation and the second convolution operation.

[0008] In some implementations, splitting the low-dimensional convolution operation into a first convolution operation and a second convolution operation includes: rounding down the quotient of the filter value of the weight input and the predetermined filter depth as the second filter value of the first convolution operation, using the predetermined filter depth as the first filter value of the first convolution operation, setting the second filter value of the second convolution operation to 1, and using the remainder of the filter value of the weight input and the predetermined filter depth as the first filter value of the second convolution operation.

[0009] In some implementations, converting the filter value of the weight input into the first filter value and the second filter value of the 3D convolver also includes: determining whether the filter value of the weight input is greater than the maximum processing filter value of the 3D convolver; in response to determining that the filter value of the weight input is greater than the maximum processing filter value of the 3D convolver, splitting the low-dimensional convolution operation into multiple convolution operations, wherein the filter value of one convolution operation among the multiple convolution operations is less than the maximum processing filter value of the 3D convolver, and the filter values ​​of other convolution operations among the multiple convolution operations are equal to the maximum processing filter value of the 3D convolver.

[0010] In some implementations, converting the pixel value of the activation input into a first pixel value and a second pixel value of the 3D convolver includes: determining a predetermined pixel depth of the 3D convolver as the second pixel value of the 3D convolver based on the predetermined pixel data block size; and determining the first pixel value of the 3D convolver based on the pixel value of the activation input and the predetermined pixel depth of the 3D convolver.

[0011] In some implementations, the method further includes: receiving a padding value for the low-dimensional convolution operation; and converting the padding value of the low-dimensional convolution operation into a first padding value of a first pixel dimension and a second padding value of a second pixel dimension of the 3D convolver.

[0012] In some implementations, converting the padding value of the low-dimensional convolution operation into a first padding value of a first pixel dimension and a second padding value of a second pixel dimension of the 3D convolver includes: when the padding value of the low-dimensional convolution operation is positive, determining the second padding value as a positive value based on a high-order bit of the padding value; when the padding value of the low-dimensional convolution operation is negative, determining the second padding value as a negative value based on a high-order bit of the padding value; and determining the first padding value based on the padding value, the second padding value, and a predetermined pixel data block size of the 3D convolver.

[0013] In some implementations, the first filter dimension and the second filter dimension of the 3D convolver are perpendicular to each other.

[0014] In some implementations, the first pixel dimension and the second pixel dimension of the 3D convolver are perpendicular to each other.

[0015] In some implementations, the low-dimensional convolution operation includes a 1D convolution operation, and the weight input further includes channel values.

[0016] In some implementations, the low-dimensional convolution operation includes a 2D convolution operation, the weight input further includes an input channel value and an output channel value, and the activation input further includes an input channel value.

[0017] According to yet another aspect of the present disclosure, a computer-readable storage medium is provided, on which a computer program code is stored. When the computer program code is executed, the method described above is executed. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The present disclosure will be better understood and other objects, details, features and advantages of the present disclosure will become more apparent through the description of specific embodiments of the present disclosure given with reference to the following drawings.

[0019] Figures 1A to 1CSchematic diagrams of weight input and activation input for 1D convolution operation, 2D convolution operation, and 3D convolution operation are shown respectively.

[0020] Figure 2A A schematic diagram of a computing device 10 for implementing a convolution operation according to the related art is shown.

[0021] Figure 2B A schematic diagram of another computing device 10 ′ for implementing a convolution operation according to the related art is shown.

[0022] Figure 3 A schematic diagram of a computing device 20 for implementing a convolution operation according to an embodiment of the present invention is shown.

[0023] Figure 4 An exemplary flow chart of a method 300 for implementing a convolution operation according to an embodiment of the present invention is shown.

[0024] Figure 5 A further detailed flow chart of a process for converting filter values ​​for weight inputs (block 320) according to some embodiments of the present invention is shown.

[0025] Figure 6 A schematic diagram of converting an activation input pixel value into a first pixel value and a second pixel value of a 3D convolver according to an embodiment of the present invention is shown.

[0026] Fig. 7A and 7B Schematic diagrams of 2D convolution operation and 3D convolution operation according to embodiments of the present invention are respectively shown. DETAILED DESCRIPTION

[0027] The preferred embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the preferred embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to make the present disclosure more thorough and complete, and to fully convey the scope of the present disclosure to those skilled in the art.

[0028] As used herein, the term "including" and its variations mean open inclusion, i.e., "including but not limited to". Unless otherwise stated, the term "or" means "and / or". The term "based on" means "based at least in part on". The terms "one embodiment" and "some embodiments" mean "at least one example embodiment". The term "another embodiment" means "at least one additional embodiment". The terms "first", "second", etc. may refer to different or the same objects.

[0029] Depending on different application requirements, the processor may need to perform convolution operations of different dimensions, such as 1D, 2D, 3D, etc. The input of each convolution operation includes a weight input and an activation input. Figures 1A to 1C Schematic diagrams of weight input and activation input for 1D convolution operation, 2D convolution operation, and 3D convolution operation are shown respectively.

[0030] like Figure 1A As shown, the input of the 1D convolution operation is a 2D weight input and a 1D activation input, where the two dimensions of the 2D weight input are the channel dimension and the filter dimension, and the dimension of the 1D activation input is the pixel dimension.

[0031] like Figure 1B As shown in the figure, the input of the 2D convolution operation is a 3D weight input and a 2D activation input, where the three dimensions of the 3D weight input are the output channel (output channel, abbreviated as och) dimension, the input channel (input channel, abbreviated as ich) dimension and the filter (filter) dimension, and the two dimensions of the 2D activation input are the input channel (input channel, abbreviated as ich) dimension and the pixel (pixel) dimension.

[0032] like Figure 1C As shown, the input of the 3D convolution operation is a 4D weight input and a 3D activation input, where the 4 dimensions of the 4D weight input are the output channel (output channel, abbreviated as och) dimension, the input channel (input channel, abbreviated as ich) dimension, the first filter (filter_x) dimension, and the second filter (filter_y) dimension, and the 3D activation input has three dimensions: the input channel (input channel, abbreviated as ich) dimension, the first pixel (pixel_x) dimension, and the second pixel (pixel_y) dimension. Note that Figure 1C The filter_x*filter_y in the expression is intended to represent two mutually perpendicular dimensions, filter_x and filter_y.

[0033] Figure 2A FIG. 1 is a schematic diagram of a computing device 10 for implementing a convolution operation according to the related art. Figure 2AAs shown in , the computing device 10 may include three hardware convolution devices 100, such as 1D convolution device 100-1, 2D convolution device 100-2 and 3D convolution device 100-3, to perform 1D convolution operation, 2D convolution operation and 3D convolution operation respectively. Each convolution device 100 has two inputs, namely a corresponding weight input and an activation input, and the convolution device 100 performs a convolution operation on the two inputs to generate an activation output. In a multi-layer convolutional neural network, the activation output can be used as the activation input of the next convolution layer (not shown in the figure).

[0034] In such Figure 2A In the computing device 10 shown, three independent hardware convolvers need to be designed to implement 1D, 2D and 3D convolution operations respectively, which makes the processor design complex and increases the hardware overhead.

[0035] Figure 2B FIG. 1 is a schematic diagram of another computing device 10′ for implementing a convolution operation according to the related art. Figure 2B As shown in , the computing device 10 ′ may include only one 3D convolver 100 ′.

[0036] For 3D convolution operations, the 3D convolver 100' can directly perform convolution operations on its 4D weight input and 3D activation input, while for 1D and 2D convolution operations, it is necessary to reshape the input data through software at another device (such as a host) 20' outside the computing device 10' to make it suitable for the 3D convolver 100', and reshape the output data of the 3D convolver 100' to meet the output requirements of the 1D and 2D convolution operations.

[0037] In such a computing device 10', since the input and output data need to be reshaped by software on the host side, the software complexity is relatively high, and the amount of data after the reshaping increases, which will occupy more storage resources.

[0038] In response to the above problems, the present disclosure provides a method for using a 3D convolution device to implement low-dimensional convolution operations (such as 1D convolution operations or 2D convolution operations), which can use a set of hardware circuits to simultaneously implement 1D, 2D and 3D convolution operations without increasing hardware resources and software complexity.

[0039] Figure 3 FIG. 2 shows a schematic diagram of a computing device 20 for implementing a convolution operation according to an embodiment of the present invention. Figure 3 As shown in FIG. , the computing device 20 includes a 3D convolver 200. In addition to the 3D convolution operation, the 3D convolver 200 can also implement a low-dimensional convolution operation (ie, a 1D convolution operation and a 2D convolution operation).

[0040] As mentioned above Figures 1A to 1CAs shown, for 1D convolution operations and 2D convolution operations, if the filter value of the filter dimension of its weight input can be converted into the first filter value of the first filter dimension and the second filter value of the second filter dimension of the weight input of the 3D convolution operation, and the pixel value of the pixel dimension of its activation input can be converted into the first pixel value of the first pixel dimension and the second pixel value of the second pixel dimension of the 3D convolution operation, then it is possible to use a 3D convolution device to implement 1D convolution operations and 2D convolution operations.

[0041] In addition, according to the principle of convolution operation, the size of the activation output generated after the weight input and activation input are convolved will be smaller than the activation input, and the pixels at the corners and edges of the activation input are used very few times during the convolution process, while pixels in other places are reused many times, which will lose a lot of information on the boundaries. In particular, in a convolutional neural network containing multiple convolutional layers, the final activation output will be greatly reduced, or even completely unable to obtain useful information.

[0042] To solve the above problem, the activation input can be padded before the convolution operation so that the activation output generated by the convolution operation of the padded activation input and the weight input is the same size as the activation input. That is, the input and output matrices before and after the convolution operation are the same size.

[0043] Figure 4 An exemplary flow chart of a method 300 for implementing a convolution operation according to an embodiment of the present invention is shown. Here, the method 300 may be performed by the computing device 20, more specifically, by the 3D convolver 200 of the computing device 20. The 3D convolver 200 may conventionally perform a 3D convolution operation, which will not be described in detail herein. The method 300 focuses on implementing a low-dimensional convolution operation (such as a 1D convolution operation and a 2D convolution operation) using the 3D convolver 200. More specifically, Figure 4 The 2D convolution operation is described as an example.

[0044] like Figure 4 As shown in , in block 310, the 3D convolver 200 may receive a weight input and an activation input of a low-dimensional convolution operation, wherein the weight input includes a filter value filter, and the activation input includes a pixel value pixel. For example, when the low-dimensional convolution operation is a 2D convolution operation, its input is a 3D weight input (och(2D), ich(2D), filter(2D)) and a 2D activation input (ich(2D), pixel(2D)), as shown in FIG. Figure 1B shown.

[0045] In block 320 , the 3D convolver 200 may convert the filter value filter(2D) of the weight input into a first filter value filter_x(3D) and a second filter value filter_y(3D) of the 3D convolver 200 based on a predetermined filter depth filter_d of the 3D convolver 200 .

[0046] Here, the predetermined filter depth filter_d of the 3D convolver 200 is determined by the hardware of the 3D convolver 200, and is the maximum number of rows / columns of data processed at one time by the 3D convolver 200, that is, the maximum value of filter_x(3D) or filter_y(3D). In this article, it is assumed that the predetermined filter depth filter_d is the maximum value of the first filter value filter_x(3D).

[0047] Figure 5 A further detailed flow chart of a process for converting filter values ​​for weight inputs (block 320) according to some embodiments of the present invention is shown.

[0048] like Figure 5 As shown in FIG. 3 , in block 322 , the 3D convolver 200 may determine whether the filter value filter(2D) of the weight input is an integer multiple of the predetermined filter depth filter_d.

[0049] If it is determined that the filter value filter(2D) of the weight input is an integer multiple of the predetermined filter depth filter_d, in block 324, the 3D convolver 200 may use the quotient of the filter value filter(2D) of the weight input and the predetermined filter depth filter_d as the second filter value filter_y(3D), and use the predetermined filter depth filter_d as the first filter value filter_x(3D). That is, filter_y(3D)=filter(2D) / filter_d, filter_x(3D)=filter_d.

[0050] If it is determined that the filter value filter(2D) of the weight input is not an integer multiple of the predetermined filter depth filter_d, then in box 326, the 3D convolver 200 can split the low-dimensional convolution operation into two 3D convolution operations, referred to as a first convolution operation and a second convolution operation.

[0051] For the first convolution operation, the quotient of the filter value filter(2D) of the weight input and the predetermined filter depth filter_d can be rounded down as the second filter value filter_y(3D) of the first convolution operation, and the predetermined filter depth filter_d can be used as the first filter value filter_x(3D) of the first convolution operation. That is: filter_x(3D)=filter_d, where Indicates a floor operation.

[0052] For the second convolution operation, the second filter value filter_y(3D) of the second convolution operation may be set to 1, and the remainder of the filter value filter(2D) of the weight input and the predetermined filter depth filter_d is used as the first filter value filter_x(3D) of the second convolution operation. That is, filter_y(3D)=1, filter_x(3D)=filter(2D)%filter_d, where % represents a remainder operation.

[0053] That is, in this case, the 3D convolver 200 can split an instruction of a 2D convolution operation into two instructions of a 3D convolution operation to complete a 2D convolution operation whose filter value exceeds its filter depth.

[0054] In addition, before block 320 (or block 322), the 3D convolver 200 may further determine whether the filter value filter(2D) of the weight input is greater than the maximum processing filter value filter_max of the 3D convolver 200. Here, the maximum processing filter value filter_max of the 3D convolver 200 is determined by the maximum value of the first filter value filter_x(3D) and the second filter value filter_y(3D) supported by the hardware of the 3D convolver 200. For example, filter_max=filter_x_max*filter_y_max, where filter_x_max is the maximum value of the first filter value filter_x(3D) of the 3D convolver 200, and filter_y_max is the maximum value of the second filter value filter_y(3D) of the 3D convolver 200. For example, when the maximum value of the first filter value filter_x(3D) (ie, the predetermined filter depth filter_d) and the maximum value of the second filter value filter_y(3D) of the 3D convolver 200 are both 8, the maximum processing filter value filter_max of the 3D convolver 200 may be 64.

[0055] If it is determined that the filter value filter(2D) of the weight input is greater than the maximum processing filter value filter_max of the 3D convolver 200, the 3D convolver 200 may split the low-dimensional convolution operation (such as the 2D convolution operation) into multiple 3D convolution operations. The filter value of one of the multiple 3D convolution operations is less than the maximum processing filter value filter_max of the 3D convolver 200, and the filter values ​​of the other 3D convolution operations are equal to the maximum processing filter value filter_max of the 3D convolver 200.

[0056] That is, assuming filter(2D) = n*filter_max + filter(2D)%filter_max, the 2D convolution operation can be split into n convolution operations with a filter value of filter_max and one convolution operation with a filter value of filter(2D)%filter_max. Here, % represents the remainder operation. For each of the split convolution operations, you can still follow the above Figure 5 For example, in the case where the maximum processing filter value filter_max of the 3D convolver 200 is 64 as described above, if the filter value of the input 2D convolution operation is 100, the 2D convolution operation can be split into one convolution operation with a filter value of 64 and one convolution operation with a filter value of 36. For the split convolution operation with a filter value of 64 and a convolution operation with a filter value of 36, the above method can be continued. Figure 5 The method described is used for conversion or further splitting.

[0057] continue Figure 4 In block 330 , the 3D convolver 200 may convert the pixel value pixel(2D) of the activation input into a first pixel value pixel_x(3D) and a second pixel value pixel_y(3D) of the 3D convolver 200 based on a predetermined pixel data block size pixel_t of the 3D convolver 200 .

[0058] Here, the size of the data block for performing the convolution operation each time on the first pixel dimension (pixel_x) and the second pixel dimension (pixel_y) can be pre-defined for the 3D convolver 200 as a predetermined pixel data block size pixel_t, and the pixel size on the first pixel dimension or the second pixel dimension can be determined as a predetermined pixel depth pixel_d of the 3D convolver 200.

[0059] Specifically, the predetermined pixel depth pixel_d of the 3D convolver 200 may be determined based on the predetermined pixel data block size pixel_t as the second pixel value pixel_y(3D) of the 3D convolver 200. For example, assuming that the predetermined pixel data block size pixel_t of the 3D convolver 200 is 4 pixels*8 pixels, the predetermined pixel depth pixel_d of the 3D convolver 200 may be determined to be 8 pixels.

[0060] Then, the first pixel value pixel_x(3D) of the 3D convolver 200 may be determined based on the activated input pixel value pixel(2D) and a predetermined pixel depth of the 3D convolver 200. For example, pixel_x(3D)=pixel(2D) / pixel_d.

[0061] In this case, if the pixel value pixel(2D) of the 2D convolution operation is an integer multiple of the predetermined pixel data block size pixel_t of the 3D convolver 200, the 2D convolution operation can be divided into multiple 3D convolution operations, each of which performs convolution operations on the first pixel dimension and the second pixel dimension with the predetermined pixel data block size pixel_t. On the other hand, if the pixel value pixel(2D) of the 2D convolution operation is not an integer multiple of the predetermined pixel data block size pixel_t of the 3D convolver 200, the remainder needs to be converted into another separate convolution operation. The computational complexity of this case is relatively high, so the pixel value pixel(2D) of the 2D convolution operation can be designed as an integer multiple of the predetermined pixel data block size pixel_t of the 3D convolver 200 in the design stage. Alternatively, pixel(2D) can be padded with zeros to obtain an integer pixel_x(3D). More preferably, the pixel value pixel(2D) of the 2D convolution operation can be designed to be equal to the predetermined pixel data block size pixel_t of the 3D convolver 200.

[0062] Figure 6 FIG. 2 shows a schematic diagram of converting the pixel value of the activation input into the first pixel value and the second pixel value of the 3D convolution according to an embodiment of the present invention. Figure 6 As shown in, assuming that the pixel value of the activation input of the 2D convolution operation is a vector [p0, p1, p2, ..., p31], and the predetermined pixel data block size pixel_t of the 3D convolver 200 is 4 pixels * 8 pixels, then the predetermined pixel depth pixel_d of the 3D convolver 200 can be determined as 8 pixels, 8 pixels in a row (second pixel dimension), and a total of 4 rows (first pixel dimension), namely [p0, p1, ..., p7], [p8, p9, ..., p15], [p16, p17, ..., p23] and [p24, p25, ..., p31].

[0063] The activation inputs to the next 2D convolution operation are arranged sequentially along the first pixel dimension.

[0064] In this way, the pixel value pixel(2D) of the 2D convolution operation is converted to a pixel dimension suitable for the 3D convolution operation.

[0065] After the filter value filter(2D) of the weight input of the 2D convolution operation is converted into a first filter value filter_x(3D) and a second filter value filter_y(3D) suitable for the 3D convolver in box 320, and the pixel value pixel(2D) of the activation input of the 2D convolution operation is converted into a first pixel value pixel_x(3D) and a second pixel value pixel_y(3D) suitable for the 3D convolver in box 330, in box 340, the 3D convolver can perform a 3D convolution operation based on these first filter value filter_x(3D), second filter value filter_y(3D), first pixel value pixel_x(3D), and second pixel value pixel_y(3D). That is, at this time, the 3D convolution unit performs a 3D convolution operation with a 4D weight input (och(2D), ich(2D), pixel_x(3D), filter_y(3D)) and a 3D activation input (ich(2D), pixel_x(3D), pixel_y(3D)).

[0066] As mentioned earlier, it is often necessary to pad the activation input before a convolution operation, so when converting the activation data of a 2D matrix operation to make it suitable for a 3D convolver, the padding values ​​also need to be converted.

[0067] Specifically, the 3D convolver 200 can receive a padding value pad(2D) for the low-dimensional convolution operation (such as a 2D convolution operation), and then can convert the padding value pad(2D) into a first padding value pad_x(3D) of the first pixel dimension pixel_x of the 3D convolver 200 and a second padding value pad_y(3D) of the second pixel dimension pixel_y.

[0068] The padding value pad(2D) can be a value configured by software. For a 2D convolution operation, when padding its activation input, it is only necessary to add (filter(2D)-1) pixels to the pixel dimension of the activation input according to the padding value pad(2D) and the filter value filter(2D) of the weight input.

[0069] For the 3D convolution operation, its pixel dimension is a plane (i.e., including a first pixel dimension pixel_x and a second pixel dimension pixel_y that are perpendicular to each other), so it is necessary to determine an appropriate first filling value pad_x(3D) and a second filling value pad_y(3D) in these two dimensions respectively.

[0070] In one embodiment, if the predetermined pixel data block size pixel_t is 4 pixels*8 pixels, the padding value pad(2D) of the 2D convolution operation is a value in the range of -63 to +63. In this case, the padding value pad(2D) of the 2D convolution operation configured by the software can be represented by 8 bits, as shown below:

[0071] sign 0 H[2] H[1] H[0] L[2] L[1] L[0]

[0072] Among them, the first bit sign is the sign bit. When it is 0, it indicates that the padding value pad(2D) is positive (filling to the left), and when it is negative, it indicates that the padding value pad(2D) is negative (filling to the right); H[2], H[1] and H[0] represent the high-order bits of the padding value pad(2D), and L[2], L[1] and L[0] represent the low-order bits of the padding value pad(2D).

[0073] When the padding value pad(2D) is positive (for example, the sign bit sign is 0), the 3D convolver 200 may determine the second padding value pad_y(3D) as a positive value based on the high-order bits H[2], H[1], and H[0] of the padding value pad(2D). Further, the 3D convolver 200 may also determine the second padding value pad_y(3D) based on the size of the low-order bits L[2], L[1], and L[0] of the padding value pad(2D). For example, when the low-order bits L[2], L[1], and L[0] are greater than or equal to "100", the second padding value pad_y(3D) is determined as the high-order bits H[2], H[1], and H[0] plus 1, that is, pad_y(3D)={0,H[2:0]+1}; and when the low-order bits L[2], L[1], and L[0] are less than "100", the second padding value pad_y(3D) is determined as the high-order bits H[2], H[1], and H[0], that is, pad_y(3D)={0,H[2:0]}. Here, the first bit 0 of the second padding value pad_y(3D) indicates that it is a positive value.

[0074] On the other hand, when the padding value pad(2D) is negative (for example, the sign bit sign is 1), the 3D convolver 200 may determine the second padding value pad_y(3D) as a negative value based on the high-order bits H[2], H[1], and H[0] of the padding value pad(2D). Further, the 3D convolver 200 may also determine the second padding value pad_y(3D) based on the size of the low-order bits L[2], L[1], and L[0] of the padding value pad(2D). For example, when the low-order bits L[2], L[1], and L[0] are greater than "100", the second padding value pad_y(3D) is determined as the high-order bits H[2], H[1], and H[0] plus 1, that is, pad_y(3D) = {1, H[2:0] + 1}; and when the low-order bits L[2], L[1], and L[0] are less than or equal to "100", the second padding value pad_y(3D) is determined as the high-order bits H[2], H[1], and H[0], that is, pad_y(3D) = {1, H[2:0]}. Here, the first bit 1 of the second padding value pad_y(3D) indicates that it is a negative value.

[0075] Then, the first padding value pad_x(3D) may be determined based on the padding value pad(2D), the second padding value pad_y(3D), and the predetermined pixel data block size pixel_t (more specifically, based on the predetermined pixel depth pixel_d) of the 3D convolver 200. For example, pad_x(3D)=pad(2D)-pixel_d*pad_y(3D).

[0076] In this way, not only can the filter values ​​of the weight input and the pixel values ​​of the activation input of the 2D convolution operation be converted into the first filter value and the second filter value and the first pixel value and the second pixel value respectively required for the 3D convolution unit to perform the 3D convolution operation, but also the padding value of the 2D convolution operation can be converted into the first padding value and the second padding value required for the 3D convolution unit to perform the 3D convolution operation, so that the 2D convolution operation can be implemented using a separate hardware 3D convolution unit.

[0077] For 1D convolution operation, as above combined Figure 1A and Figure 1B As shown, the 2D weight input of the 1D convolution operation is equivalent to the 3D weight input with the output channel (och) of the 2D convolution operation being 1, and the 1D activation input of the 1D convolution operation is equivalent to the 2D activation input with the input channel (ich) of the 2D convolution operation being 1. Therefore, the 1D convolution operation can be implemented in the same way as the 2D convolution operation described above, and will not be described in detail in this article.

[0078] Fig. 7A and 7BSchematic diagrams of 2D convolution operation and 3D convolution operation according to embodiments of the present invention are respectively shown.

[0079] like Fig. 7A As shown in , assuming that the pixel value pixel(2D) of the activation input of the 2D convolution operation is a vector [a0,a1,a2,…,a31], the pixel value pixel_p(2D) of the activation input after padding is a vector [0,0,0,0,0,a0,a1,a2,a3,a4,a5,…,a30,a31,0,0,0,0,0,0,0,0,0,0], and the filter value filter(2D) of the weight input is a vector [w0,w1,w2,…,w15], then according to the principle of convolution operation, the activation output of the 2D convolution operation is:

[0080] z0=a0*w5+a1*w6+a2*w7+a3*w8+a4*w9+a5*w10+a6*w11+a7*w12+a8*w13+a9*w14+a10*w15

[0081] z1=a0*w4+a1*w5+a2*w6+a3*w7+a4*w8+a5*w9+a6*w10+a7*w11+a8*w12+a9*w13+a10*w14+a11*w15

[0082] z2=a0*w3+a1*w4+a2*w5+a3*w6+a4*w7+a5*w8+a6*w9+a7*w10+a8*w11+a9*w12+a10*w13+a11*w14+a12*w15

[0083] …

[0084] z30=a25*w0+a26*w1+a27*w2+a28*w3+a29*w4+a30*w5+a31*w6

[0085] z31=a26*w0+a27*w1+a28*w2+a29*w3+a30*w4+a31*w5

[0086] like Figure 7BAs shown in , the pixel value pixel(2D) of the activation input of the above 2D convolution operation is converted into a first pixel value pixel_x(3D) and a second pixel value pixel_y(3D), the first pixel value pixel_x(3D) and the second pixel value pixel_y(3D) are filled with a first filling pixel value pixel_xp(3D) and a second filling pixel value pixel_yp(3D), and the filter value filter(2D) of the weight input is converted into a first filter value filter_x(3D) and a second filter value filter_y(3D). According to the principle of convolution operation, the activation output of the 3D convolution operation is:

[0087] z0=a0*w5+a1*w6+a2*w7+a3*w8+a4*w9+a5*w10+a6*w11+a7*w12+a8*w13+a9*w14+a10*w15

[0088] z1=a0*w4+a1*w5+a2*w6+a3*w7+a4*w8+a5*w9+a6*w10+a7*w11+a8*w12+a9*w13+a10*w14+a11*w15

[0089] z2=a0*w3+a1*w4+a2*w5+a3*w6+a4*w7+a5*w8+a6*w9+a7*w10+a8*w11+a9*w12+a10*w13+a11*w14+a12*w15

[0090] …

[0091] z30=a25*w0+a26*w1+a27*w2+a28*w3+a29*w4+a30*w5+a31*w6

[0092] z31=a26*w0+a27*w1+a28*w2+a29*w3+a30*w4+a31*w5

[0093] It can be seen that after converting the 2D convolution operation to the 3D convolution operation through the above method, the pixel values ​​of the activation output are the same (only the dimensions are different).

[0094] Furthermore, in order to ensure that the pixel value of the activation output of the 2D convolution operation still meets the requirements of the 2D convolution operation, the activation output of the above 3D convolution operation can be converted into a 1D pixel value.

[0095] The above describes the computing device and method for implementing the convolution operation according to the present disclosure in conjunction with the accompanying drawings. However, it can be understood by those skilled in the art that the execution of the above method is not limited to the order shown in the figure and described above, but can be executed in any other reasonable order. In addition, the computing device does not necessarily include all the components shown in the figure, it may only include some of the components or more components necessary to perform the functions described in the present disclosure, and the connection method of these components is not limited to the form shown in the figure.

[0096] The present disclosure may be implemented as a method, a computing device, a system and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for executing various aspects of the present disclosure. The computing device may include at least one processor and at least one memory coupled to the at least one processor, the memory may store instructions for execution by the at least one processor. When the instructions are executed by the at least one processor, the computing device may execute the above method.

[0097] In one or more exemplary designs, the functions described in the present disclosure may be implemented using hardware, software, firmware, or any combination thereof. For example, if implemented using software, the functions may be stored as one or more instructions or codes on a computer-readable medium, or transmitted as one or more instructions or codes on a computer-readable medium.

[0098] The various units of the apparatus disclosed herein may be implemented using discrete hardware components or may be integrated on a hardware component, such as a processor. For example, the various exemplary logic blocks, modules, and circuits described in conjunction with the present disclosure may be implemented or executed using a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof for performing the functions described herein.

[0099] Those skilled in the art should also understand that the various illustrative logical blocks, modules, circuits, and algorithm steps described in conjunction with the embodiments of the present disclosure may be implemented as electronic hardware, computer software, or a combination of the two.

[0100] The above description of the present disclosure is intended to enable any person of ordinary skill in the art to implement or use the present disclosure. Various modifications of the present disclosure are obvious to those of ordinary skill in the art, and the general principles defined herein may also be applied to other variations without departing from the spirit and scope of the present disclosure. Therefore, the present disclosure is not limited to the examples and designs described herein, but is consistent with the broadest scope of the principles and novel features disclosed herein.

Claims

1. A method for implementing a convolution operation, comprising, in a 3D convolver: Receiving a weight input and an activation input of a low-dimensional convolution operation, wherein the weight input comprises a filter value and the activation input comprises a pixel value; Based on a predetermined filter depth of the 3D convolver, converting the filter value of the weight input into a first filter value and a second filter value of the 3D convolver, wherein the predetermined filter depth of the 3D convolver represents a maximum number of rows or columns processed by the 3D convolver at one time; converting the pixel value of the activation input into a first pixel value and a second pixel value of the 3D convolver based on a predetermined pixel data block size of the 3D convolver; and performing a 3D convolution operation based on the first filter value and the second filter value of the 3D convolver and the first pixel value and the second pixel value of the 3D convolver, The step of converting the filter value of the weight input into the first filter value and the second filter value of the 3D convolver comprises: Determining whether the filter value of the weight input is an integer multiple of the predetermined filter depth; In response to determining that the filter value of the weight input is not an integer multiple of the predetermined filter depth, splitting the low-dimensional convolution operation into a first convolution operation and a second convolution operation to determine a first filter value and a second filter value of the first convolution operation and the second convolution operation, respectively, and Wherein splitting the low-dimensional convolution operation into a first convolution operation and a second convolution operation comprises: rounding down the quotient of the filter value of the weight input and the predetermined filter depth as the second filter value of the first convolution operation, using the predetermined filter depth as a first filter value for the first convolution operation, setting the second filter value of the second convolution operation to 1, and The remainder of the filter value of the weight input and the predetermined filter depth is used as the first filter value of the second convolution operation.

2. The method of claim 1, wherein converting the filter value of the weight input into the first filter value and the second filter value of the 3D convolver further comprises: In response to determining that the filter value of the weight input is an integer multiple of the predetermined filter depth, a quotient of the filter value of the weight input and the predetermined filter depth is taken as the second filter value, and the predetermined filter depth is taken as the first filter value.

3. The method of claim 2, wherein converting the filter value of the weight input into the first filter value and the second filter value of the 3D convolver further comprises: Determining whether the filter value of the weight input is greater than the maximum processing filter value of the 3D convolver; In response to determining that the filter value of the weight input is greater than the maximum processing filter value of the 3D convolution device, the low-dimensional convolution operation is split into multiple convolution operations, wherein the filter value of one of the multiple convolution operations is less than the maximum processing filter value of the 3D convolution device, and the filter values ​​of other convolution operations in the multiple convolution operations are equal to the maximum processing filter value of the 3D convolution device.

4. The method of claim 1, wherein converting the pixel value of the activation input to the first pixel value and the second pixel value of the 3D convolver comprises: Determining a predetermined pixel depth of the 3D convolver as a second pixel value of the 3D convolver based on the predetermined pixel data block size; as well as A first pixel value of the 3D convolver is determined based on the pixel value of the activation input and a predetermined pixel depth of the 3D convolver, wherein the predetermined pixel depth of the 3D convolver represents a pixel size in a first pixel dimension or a second pixel dimension of the 3D convolver.

5. The method of claim 1, further comprising: Receiving a padding value for the low-dimensional convolution operation; as well as The padding value of the low-dimensional convolution operation is converted into a first padding value of a first pixel dimension and a second padding value of a second pixel dimension of the 3D convolver.

6. The method of claim 5, wherein converting the padding value of the low-dimensional convolution operation to a first padding value of a first pixel dimension and a second padding value of a second pixel dimension of the 3D convolver comprises: When the filling value of the low-dimensional convolution operation indicates left filling, determining the second filling value as a left filling value based on a high-order bit of the filling value; When the padding value of the low-dimensional convolution operation indicates right padding, determining the second padding value as a right padding value based on a high-order bit of the padding value; and The first padding value is determined based on the padding value, the second padding value, and a predetermined pixel data block size of the 3D convolver.

7. The method of claim 1, wherein the first filter dimension and the second filter dimension of the 3D convolver are perpendicular to each other.

8. The method of claim 1, wherein the first pixel dimension and the second pixel dimension of the 3D convolver are perpendicular to each other.

9. The method of claim 1, wherein the low-dimensional convolution operation comprises a 1D convolution operation and the weight input further comprises channel values.

10. The method of claim 1, wherein the low-dimensional convolution operation comprises a 2D convolution operation, the weight input further comprises an input channel value and an output channel value, and the activation input further comprises an input channel value.

11. A computing device comprising: A memory non-transitorily stores computer executable instructions; a processor configured to execute the computer executable instructions; Wherein, when the computer executable instructions are executed by the processor, the method according to any one of claims 1 to 10 is implemented.

12. A computer-readable storage medium having a computer program code stored thereon, wherein the computer program code executes the method according to any one of claims 1 to 10 when executed.

Images