Convolution operation acceleration method, circuit, system, device and image processing method

Abstract

The application belongs to the field of computers, and particularly relates to a convolution operation acceleration method, a circuit, a system, a device, a medium and an image processing method. The convolution operation acceleration method comprises the following steps: determining a channel sliding point-by-point convolution kernel and a corresponding sliding step length based on the number of channels; splitting the channel sliding point-by-point convolution kernel according to the sliding step length and a predetermined convolution acceleration algorithm; and accelerating the sliding convolution calculation of the split channel sliding point-by-point convolution kernel in the channel dimension by using the predetermined convolution acceleration algorithm. The convolution operation acceleration scheme provided by the application uses a channel sliding point-by-point convolution to replace a traditional ordinary convolution, and uses an acceleration algorithm to accelerate the sliding calculation of the channel sliding point-by-point convolution in the data channel, so that the energy efficiency ratio can be significantly improved.

Description

Technical Field

[0001] This invention belongs to the field of computer science, and specifically relates to methods, circuits, systems, devices, media, and image processing methods for accelerating convolution operations. Background Technology

[0002] In recent years, convolutional neural networks (CNNs) have achieved tremendous success in the field of computer vision, becoming a fundamental computational method in applications such as image classification, object detection, and image segmentation. However, with the development of neural networks, while the accuracy of the networks has increased, the number of network parameters has also grown significantly. This large number of parameters makes the hardware implementation of CNNs difficult, thus attracting widespread attention to research on neural network compression and simplification as well as hardware acceleration.

[0003] With the development of convolutional neural network algorithms, depthwise separable convolution (DSC) has become the mainstream lightweight convolution method. DSC consists of pointwise convolution (PWC) and depthwise convolution (DWC). The Winograd algorithm is a commonly used method for accelerating convolutional neural networks. Using Winograd can reduce the number of multiplications in convolution. For example, directly implementing an FIR filter F(y, r) with y outputs and r parameters requires y×r multiplications, but using the Winograd algorithm, if the transformation process is ignored, only y+r-1 multiplications are needed. Subsequently, Winograd has been widely used to accelerate neural networks, and due to its excellent characteristics, many hardware accelerator studies based on Winograd convolution have emerged. However, the benefits of using the Winograd algorithm to accelerate depthwise separable convolutions are very small. The kernel size of pointwise convolution is 1×1, and there is no overlapping region when the kernel slides in space. It is not a typical filtering form and is not suitable for acceleration by the Winograd convolution operator. Winograd can only accelerate depthwise convolutions in depthwise separable convolutions. The PWC in depthwise separable convolutions is tens or even hundreds of times more computationally expensive than DWC. Using Winograd only to accelerate depthwise convolutions results in a negligible overall acceleration effect. The Winograd algorithm has been almost abandoned in depthwise separable convolutional neural networks.

[0004] In summary, there is currently no effective solution to the problem that the Winograd algorithm cannot efficiently reduce the multiplication computation of depthwise separable convolutions, and that traditional Winograd convolutional neural network hardware accelerators cannot efficiently accelerate depthwise separable convolutional neural networks. Summary of the Invention

[0005] To address the above problems, this invention proposes a method for accelerating convolution operations, comprising:

[0006] The sliding pointwise convolution kernel and the corresponding sliding stride are determined based on the number of channels.

[0007] The channel sliding point-by-point convolution kernel is split according to the sliding stride and the predetermined convolution acceleration algorithm;

[0008] The predetermined convolution acceleration algorithm accelerates the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension.

[0009] In some embodiments of the present invention, splitting the channel sliding point-by-point convolution kernel according to the sliding stride and the predetermined convolution acceleration algorithm includes:

[0010] The channel sliding pointwise convolution kernel is split into convolution kernels that conform to the basic convolution operator of the predetermined convolution acceleration algorithm according to the sliding stride.

[0011] In some embodiments of the present invention, splitting the channel sliding pointwise convolution kernel into convolution kernels conforming to the basic convolution operators of the predetermined convolution acceleration algorithm according to the sliding stride includes:

[0012] Determine the dimension of the channel sliding pointwise convolution kernel;

[0013] The dimension of the convolution kernel of the predetermined convolution acceleration algorithm is determined based on the dimension, and the characteristics of the basic convolution kernel of the predetermined convolution acceleration algorithm are determined.

[0014] Multiple basic convolution kernels are formed by extracting elements from the channel sliding point-by-point convolution kernel according to the sliding stride.

[0015] In some embodiments of the present invention, extracting elements from the channel sliding convolution kernel according to the sliding stride to form multiple basic convolution kernels includes:

[0016] The value interval is determined by subtracting one from the sliding step size;

[0017] Based on the value interval, elements are extracted from the channel sliding point-by-point convolution kernel to form multiple basic convolution kernels.

[0018] In some embodiments of the present invention, the step of extracting elements from the channel sliding point-by-point convolution kernel according to the value interval to form a plurality of the basic convolution kernels includes:

[0019] Determine whether the length of the first convolutional kernel, which is composed of elements extracted from the convolutional kernel by sliding point-by-point convolutional kernel from the channel at the specified value interval, is greater than or less than the basic convolutional kernel;

[0020] In response to the first convolutional kernel having a length greater than the basic convolutional kernel, the first convolutional kernel is split once or multiple times according to the size of the basic convolutional kernel until the size of the split convolutional kernel is the same as that of the basic convolutional kernel; or

[0021] In response to the first convolution kernel having a length less than the base convolution kernel, the first convolution kernel is padded with a predetermined value so that its length is equal to the length of the base convolution kernel.

[0022] In some embodiments of the present invention, determining the channel sliding pointwise convolution kernel and the corresponding sliding stride based on the number of channels includes:

[0023] The channel sliding pointwise convolution kernel and its corresponding sliding stride are determined by comparing the results with those of a neural network model trained with a depthwise separable convolution kernel or by a neural network search algorithm.

[0024] In some embodiments of the present invention, determining the channel-sliding pointwise convolutional kernel and the corresponding sliding stride by comparing the results with a neural network model trained with a depthwise separable convolutional kernel or by a neural network search algorithm includes:

[0025] Randomly generate several channels of different lengths, slide point-by-point convolution kernels and sliding strides to train the neural network model;

[0026] The scores of several neural network models corresponding to sliding pointwise convolution kernels of several different channel lengths are compared with the scores of neural network models trained with depthwise separable convolution kernels.

[0027] Based on the comparison results, the channel sliding point-by-point convolution kernel and the corresponding sliding stride were determined.

[0028] In some embodiments of the present invention, determining the channel-sliding pointwise convolution kernel and the corresponding sliding stride based on the comparison results includes:

[0029] Determine the score that is closest to or exceeds the score of the neural network model trained with depth-separable convolutional kernels;

[0030] The length and stride of the sliding pointwise convolution kernel corresponding to the determined score are used as the determined length and stride of the sliding pointwise convolution kernel.

[0031] In some embodiments of the present invention, determining the channel-sliding pointwise convolutional kernel and the corresponding sliding stride by comparing the results with a neural network model trained with a depthwise separable convolutional kernel or by a neural network search algorithm includes:

[0032] Based on the number of channels, multiple sliding point-by-point convolutional kernels and sliding strides of corresponding lengths are generated iteratively using a binary search method, and the model is trained using the multiple sliding point-by-point convolutional kernels and sliding strides of the channels.

[0033] Based on the training results, determine the channel sliding pointwise convolution kernel and the corresponding sliding stride.

[0034] In some embodiments of the present invention, determining the channel-sliding pointwise convolutional kernel and the corresponding sliding stride by comparing the results with a neural network model trained with a depthwise separable convolutional kernel or by a neural network search algorithm includes:

[0035] The length of the sliding pointwise convolution kernel and the sliding stride are obtained by using a neural architecture search algorithm and reinforcement learning.

[0036] In some embodiments of the present invention, the method further includes:

[0037] Convert ordinary convolutional kernels used for multiple data channels into depthwise convolutional kernels;

[0038] Intermediate data with multiple data channels is generated based on the depthwise convolution kernel.

[0039] In some embodiments of the present invention, accelerating the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension using the predetermined convolution acceleration algorithm includes:

[0040] The predetermined convolution acceleration algorithm accelerates the sliding pointwise convolution kernel of the split channels to perform sliding convolution calculations on the intermediate data of the multiple data channels.

[0041] In some embodiments of the present invention, the method further includes:

[0042] Convert ordinary convolutional kernels used for multiple data channels into depthwise convolutional kernels and pointwise convolutional kernels;

[0043] Intermediate data with multiple data channels is generated based on the depth-wise convolution kernel;

[0044] The first intermediate data allocation ratio of the pointwise convolution kernel and the second intermediate data allocation ratio of the channel sliding pointwise convolution kernel are set based on the minimum time for calculating the convolutional data of the multiple data channels, the time consumed by the pointwise convolution kernel calculation, and the time consumed by the channel sliding pointwise convolution kernel calculation.

[0045] In some embodiments of the present invention, accelerating the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension using the predetermined convolution acceleration algorithm includes:

[0046] Based on the pointwise convolution kernel, convolution calculation is performed on the intermediate data allocated to the first intermediate data.

[0047] The predetermined convolution acceleration algorithm accelerates the sliding convolution calculation of the intermediate data allocated to the second intermediate data by the sliding point-by-point convolution kernel of the split channel.

[0048] In some embodiments of the present invention, the method further includes: setting up a one-dimensional computation unit according to the predetermined convolution acceleration algorithm.

[0049] In some embodiments of the present invention, the method further includes:

[0050] Several one-dimensional computing units are combined into a two-dimensional computing unit.

[0051] In some embodiments of the present invention, generating intermediate data with multiple data channels based on the depthwise convolution kernel includes:

[0052] The two-dimensional computing unit calculates intermediate data for multiple data channels based on the depthwise convolution kernel.

[0053] In some embodiments of the present invention, performing convolution calculations on the intermediate data allocated to the first intermediate data based on the pointwise convolution kernel includes:

[0054] The one-dimensional computing unit performs convolution calculations on the intermediate data allocated to the first intermediate data based on the pointwise convolution kernel.

[0055] In some embodiments of the present invention, accelerating the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension using the predetermined convolution acceleration algorithm includes:

[0056] The one-dimensional computing unit accelerates the sliding convolution calculation of the corresponding intermediate data by the sliding point-by-point convolution kernel of the channel after being split according to the predetermined convolution acceleration algorithm.

[0057] Another aspect of the present invention provides a hardware acceleration circuit, comprising:

[0058] An accelerated computing module is configured to accelerate the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension using the predetermined convolution acceleration algorithm.

[0059] Another aspect of the present invention provides a convolution operation acceleration system, comprising:

[0060] A kernel determination module is configured to determine the channel sliding point-by-point convolution kernel and the corresponding sliding stride based on the number of channels.

[0061] A kernel splitting module is configured to split the channel sliding point-by-point convolution kernel according to the sliding stride and a predetermined convolution acceleration algorithm;

[0062] A convolution operation acceleration module is configured to accelerate the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension through the predetermined convolution acceleration algorithm.

[0063] Another aspect of the present invention provides a computer device comprising:

[0064] At least one processor; and

[0065] A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of any of the methods described in the above embodiments.

[0066] Another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the methods described in the above embodiments.

[0067] Another aspect of the present invention provides an image processing method, comprising:

[0068] Obtain the image to be processed;

[0069] The image is input into a convolutional neural network model for image data processing, wherein the convolutional neural network model is accelerated by the convolution operation acceleration method described in the above embodiments.

[0070] The present invention proposes a method, system, device, medium, and circuit for accelerating convolution operations, which transforms traditional ordinary convolution into depthwise convolution and channel sliding pointwise convolution. Channel sliding pointwise convolution replaces the pointwise convolution in traditional depthwise separable convolution, and the one-dimensional operator of the Winograd algorithm is used to accelerate the sliding calculation of channel sliding pointwise convolution on the data channel, which can significantly improve the energy efficiency ratio.

[0071] Furthermore, depthwise convolution calculations in depthwise separable convolution can be accelerated by combining several one-dimensional computing units into a two-dimensional computing unit. Both depthwise convolution calculations and sliding pointwise convolution calculations can be accelerated by the most basic one-dimensional computing unit, effectively improving the utilization and flexibility of hardware computing units.

[0072] Furthermore, in depthwise separable convolution computation, when calculating the intermediate data output by depthwise convolution, the simultaneous computation of pointwise convolution and channel sliding pointwise convolution on the intermediate data can be achieved according to the specific computing device and the time requirements of convolution computation in the network layer corresponding to the model. On the one hand, the Winograd algorithm can be used to accelerate channel sliding pointwise convolution, thereby reducing the convolution computation time to a certain extent; on the other hand, the computational characteristics of pointwise convolution in all channels can be maintained to stabilize the accuracy of the model.

[0073] Furthermore, the proposed scheme for accelerating depthwise separable convolution computation can significantly improve the computational performance of models containing convolutional neural networks while also expanding the size of the image data corresponding to the convolutional neural network. This means that the size of the image data or other types of feature map data input to the model can also be larger. Therefore, under the premise of a certain computational power, achieving a larger input image can improve the model's recognition ability to a certain extent, thus offsetting the model accuracy loss when using channel sliding pointwise convolution kernels. Attached Figure Description

[0074] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0075] Figure 1 A flowchart illustrating a method for accelerating convolution operations provided in an embodiment of the present invention;

[0076] Figure 2 A schematic diagram of a hardware acceleration circuit provided in an embodiment of the present invention;

[0077] Figure 3 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention;

[0078] Figure 4 This is a schematic diagram of the structure of a computer-readable storage medium provided in an embodiment of the present invention;

[0079] Figure 5 A comparison diagram of ordinary convolution and depthwise convolution + channel sliding pointwise convolution provided in the embodiments of the present invention;

[0080] Figure 6 A detailed structural schematic diagram of a hardware acceleration circuit provided in an embodiment of the present invention;

[0081] Figure 7A schematic diagram of the calculation process of a one-dimensional accelerated computing circuit provided in an embodiment of the present invention;

[0082] Figure 8 This is a schematic diagram of a one-dimensional accelerated computing circuit structure provided in an embodiment of the present invention.

[0083] Figure 9 A schematic diagram of the row transformation calculation mode provided in an embodiment of the present invention;

[0084] Figure 10 A schematic diagram of the parallel computation mode of depthwise convolution and sliding channel convolution provided in an embodiment of the present invention;

[0085] Figure 11 This is a schematic diagram of a convolution operation acceleration system provided in an embodiment of the present invention;

[0086] Figure 12 This is a flowchart illustrating an image processing method provided in an embodiment of the present invention. Detailed Implementation

[0087] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0088] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0089] As mentioned above, this invention aims to address the problem that depthwise separable convolution can compress ordinary neural networks, but the Winograd algorithm provides very little acceleration benefit for depthwise separable convolution, and the two cannot be effectively combined.

[0090] like Figure 1 As shown, this invention proposes a method for accelerating convolution operations, comprising:

[0091] Step S1: Determine the channel sliding pointwise convolution kernel and the corresponding sliding stride based on the number of channels;

[0092] Step S2: Split the channel sliding point-by-point convolution kernel according to the sliding stride and the predetermined convolution acceleration algorithm;

[0093] Step S3: Accelerate the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension using the predetermined convolution acceleration algorithm.

[0094] In this invention, such as Figure 5 As shown in the upper part, traditional convolution operations, when designing multi-channel data, directly perform convolution operations on all channels of data using ordinary convolution kernels equal to the number of data channels. As a further improvement, traditional depthwise separable convolution consists of pointwise convolution and depthwise convolution, such as... Figure 5 The lower part shows the process from "depth-wise convolution kernel" to "pointwise convolution kernel," but in this process, the length of the pointwise convolution kernel and the number of data channels are both M. Therefore, the calculation is similar to... Figure 5 The calculation method of ordinary convolution kernels in the previous part is the same, which requires convolution operation on all channels of data, and cannot be accelerated. In view of this, the present invention proposes a channel sliding point-by-point convolution kernel that realizes sliding convolution calculation on the data channels, that is... Figure 5 The calculation process from "depth-wise convolution kernel" to "channel sliding point-wise convolution kernel" in the lower middle part replaces the traditional calculation method from "depth-wise convolution kernel" to "point-wise convolution kernel part". The predetermined convolution acceleration algorithm involved in this invention may include the Winograd algorithm.

[0095] It should be noted that the channel sliding pointwise convolution kernel is a 1x1 convolution kernel of length r. The term "pointwise" means that the size of the convolution kernel is 1.

[0096] Furthermore, in an embodiment of the present invention, in step S1, based on the dimension of the data channel (e.g. Figure 5 In the input feature map shown, the dimension of the data channels is M. Generate a channel sliding pointwise convolutional kernel with a length less than M and the corresponding sliding stride. Specifically, the generation method can employ a Neural Architecture Search (NAS) algorithm, or randomly generate multiple channel sliding pointwise convolutional kernels of different lengths and compare the accuracy of these kernels with that of a traditional depthwise separable convolution model to determine the appropriate length r and stride d of the channel sliding pointwise convolutional kernel.

[0097] It should be noted that step S1 is usually used in model training to determine the corresponding channel sliding pointwise convolution kernel and the corresponding stride.

[0098] In addition, after determining the channel sliding pointwise convolution kernel and stride in the neural network in step S1, the corresponding model is trained using a corresponding training framework, such as PyTorch or TensorFlow.

[0099] Furthermore, in step S2, after the model has been trained, the length and stride of the corresponding channel sliding pointwise convolution kernels involved in convolution operations for each network layer are determined. Therefore, during model inference, calculations need to be performed based on the determined length r of the channel sliding pointwise convolution kernels and the corresponding stride d. Since the length of the channel sliding pointwise convolution kernel is r, and the design of this invention allows for sliding calculations in the data channel dimension with a stride of d, this method results in adjacent sliding portions sharing data in the data channel dimension, which can be accelerated using the Winograd algorithm.

[0100] The principle of using the Winograd algorithm for acceleration is as follows: the Winograd algorithm is used to reduce the multiplication operations of the network. In some embodiments, 2D Winograd is used for depthwise convolution and 1D Winograd is used for channel sliding pointwise convolution.

[0101] For channel-sliding pointwise convolution, a single pixel position can be viewed as a convolution kernel sliding along the channel dimension. Therefore, the 1D Winograd algorithm can be used for acceleration. A commonly used 1D Winograd fast convolution operator is F(2,3), indicating two outputs and a kernel size of 3. For the input signal... convolution kernel 1D convolution (one-dimensional convolution) can be written in matrix multiplication form:

[0102] ;

[0103] Calculating this convolution using ordinary matrix multiplication requires 6 multiplications and 4 additions. A close inspection of the input matrix reveals a regular distribution of repeating elements, such as... and Appearing in both the first and second rows, the matrix multiplication transformed by convolution has a smaller problem domain than that of ordinary matrix multiplication. Winograd's approach is as follows.

[0104] ;

[0105] in,

[0106] ;

[0107] ;

[0108] ;

[0109] ;

[0110] During the inference process of a convolutional neural network, the parameters of the convolution kernel remain unchanged, therefore, regarding the convolution kernel... q The computation can be pre-calculated and the results stored for reuse. During inference, this computation can be ignored. Therefore, after optimization with Winograd, only 4 multiplications and 8 additions are required, which is 2 fewer multiplications and 4 fewer additions than general matrix multiplication. Addition is much simpler than multiplication, thus achieving speedup. The Winograd fast convolution algorithm can be written in matrix form as follows:

[0111] ;

[0112] in, For element-wise multiplication, it is equivalent to to Multiplication in , , G They are shown below:

[0113] ;

[0114] ;

[0115] ;

[0116] G The convolution kernel transformation matrix, It is the input transformation matrix. The output transformation matrix is ​​used, therefore the entire computation is divided into four steps: input transformation, kernel transformation, element-wise multiplication, and output transformation. Other Winograd convolution operators with different kernel sizes and output counts have corresponding transformation matrices. The F(2,3) transformation matrix only contains 0, ... and , Corresponding to addition and subtraction operations, and Related operations can be implemented using bit shifting. Other F(y, r) transformation matrices are more complex and may introduce additional operations or precision loss. Therefore, F(2, 3) is the most commonly used Winograd convolution operator. When the number of output elements is greater than 2, the input can be simply pruned and split into multiple F(2, 3) inputs, and then calculated separately.

[0117] Furthermore, to integrate more input channels, the kernel size of channel sliding pointwise convolution is usually greater than 3, and the stride is not 1. In order to use F(2,3) to calculate channel sliding pointwise convolution, it is necessary to process the channel sliding pointwise convolution. A 1D convolution with 2 output elements, kernel size r, and stride d can be written as F(2,r,d). F(2,r,d) can be decomposed into d convolution kernels of size r / d, each with a stride of 1, i.e., d F(2,r / d). If r is very large and d is very small, r / d will be greater than 3. In this case, it is necessary to continue to decompose F(2,r / d). F(2,r / d) will be decomposed into t convolution kernels of size 3, i.e., t F(2,3). The above two steps are decomposed into F(2, r, d) → d×F(2, r / d) → (r / 3)×F(2, 3). After the two-step decomposition, F(2, r, d) is decomposed into r / 3 F(2, 3). All F(2, 3) are calculated separately and the results are added together to obtain the final result.

[0118] Furthermore, in step S3, after the channel sliding point-by-point convolution kernel of length r is split into the smallest operator F(2,3) of the Winograd algorithm according to the sliding stride, the corresponding convolution calculation result can be obtained by sliding calculation in the dimension of the data channel through Winograd calculation.

[0119] In some embodiments of the present invention, splitting the channel sliding point-by-point convolution kernel according to the sliding stride and the predetermined convolution acceleration algorithm includes:

[0120] The channel sliding pointwise convolution kernel is split into convolution kernels that conform to the basic convolution operator of the predetermined convolution acceleration algorithm according to the sliding stride.

[0121] In some embodiments of the present invention, splitting the channel sliding pointwise convolution kernel into convolution kernels conforming to the basic convolution operators of the predetermined convolution acceleration algorithm according to the sliding stride includes:

[0122] Determine the dimension of the channel sliding pointwise convolution kernel;

[0123] The dimension of the convolution kernel of the predetermined convolution acceleration algorithm is determined based on the dimension, and the characteristics of the basic convolution kernel of the predetermined convolution acceleration algorithm are determined.

[0124] Multiple basic convolution kernels are formed by extracting elements from the channel sliding point-by-point convolution kernel according to the sliding stride.

[0125] In some embodiments of the present invention, extracting elements from the channel sliding convolution kernel according to the sliding stride to form multiple basic convolution kernels includes:

[0126] The value interval is determined by subtracting one from the sliding step size;

[0127] Based on the value interval, elements are extracted from the channel sliding point-by-point convolution kernel to form multiple basic convolution kernels.

[0128] In some embodiments of the present invention, as mentioned above, the dimension of the channel sliding pointwise convolution kernel is 1. Therefore, the dimension is determined. Further, the length of the channel sliding pointwise convolution kernel is r, and the sliding stride is d. The purpose of splitting in the present invention is to split the channel sliding pointwise convolution kernel of length r into the smallest convolution operator F(2,3) in a one-dimensional scene under the Winograd algorithm. F(2,3) indicates that the length of the convolution kernel is 3. Then, the channel sliding pointwise convolution kernel of length r can be split into r / 3 (F(2,3) convolution kernels. At the same time, the splitting should also be performed according to the stride d. Specifically, when the stride is 1, the channel sliding pointwise convolution kernel of length r will continuously extract 3 elements as a Winograd algorithm F(2,3). If the sliding stride is 2, the channel sliding pointwise convolution data will be extracted in the form of 1, 3, 5; 2, 4, 6 every other element as a Winograd algorithm F(2,3) convolution kernel.

[0129] In some embodiments of the present invention, the step of extracting elements from the channel sliding point-by-point convolution kernel according to the value interval to form a plurality of the basic convolution kernels includes:

[0130] Determine whether the length of the first convolutional kernel, which is composed of elements extracted from the convolutional kernel by sliding point-by-point convolutional kernel from the channel at the specified value interval, is greater than or less than the basic convolutional kernel;

[0131] In response to the first convolutional kernel having a length greater than the basic convolutional kernel, the first convolutional kernel is split once or multiple times according to the size of the basic convolutional kernel until the size of the split convolutional kernel is the same as that of the basic convolutional kernel; or

[0132] In response to the first convolution kernel having a length less than the base convolution kernel, the first convolution kernel is padded with a predetermined value so that its length is equal to the length of the base convolution kernel.

[0133] In some embodiments of the present invention, the length of the channel sliding pointwise convolution kernel is r, and the sliding stride is d. Regarding how to design the relationship between r and d so that only F(2, 3) remains after decomposition, the following method can be used: First, the general form F(2, r, d) can be decomposed into d convolution kernels of size r / d (to more clearly and easily illustrate this decomposition method, it is assumed that r is divisible by d, i.e., r%d=0; otherwise, the decomposed convolution kernels will not be of the same size, potentially resulting in multiple sizes of convolution kernels). The stride of each convolution kernel is 1, i.e., d F(2, r / d). If r is large and d is small, r / d will be greater than 3. In this case, the method of decomposing large convolution kernels with a stride of 1 used previously needs to decompose F(2, r / d). The decomposition of r / d is divided into t convolutional kernels of size 3 (to more easily illustrate the decomposition method, it is assumed that r / d is divisible by 3, i.e., r / d = 3t; otherwise, the decomposed convolutional kernels will not be of the same size, and multiple sizes of convolutional kernels may be generated. For convolutional kernels of multiple sizes, when converting them into basic convolutional kernels of the form F(2,3), the parts with a length smaller than F(2,3) are padded with 0s or 1s to align them with or be the same as F(2,3)). That is, t F(2,3). The above two-step decomposition is F(2,r,d) → d×F(2,r / d) → (r / 3)×F(2,3). After the two-step decomposition, F(2,r,d) is decomposed into r / 3 F(2,3). The final result is obtained by calculating all F(2,3) separately and then adding the results. To ensure that the decomposed F(2, r, d) consists only of F(2, 3), the values ​​of r and d should be carefully designed. If the values ​​of r and d are set arbitrarily, other sizes of convolution kernels will be generated after decomposition, which is not hardware-friendly and does not facilitate the use of a unified hardware acceleration computing unit for acceleration. Therefore, when using SKC (channel sliding pointwise convolution kernel) convolution, the values ​​of r and d should be carefully designed for better hardware friendliness. The principle to be followed is r = 3dt, where t is a positive integer. In this way, F(2, r, d) can be decomposed into r / 3 = dt commonly used F(2, 3) kernels without other types of convolution kernels.

[0134] In some embodiments of the present invention, determining the channel sliding pointwise convolution kernel and the corresponding sliding stride based on the number of channels includes:

[0135] The channel sliding pointwise convolution kernel and its corresponding sliding stride are determined by comparing the results with those of a neural network model trained with a depthwise separable convolution kernel or by a neural network search algorithm.

[0136] In other words, when determining the channel sliding point-by-point convolution kernel and the corresponding sliding stride, the comparison results with the traditional depthwise separable convolution kernel training model or the neural network search algorithm can be used to determine the channel sliding point-by-point convolution kernel and the corresponding sliding stride.

[0137] In some embodiments of the present invention, determining the channel-sliding pointwise convolutional kernel and the corresponding sliding stride by comparing the results with a neural network model trained with a depthwise separable convolutional kernel or by a neural network search algorithm includes:

[0138] Randomly generate several channels of different lengths, slide point-by-point convolution kernels and sliding strides to train the neural network model;

[0139] The scores of several neural network models corresponding to sliding pointwise convolution kernels of several different channel lengths are compared with the scores of neural network models trained with depthwise separable convolution kernels.

[0140] Based on the comparison results, the channel sliding point-by-point convolution kernel and the corresponding sliding stride were determined.

[0141] In this embodiment, as mentioned above, when determining the channel sliding pointwise convolution kernel and sliding stride, multiple channel sliding pointwise convolution kernels and sliding strides can be randomly generated for subsequent model training. Then, the scores of the models corresponding to the multiple channel sliding pointwise convolution kernels and sliding strides are compared with the scores of the models trained using traditional depthwise separable convolution kernels. The channel sliding pointwise convolution kernel and sliding stride combination whose score is closest to or exceeds that of the neural network model trained using depthwise separable convolution kernels is selected as the preferred channel sliding pointwise convolution kernel and sliding stride.

[0142] In some embodiments of the present invention, determining the channel-sliding pointwise convolution kernel and the corresponding sliding stride based on the comparison results includes:

[0143] Determine the score that is closest to or exceeds the score of the neural network model trained with depth-separable convolutional kernels;

[0144] The length and stride of the sliding pointwise convolution kernel corresponding to the determined score are used as the determined length and stride of the sliding pointwise convolution kernel.

[0145] In some embodiments of the present invention, determining the channel-sliding pointwise convolutional kernel and the corresponding sliding stride by comparing the results with a neural network model trained with a depthwise separable convolutional kernel or by a neural network search algorithm includes:

[0146] Based on the number of channels, the binary search method iteratively generates multiple sliding point-by-point convolution kernels and sliding strides of corresponding lengths for the channels, and trains the model using the multiple sliding point-by-point convolution kernels and sliding strides.

[0147] Based on the training results, determine the channel sliding pointwise convolution kernel and the corresponding sliding stride.

[0148] In some embodiments, instead of randomly generating multiple channel sliding point-by-point convolution kernels and sliding strides of corresponding lengths as described above, a binary search method can be used to quickly find the corresponding preferred channel sliding point-by-point convolution kernel and sliding stride.

[0149] In some embodiments of the present invention, determining the channel-sliding pointwise convolutional kernel and the corresponding sliding stride by comparing the results with a neural network model trained with a depthwise separable convolutional kernel or by a neural network search algorithm includes:

[0150] The length of the sliding pointwise convolution kernel and the sliding stride are obtained by using a neural architecture search algorithm and reinforcement learning.

[0151] Furthermore, in embodiments of the present invention, when determining the preferred channel sliding pointwise convolution kernel and sliding stride, Neural Architecture Search (NAS) technology can be used to search for r and d values ​​using reinforcement learning. Before the search, the relationship between r and d is constrained to satisfy the following: that is, for the data channels M of the input feature image, in order to fuse sufficient channel features and ensure network accuracy, the channel sliding pointwise convolution kernel size r should be greater than M / 2, and the stride d should be less than r. Simultaneously, to ensure that the size of the decomposed convolution kernel is only 3, the above-described relationship between r and d should also be satisfied.

[0152] In some embodiments of the present invention, the method further includes:

[0153] Convert ordinary convolutional kernels used for multiple data channels into depthwise convolutional kernels;

[0154] Intermediate data with multiple data channels is generated based on the depthwise convolution kernel.

[0155] like Figure 5 As shown, in this embodiment, the traditional ordinary convolution kernel is transformed into depthwise convolution in depthwise separable convolution, and the depthwise convolution generates intermediate data with multiple data channels.

[0156] In some embodiments of the present invention, accelerating the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension using the predetermined convolution acceleration algorithm includes:

[0157] The predetermined convolution acceleration algorithm accelerates the sliding pointwise convolution kernel of the split channels to perform sliding convolution calculations on the intermediate data of the multiple data channels.

[0158] In this embodiment, the present invention replaces the pointwise convolution in traditional depthwise separable convolution with channel sliding pointwise convolution. Therefore, as Figure 5The calculation process of the "channel sliding pointwise convolution kernel" in the lower half of the diagram is shown. The channel sliding pointwise convolution kernel with a length of r and less than the number of channels M of the intermediate data performs convolution calculation in a sliding manner on the intermediate data. Therefore, there will be duplicate channel data in the sliding calculation of adjacent data channels. For such cases, the Winograd algorithm can be used to accelerate the calculation. That is, the aforementioned split channel sliding pointwise convolution kernel is used to perform sliding convolution calculation on the intermediate data channels (while using the Winograd algorithm for acceleration).

[0159] In some embodiments of the present invention, the method further includes:

[0160] Convert ordinary convolutional kernels used for multiple data channels into depthwise convolutional kernels and pointwise convolutional kernels;

[0161] Intermediate data with multiple data channels is generated based on the depth-wise convolution kernel;

[0162] The first intermediate data allocation ratio of the pointwise convolution kernel and the second intermediate data allocation ratio of the channel sliding pointwise convolution kernel are set based on the minimum time for calculating the convolutional data of the multiple data channels, the time consumed by the pointwise convolution kernel calculation, and the time consumed by the channel sliding pointwise convolution kernel calculation.

[0163] In some embodiments of the present invention, such as Figure 5 As shown, ordinary convolutional kernels are also converted into depthwise and pointwise convolutional kernels using the traditional depthwise separable convolution method, and the convolution calculation method proposed in this invention, which involves the coexistence of channel-sliding pointwise convolutional kernels and pointwise convolutional kernels, is adopted. Because the proposed "channel-sliding pointwise convolutional kernel" and the traditional pointwise convolutional kernel both participate in the calculation of intermediate data, and the proposed channel-sliding pointwise convolutional kernel can be accelerated by the Winograd algorithm, there is a difference in the speed of convolution calculation between the two. This difference can be used as an allocation condition when allocating intermediate data. Specifically, based on the minimum time requirement for the convolution calculation of the neural network layer where the convolution calculation is performed, and based on the time consumed by the channel-sliding pointwise convolutional kernel and the traditional pointwise convolutional kernel in calculating the same amount of intermediate data, the second and first intermediate data allocation ratios corresponding to the channel-sliding pointwise convolutional kernel and the traditional pointwise convolutional kernel are calculated based on their respective time consumption and the minimum time requirement for the convolution calculation of the neural network layer.

[0164] In some embodiments of the present invention, accelerating the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension using the predetermined convolution acceleration algorithm includes:

[0165] Based on the pointwise convolution kernel, convolution calculation is performed on the intermediate data allocated to the first intermediate data.

[0166] The predetermined convolution acceleration algorithm accelerates the sliding convolution calculation of the intermediate data allocated to the second intermediate data by the sliding point-by-point convolution kernel of the split channel.

[0167] In some embodiments of the present invention, after determining the second and first intermediate data allocation ratios corresponding to the channel sliding pointwise convolution kernel and the traditional pointwise convolution kernel, the intermediate data is divided into two parts based on the corresponding ratios of the channel sliding pointwise convolution kernel and the traditional pointwise convolution kernel, and calculated separately by the pointwise convolution kernel and the channel sliding pointwise convolution kernel. Simultaneously, the Winograd algorithm can be used to accelerate the calculation of intermediate data by the channel sliding pointwise convolution kernel. This approach balances model accuracy with the given computing power of the device, allowing for the completion of convolution calculations for different neural network layers within a specified time.

[0168] In some embodiments of the present invention, the method further includes: setting up a one-dimensional computation unit according to the predetermined convolution acceleration algorithm.

[0169] In some embodiments of the present invention, the method further includes: combining a plurality of the one-dimensional computing units into a two-dimensional computing unit.

[0170] In some embodiments of the present invention, the present invention implements a one-dimensional computation unit based on the Winograd algorithm. The one-dimensional computation unit can be a one-dimensional basic convolution operator of the Winograd algorithm implemented by hardware circuit, namely the F(3,2) convolution operator, which can be directly used for accelerated computation after channel sliding point-by-point convolution splitting.

[0171] In some embodiments of the present invention, a number of one-dimensional computing units are combined to form a two-dimensional computing unit, which can be used to accelerate the two-dimensional convolution operation of the Winograd algorithm.

[0172] In some embodiments of the present invention, generating intermediate data with multiple data channels based on the depthwise convolution kernel includes:

[0173] The two-dimensional computing unit calculates intermediate data for multiple data channels based on the depthwise convolution kernel.

[0174] In the embodiments of this invention, as described above, this invention is an improvement on depthwise separable convolution operations. The first step of depthwise separable convolution operations is to obtain intermediate data by performing convolution calculations on the input feature data using depthwise convolution kernels. The depthwise convolution kernel is a k×k two-dimensional convolution kernel, so the calculation of the depthwise convolution kernel in the depthwise separable convolution operation process can be realized through a two-dimensional computing unit composed of multiple one-dimensional computing units, and the calculation of the depthwise convolution kernel can be accelerated by using the Winograd algorithm.

[0175] In some embodiments of the present invention, accelerating the sliding pointwise convolutional kernel of the split channels to perform sliding convolution calculations on the intermediate data of the multiple data channels using the predetermined convolution acceleration algorithm includes:

[0176] The sliding convolution calculation of the channel sliding point-by-point convolution kernel after the channel splitting is accelerated based on the one-dimensional computing unit.

[0177] In this embodiment, the channel sliding pointwise convolution kernel proposed in this invention is split into several convolution kernels of the basic operator F(3,2) of the Winograd algorithm, and the split channel sliding pointwise convolution kernels are respectively given to the corresponding one-dimensional computing units for hardware circuit acceleration calculation.

[0178] In some embodiments of the present invention, accelerating the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension using the predetermined convolution acceleration algorithm includes:

[0179] The one-dimensional computing unit accelerates the sliding convolution calculation of the corresponding intermediate data by the sliding point-by-point convolution kernel of the channel after being split according to the predetermined convolution acceleration algorithm.

[0180] In this embodiment, when performing a partial convolution operation using a combination of channel sliding pointwise convolution kernels and pointwise convolution kernels for depth-separable convolution, the sliding convolution operation of the split channel sliding pointwise convolution kernels on all or the intermediate data of the second intermediate data allocation ratio can be accelerated by a one-dimensional computing unit according to the Winograd algorithm.

[0181] like Figure 2 As shown, another aspect of the present invention also proposes a hardware acceleration circuit, including an acceleration computing module 2, which is configured to accelerate the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension through the predetermined convolution acceleration algorithm.

[0182] like Figure 5 As shown, in some further embodiments of the present invention, the hardware acceleration circuit proposed in the present invention further includes:

[0183] Finite state machine 1 and cache module 3. Finite state machine 1 is used to configure the data in the convolution calculation of different neural network layers and the calculation method of acceleration computing module 2 during the training or inference phase of the neural network according to the specifications of the convolution operation of different network layers. Cache module 3 is used to provide operation data to the acceleration computing module and cache the output data of the acceleration computing module, and send the corresponding output data to memory according to preset logic.

[0184] In some embodiments, the hardware circuit can be implemented by an FPGA (Field Programmable Gate Array) chip, including: a finite-state machine 1, an accelerated computing module 2, and a cache module 3. The finite-state machine 1 (FSM) is a module that controls the computation method of the PL (Programmable Logic). In this embodiment, based on the length of the channel sliding point-by-point convolution in the convolution operation of each layer of the trained neural model, it is divided into several minimum operators under the F(2,3) Winograd algorithm, and these are assigned to the accelerated computing module 2 for accelerated computation according to the Winograd algorithm. That is, the channel sliding point-by-point convolution kernel and its corresponding sliding stride are split into multiple minimum operators under the Winograd algorithm, i.e., convolution kernels of the F(2,3) form, and configured for computation in the accelerated computing module 2.

[0185] In an embodiment of the present invention, the accelerated computing module 2 obtains the corresponding input feature data or intermediate input feature data and the corresponding weight data (corresponding convolution kernel) from the cache module 3, and then performs convolution calculation on a certain neural network layer in the convolution calculation mode configured in the finite state machine 1.

[0186] Furthermore, the caching module 3 is used to provide the convolution kernel and input features for convolution calculation to the accelerated computing module 2, or to save the calculated output feature results. In some embodiments of the present invention, the accelerated computing module 2 further includes:

[0187] A one-dimensional computing unit 21 is implemented by a corresponding logic circuit according to the one-dimensional computing process of a predetermined convolution acceleration algorithm, and is configured to implement the one-dimensional operation process of the predetermined convolution acceleration algorithm with the corresponding logic circuit.

[0188] Furthermore, in this embodiment, the accelerated computing module 2 is composed of multiple one-dimensional computing units 21, such as... Figure 8 As shown, the one-dimensional computing unit 21 is a 1D Winograd F(2,3) operator implemented in hardware circuitry, which can directly accelerate the convolution operation with a sliding channel pointwise convolution kernel length of 3.

[0189] In some embodiments of the present invention, the one-dimensional computing unit includes:

[0190] Row transformation module 211, configured to perform row transformation on the input data in the one-dimensional calculation process of the predetermined convolution acceleration algorithm;

[0191] Element multiplication module 212 is configured to control the multiplication of input data input to the one-dimensional calculation unit with the corresponding weights.

[0192] In this embodiment, the one-dimensional calculation unit 21 also includes a row transformation module 211 and an element multiplication module 212. Further... Figure 7 As shown, the row transformation module 211 controls the one-dimensional calculation unit 21 to perform row transformations on the input data for the 1D Winograd F(2,3) operator accelerated calculation, i.e., according to... Figure 7 The Winograd algorithm's subtraction, addition, subtraction, and subtraction operations are implemented in the manner shown. The element-multiplication module 212 then multiplies the results of the above subtraction, addition, subtraction, and subtraction operations by the corresponding values ​​on the corresponding convolution kernels.

[0193] In some embodiments of the present invention, the accelerated computing module 2 further includes:

[0194] A two-dimensional computing unit 22 is composed of multiple one-dimensional computing units 21 according to the two-dimensional computing process of the predetermined convolution acceleration algorithm, and is configured to implement the two-dimensional operation process of the predetermined convolution acceleration algorithm through multiple one-dimensional computing units.

[0195] In this embodiment, as Figure 10 As shown, in acceleration module 2, four one-dimensional computation units 21 are combined to form a two-dimensional computation unit 22 that can implement the 2DWinograd algorithm. This can be used to accelerate the convolution calculation of depthwise convolution in depthwise separable convolution.

[0196] In some embodiments of the present invention, the finite state machine 1 is configured to:

[0197] Based on the convolution calculation method of the corresponding network layer of the deep neural network, one or more of the one-dimensional or two-dimensional computing units are configured to perform the convolution calculation of the network layer.

[0198] In this embodiment, the finite state machine 1 flexibly configures the one-dimensional or two-dimensional computing units in the accelerated computing module 2 to perform the convolution calculation of depthwise convolution and the channel sliding pointwise convolution calculation in separable convolution, according to the convolution calculation method of the corresponding network layer of the neural network.

[0199] Specifically, such as Figure 10As shown, depthwise convolution (DWC) can unfold 16 output channels and 4 output pixels, with a 3x3 convolution kernel; channel sliding pointwise convolution can unfold 12 input channels and 32 output channels. A two-dimensional computing unit 22 can perform parallel calculations of 12 input channels and 2 output channels when used for channel sliding pointwise convolution, and can perform parallel calculations of F(2x2, 3x3) when used for depthwise convolution (DWC), which is equivalent to unfolding the convolution kernel and 4 output pixels.

[0200] In some embodiments of the present invention, the cache module 3 includes:

[0201] The first cache module 31 is configured to retrieve feature data used by the accelerated computing module from memory.

[0202] In some embodiments of the present invention, the caching module includes:

[0203] The second cache module 32, configured to receive the calculation results from the accelerated computing module and send the calculation results to memory, is further configured to:

[0204] Receive the calculation results from the accelerated computing module, and provide corresponding intermediate data to the accelerated computing module according to the data access request from the accelerated computing module.

[0205] In some embodiments of the present invention, the caching module further includes:

[0206] The weight caching module 33 is configured to retrieve the convolution kernel data corresponding to each layer of the deep neural network from memory and send the convolution kernel data to the acceleration computing module.

[0207] In this embodiment, as Figure 6 As shown, the caching module 3 includes a first caching module 31, a second caching module 32, and a weight caching module 33. The first caching module 31 is used to retrieve the input feature data, i.e., the input feature image, from memory for subsequent convolution calculations.

[0208] The second cache module 32 is used to cache the output results of the accelerated computing module 2 and send the output results to memory. On the other hand, it is also used to cache the temporary results output by the accelerated computing module 2 and feed the temporary results back to the accelerated computing module 2 when necessary.

[0209] In some embodiments of the present invention, it further includes:

[0210] Direct memory access module 4 is configured to enable the cache module to quickly access data in memory.

[0211] In this embodiment, as Figure 6 As shown, the hardware acceleration circuit provided by this invention and the external data interaction are both implemented by the direct memory access module 4.

[0212] like Figure 11 As shown, another aspect of the present invention proposes a convolution operation acceleration system, comprising:

[0213] The kernel determination module 61 is configured to determine the channel sliding point-by-point convolution kernel and the corresponding sliding stride based on the number of channels.

[0214] A kernel splitting module 62 is configured to split the channel sliding point-by-point convolution kernel according to the sliding stride and a predetermined convolution acceleration algorithm.

[0215] The convolution operation acceleration module 63 is configured to accelerate the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension through the predetermined convolution acceleration algorithm.

[0216] like Figure 12 As shown, another aspect of the present invention also proposes an image processing method, comprising:

[0217] Step S21: Obtain the image to be processed;

[0218] Step S22: Input the image into a convolutional neural network model for image data processing, wherein the convolutional neural network model is accelerated by the convolution operation acceleration method described in the above embodiments.

[0219] In the embodiments of the present invention, the image processing method proposed in the present invention can be widely applied to various neural networks that require convolution calculations on images. That is, as long as there are convolution calculations in the corresponding network layers of the deep neural network, the convolution calculations in the network layers can be accelerated by the above-mentioned convolution acceleration algorithm.

[0220] It should be noted that the convolutional neural network model in step S22 does not refer solely to CNNs (Convolutional Neural Networks), but rather to neural networks that involve convolutional computations. Examples include image recognition models, image classification models, text-to-image generation models, and other models that involve convolutional computations on image data.

[0221] Several further embodiments are given below:

[0222] Taking the VGG16 algorithm as an example, this paper implements the convolution acceleration method proposed in this invention and accelerates the execution of the convolution acceleration method through hardware.

[0223] First, replace ordinary convolution with depthwise convolution and channel sliding pointwise convolution as described above; the determination of the channel sliding pointwise convolution kernel and stride can be made according to the above method.

[0224] Furthermore, the ordinary convolutions in the VGG16 network are replaced with depthwise convolutions and channel sliding pointwise convolutions. A comparison of ordinary convolutions, depthwise convolutions, and channel sliding pointwise convolutions is shown in the figure below. Figure 5 As shown, the kernel size of a regular convolution is k×k (k is generally greater than 1), which can extract spatial features in the spatial dimension. The kernel of a regular convolution fuses information from all input channels to obtain a single output channel, and different output channels use different kernels, which results in a large number of parameters and computational cost for regular convolution. Depthwise convolution has a kernel size of k×k, which can extract spatial features, but the channel dimension is 1. While depthwise convolution can extract spatial features, it cannot fuse channel features. Channel sliding pointwise convolution is used after depthwise convolution to fuse channel information. Channel sliding pointwise convolution differs from ordinary pointwise convolution (PWC). Both channel sliding pointwise convolution and PWC have a kernel size of 1×1 and are used for channel feature fusion. However, the channel dimension of the kernel in channel sliding pointwise convolution is smaller than that in PWC. Only a portion of the input channels are used to obtain an output channel, which reduces the number of parameters and computational cost. Furthermore, the input channels of adjacent output channels overlap, and through multiple convolutions, the information exchange between different input channels can be enhanced. The kernel of channel sliding pointwise convolution is shared across the channel dimension, which reduces a significant number of weight parameters. The number of parameters and computational cost of depthwise convolution and channel sliding pointwise convolution are far less than those of ordinary convolution.

[0225] Furthermore, the network is retrained using deep learning frameworks such as PyTorch and TensorFlow.

[0226] Furthermore, during the model inference phase, based on the channel-sliding pointwise convolution kernels and corresponding strides used during training, the model is decomposed in the manner described above. This decomposes the convolution operation of each layer into the computational form of the most basic operator F(2,3) of the 1D Winograd algorithm. Hardware acceleration is then used for the computation.

[0227] Decomposing the channel-sliding pointwise convolution kernel and its corresponding stride is to fuse more input channels for easier hardware acceleration and parallel computation. For example, the kernel size of a channel-sliding pointwise convolution is usually greater than 3, and the stride is not 1. To use F(2,3) to calculate the channel-sliding pointwise convolution, the channel-sliding pointwise convolution needs to be processed. A 1D convolution with 2 outputs, a kernel size of r, and a stride of d can be written as F(2,r,d). F(2,r,d) can be decomposed into d convolution kernels of size r / d, each with a stride of 1, i.e., d F(2,r / d). If r is large and d is small, r / d will be greater than 3. In this case, F(2,r / d) needs to be further decomposed into t convolution kernels of size 3, i.e., t F(2,3). The above two steps are decomposed into F(2, r, d) → d×F(2, r / d) → (r / 3)×F(2, 3). After the two-step decomposition, F(2, r, d) is decomposed into r / 3 F(2, 3). All F(2, 3) are calculated separately and the results are added together to obtain the final result.

[0228] For depthwise convolution, a commonly used kernel size is 3×3. It can be directly computed using the 2DWinograd fast convolution operator F(2×2, 3×3), written in matrix multiplication form as follows:

[0229] Y ;

[0230] A, G, and B are transformation matrices, which are the same as the transformation matrices of the 1D Winograd fast convolution operator.

[0231] A schematic diagram of the hardware acceleration circuit proposed in this invention is shown below. Figure 6 As shown, the convolutional computation engine 2 (equivalent to the accelerated computation module) is composed of PE (Processing Element) units in the FPGA. Each PE unit is individually implemented as the computational form of the most basic operator F(2,3) of the 1D Winograd algorithm. Both 1D and 2D Winograd can be computed through one or more PEs, thus depthwise convolution and sliding channel pointwise convolution share the same computational engine. The caching module 3 includes a weight caching module 33 for depthwise convolution and sliding channel convolution, a first caching module 31 for caching input feature maps, and a second caching module 32 for caching intermediate results or output feature maps. To reduce errors, the second caching module 32, which stores intermediate results, requires a wider bit width. The accelerator end implements logic control through a finite state machine 1. The image to be processed and the network weights are stored in memory. The processor sends a start signal to the accelerator end to start the accelerator and complete the entire inference process of the network.

[0232] Sliding channel convolution is essentially a 1D convolution in the channel dimension, while depthwise convolution is a 2D convolution in the spatial dimension. Traditional hardware accelerators supporting Winograd are designed for standard convolutions, resulting in only one type of convolution in the network and therefore only one computation engine. This invention focuses on two types of convolutions (depthwise convolution and channel sliding pointwise convolution). Designing two computation engines would lead to low utilization of computational resources. The computation engine proposed in this invention has two computation modes: the depthwise convolution Winograd computation mode and the channel sliding convolution Winograd computation mode. In other words, depthwise convolution and channel sliding convolution share the same computation engine.

[0233] The computing engine is composed of stacked computing units (PEs), and the structure of a single PE is as follows: Figure 8 As shown, a PE can compute a 1D F(2,3) Winograd convolution operator, where I0 to I3 are the four input elements (z) of F(2,3). The input elements must first undergo an input transformation (B). T The transformations are as follows: Tw0 to Tw3 are the transformed convolution kernel weights (Gq), and Y0 to Y3 are the results of element-wise multiplication. The final output is obtained after the output transformation. In an FPGA, one DSP can perform one multiplication and one addition. To fully utilize the multiplication-addition capabilities of DSPs (Digital Signal Processors), the input Winograd transformation and element-wise multiplication are performed together. This avoids using additional LUT resources to implement addition and subtraction in the input transformation. The convolution kernel transformation is not performed in the accelerator; the same weights are used for each inference. To avoid repeating the convolution kernel transformation for each inference, the transformation is performed before deployment and stored in memory. Four DSP resources in a single PE can complete the input transformation and element-wise multiplication of F(2,3). The output transformation requires additional LUT resources. For depthwise convolution F(2×2, 3×3), one PE can perform column transformation and element-wise multiplication of one column. A QPE (Quadruple Processing Element), consisting of four PEs, can complete the entire F(2×2, 3×3) calculation. After row and column transformations of the output, the final four (2×2) output elements are obtained.

[0234] Figure 8The PE shown can form the computational engine of an accelerator. The F(2×2, 3×3) Winograd convolution operator can accelerate the computation of 3×3 depthwise convolutions. After decomposing F(2, 48, 16) into 16 F(2, 3), the F(2, 3) Winograd convolution operator can compute channel-sliding pointwise convolutions. Figure 10 As shown, the accelerator's computing engine consists of 16 QPEs, each containing four PEs.

[0235] Furthermore, when calculating channel sliding convolution, one PE (one-dimensional computation unit 21) can compute one F(2,3) Winograd convolution operator, and one QPE (two-dimensional computation unit 22) can compute four F(2,3) Winograd convolution operators in parallel. The output of a QPE is the sum of the four PEs. One F(2,3) is equivalent to unfolding two output channels and three input channels. Therefore, one QPE is equivalent to unfolding two output channels and computing 12 input channels in parallel, resulting in a partial sum. The computation engine contains 16 QPEs, and the output channels of different QPEs are parallel. Therefore, the parallelism when calculating channel sliding convolution is 12 input channels and 32 output channels. Without the Winograd convolution operator, this parallel method requires 384 multiplications, while with Winograd, only 256 multiplications are required, resulting in a speedup of 1.5.

[0236] When computing depthwise convolution (DWC), a QPE consisting of four PEs can compute a 2D F(2×2, 3×3) Winograd convolution operator in parallel. This is equivalent to unfolding a 3×3 convolution kernel and 2×2 output pixels. One QPE can produce a complete result with 4 output pixels. The computation engine contains 16 QPEs, and the output channels of different QPEs are parallel. Therefore, the parallelism in computing DWC is unfolded to unfold 16 output channels, 2×2 output pixels, and a 3×3 convolution kernel. Without using the 2D F(2×2, 3×3) Winograd convolution operator, this parallel method requires 576 multiplications. After acceleration using the Winograd convolution operator, only 256 multiplications are required, resulting in a speedup of 2.25.

[0237] Depthwise convolution is computed using the 2D F(2×2, 3×3) Winograd operator. The input window size is 4×4, and each iteration produces a 2×2 output. Therefore, as... Figure 9 As shown, the interval between adjacent F(2×2, 3×3) windows is 2. Each time F(2×2, 3×3) is calculated, the input needs to undergo row and column transformations. The common practice is to perform row and column transformations separately for each sliding window. This can be observed... Figure 9We can observe that two rows overlap between adjacent windows. Before column transformation, these rows are independent and do not affect each other's transformed values. Therefore, after row transformation in the first window, the results of the last two rows can be retained. When calculating the second window, the first two rows are the last two rows of the first window. We can directly use the row transformation results of the last two rows temporarily stored during the calculation of the first window, performing row transformation only on the last two rows of the second window, without needing to perform row transformation on all four rows of the window. This reduces eight additions and subtractions. The subsequent windows are similar to the second window; only the last two rows need to be transformed, as the row transformation results of the first two rows have already been calculated by the previous window and can be used directly. This method avoids repetitive row transformation operations, reducing them by approximately half. Column transformation cannot be performed in this way; each window requires a complete column transformation because column transformation depends on the transformed row values. This calculation method of performing row transformation first and then column transformation reduces repetitive row transformation operations. Performing column transformation first and then row transformation reduces repetitive column transformation operations.

[0238] Furthermore, the final network output is obtained by sequentially calculating all convolutional layers of the neural network.

[0239] Finally, the depthwise convolution + channel sliding pointwise convolution method proposed in this invention replaced all convolutional layers in VGG16. Experimental results show that this method can reduce the number of parameters and computation by 90%, with an accuracy loss of only 2.2% on the ImageNet dataset. Then, the compressed VGG16 was implemented on a Xilinx Ultra96V2 FPGA development board using the hardware acceleration method proposed in this invention. Table 1 shows the comparison between the effects of this invention and existing technologies. Compared with existing technologies, this method adopts a hardware and software co-acceleration technology. First, it proposes a channel sliding pointwise convolution suitable for Winograd algorithm acceleration to compress the neural network. Then, it proposes a dedicated hardware acceleration method. Compared with existing technologies, it can process more images within a unit power consumption and has a higher energy efficiency ratio.

[0240] The comparison results are shown in the table below:

[0241]

[0242] This invention proposes a lightweight hardware-software co-acceleration method for convolutional neural networks based on the Winograd algorithm. It utilizes channel sliding pointwise convolution to compress ordinary pointwise convolution, and further accelerates channel sliding pointwise convolution using the 1D Winograd algorithm. Depthwise convolution can be accelerated using the 2D Winograd algorithm. Therefore, the compressed network as a whole can be accelerated using the Winograd algorithm. The hardware acceleration method proposed in this invention can efficiently compute 1D and 2D Winograd convolutions (depthwise convolution and channel sliding pointwise convolution) using the same computing engine, and reduces the number of computations by minimizing repeated row transformations. Finally, the hardware-software co-acceleration significantly improves the energy efficiency ratio.

[0243] like Figure 3 As shown, another aspect of the present invention also provides a computer device comprising:

[0244] At least one processor 2001; and

[0245] The memory 2002 stores computer instructions 2003 that can be executed on the processor 2001, which, when executed by the processor 2001, implement the steps of any of the methods described in the above embodiments.

[0246] like Figure 4 As shown, another aspect of the present invention provides a computer-readable storage medium 401 storing a computer program 402, which, when executed by a processor, implements the steps of any of the methods described in the above embodiments.

[0247] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0248] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0249] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0250] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0251] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of the different aspects of the invention as described above exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method for accelerating convolution operations, characterized in that, include: The sliding pointwise convolution kernel and the corresponding sliding stride are determined based on the number of channels. The channel sliding point-by-point convolution kernel is split according to the sliding stride and the predetermined convolution acceleration algorithm; The predetermined convolution acceleration algorithm is used to accelerate the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension. Among them, the ordinary convolution kernels used for multiple data channels are converted into depthwise convolution kernels and pointwise convolution kernels; Intermediate data with multiple data channels is generated based on the depth-wise convolution kernel; The first intermediate data allocation ratio of the point-to-point convolution kernel and the second intermediate data allocation ratio of the channel sliding point-to-point convolution kernel are set according to the minimum time for calculating the convolution data of the multiple data channels, the time consumed by calculating the point-to-point convolution kernel, and the time consumed by calculating the channel sliding point-to-point convolution kernel. Based on the pointwise convolution kernel, convolution calculation is performed on the intermediate data allocated to the first intermediate data. The predetermined convolution acceleration algorithm accelerates the sliding convolution calculation of the intermediate data allocated to the second intermediate data by the sliding point-by-point convolution kernel of the split channel; A one-dimensional computational unit is set according to the predetermined convolution acceleration algorithm; Combining several of the one-dimensional computing units into a two-dimensional computing unit; The intermediate data of multiple data channels is calculated by the two-dimensional computing unit based on the depthwise convolution kernel; The one-dimensional computing unit performs convolution calculations on the intermediate data allocated to the first intermediate data based on the pointwise convolution kernel.

2. The method according to claim 1, characterized in that, The step of splitting the channel sliding pointwise convolution kernel according to the sliding stride and the predetermined convolution acceleration algorithm includes: The channel sliding pointwise convolution kernel is split into convolution kernels that conform to the basic convolution operator of the predetermined convolution acceleration algorithm according to the sliding stride.

3. The method according to claim 2, characterized in that, The convolution kernel that splits the channel sliding point-by-point convolution kernel into basic convolution operators conforming to the predetermined convolution acceleration algorithm according to the sliding stride includes: Determine the dimension of the channel sliding pointwise convolution kernel; The dimension of the convolution kernel of the predetermined convolution acceleration algorithm is determined based on the dimension, and the characteristics of the basic convolution kernel of the predetermined convolution acceleration algorithm are determined. Multiple basic convolution kernels are formed by extracting elements from the channel sliding pointwise convolution kernel according to the sliding stride.

4. The method according to claim 3, characterized in that, Multiple basic convolutional kernels are formed by extracting elements from the channel sliding pointwise convolutional kernel according to the sliding stride, including: The value interval is determined by subtracting one from the sliding step size; Elements are extracted from the channel sliding point-by-point convolution kernel according to the value interval to form multiple basic convolution kernels.

5. The method according to claim 4, characterized in that, The step of extracting elements from the channel sliding point-by-point convolution kernel according to the value interval to form multiple basic convolution kernels includes: Determine whether the length of the first convolutional kernel, which is composed of elements extracted from the convolutional kernel by sliding the convolutional kernel point by point at the specified value interval, is greater than or less than the basic convolutional kernel; In response to the first convolutional kernel having a length greater than the basic convolutional kernel, the first convolutional kernel is split once or multiple times according to the size of the basic convolutional kernel until the size of the split convolutional kernel is the same as that of the basic convolutional kernel; or In response to the first convolution kernel having a length less than the base convolution kernel, the first convolution kernel is padded with a predetermined value so that its length is equal to the length of the base convolution kernel.

6. The method according to claim 1, characterized in that, The process of determining the channel-sliding pointwise convolution kernel and the corresponding sliding stride based on the number of channels includes: The channel sliding pointwise convolution kernel and its corresponding sliding stride are determined by comparing the results with those of a neural network model trained with a depthwise separable convolution kernel or by a neural network search algorithm.

7. The method according to claim 6, characterized in that, The determination of the channel-sliding pointwise convolutional kernel and its corresponding sliding stride through comparison with a neural network model trained with a depthwise separable convolutional kernel or through a neural network search algorithm includes: Randomly generate several channels of different lengths, slide point-by-point convolution kernels and sliding strides to train the neural network model; The scores of several neural network models corresponding to sliding pointwise convolution kernels of several different channel lengths are compared with the scores of neural network models trained with depthwise separable convolution kernels. Based on the comparison results, the channel sliding point-by-point convolution kernel and the corresponding sliding stride are determined.

8. The method according to claim 7, characterized in that, The process of determining the channel-sliding pointwise convolution kernel and the corresponding sliding stride based on the comparison results includes: Determine the score that is closest to or greater than the score of the neural network model trained with depth-separable convolutional kernels; The length and stride of the sliding pointwise convolution kernel corresponding to the determined score are used as the determined length and stride of the sliding pointwise convolution kernel.

9. The method according to claim 6, characterized in that, The determination of the channel-sliding pointwise convolutional kernel and its corresponding sliding stride through comparison with a neural network model trained with a depthwise separable convolutional kernel or through a neural network search algorithm includes: Based on the number of channels, multiple sliding point-by-point convolutional kernels and sliding strides of corresponding lengths are generated iteratively using a binary search method, and the model is trained using the multiple sliding point-by-point convolutional kernels and sliding strides of the channels. Based on the training results, the channel sliding pointwise convolution kernel and the corresponding sliding stride are determined.

10. The method according to claim 6, characterized in that, The determination of the channel-sliding pointwise convolutional kernel and its corresponding sliding stride through comparison with a neural network model trained with a depthwise separable convolutional kernel or through a neural network search algorithm includes: The length of the sliding pointwise convolution kernel and the sliding stride are obtained by using a neural architecture search algorithm and reinforcement learning.

11. The method according to claim 1, characterized in that, Also includes: Convert ordinary convolutional kernels used for multiple data channels into depthwise convolutional kernels; Intermediate data with multiple data channels is generated based on the depthwise convolution kernel.

12. The method according to claim 11, characterized in that, The sliding convolution calculation of the channel-wise sliding pointwise convolution kernel after being accelerated by the predetermined convolution acceleration algorithm includes: The predetermined convolution acceleration algorithm accelerates the sliding pointwise convolution kernel of the split channels to perform sliding convolution calculations on the intermediate data of the multiple data channels.

13. The method according to claim 1, characterized in that, The sliding convolution calculation of the channel-wise sliding pointwise convolution kernel after being accelerated by the predetermined convolution acceleration algorithm includes: The one-dimensional computing unit accelerates the sliding convolution calculation of the corresponding intermediate data by the sliding point-by-point convolution kernel of the channel after being split according to the predetermined convolution acceleration algorithm.

14. A hardware acceleration circuit implemented according to any one of claims 1-13, characterized in that, include: An accelerated computing module is configured to accelerate the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension using the predetermined convolution acceleration algorithm.

15. A convolution operation acceleration system, characterized in that, include: A kernel determination module is configured to determine the channel sliding point-by-point convolution kernel and the corresponding sliding stride based on the number of channels. A kernel splitting module is configured to split the channel sliding point-by-point convolution kernel according to the sliding stride and a predetermined convolution acceleration algorithm; A convolution operation acceleration module is configured to accelerate the sliding convolution calculation of the split channel sliding pointwise convolution kernel in the channel dimension through the predetermined convolution acceleration algorithm; wherein, the ordinary convolution kernel used for multiple data channels is converted into a depthwise convolution kernel and a pointwise convolution kernel; Intermediate data with multiple data channels is generated based on the depth-wise convolution kernel; The first intermediate data allocation ratio of the point-to-point convolution kernel and the second intermediate data allocation ratio of the channel sliding point-to-point convolution kernel are set according to the minimum time for calculating the convolution data of the multiple data channels, the time consumed by calculating the point-to-point convolution kernel, and the time consumed by calculating the channel sliding point-to-point convolution kernel. Based on the pointwise convolution kernel, convolution calculation is performed on the intermediate data allocated to the first intermediate data. The predetermined convolution acceleration algorithm accelerates the sliding convolution calculation of the intermediate data allocated to the second intermediate data by the sliding point-by-point convolution kernel of the split channel; A one-dimensional computational unit is set according to the predetermined convolution acceleration algorithm; Combining several of the one-dimensional computing units into a two-dimensional computing unit; The intermediate data of multiple data channels is calculated by the two-dimensional computing unit based on the depthwise convolution kernel; The one-dimensional computing unit performs convolution calculations on the intermediate data allocated to the first intermediate data based on the pointwise convolution kernel.

16. A computer device, characterized in that, include: At least one processor; as well as A memory storing computer instructions executable on the processor, which, when executed by the processor, implement the steps of the method according to any one of claims 1-13.

17. A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1-13.

18. An image processing method, characterized in that... include: Obtain the image to be processed; The image is input into a convolutional neural network model for image data processing, wherein the convolutional neural network model is accelerated by the convolution operation acceleration method according to any one of claims 1-13.

Images