Convolutional neural network acceleration method, apparatus, and system

Abstract

The application provides a convolutional neural network acceleration method, device and system, which comprises the following steps: obtaining a convolution result corresponding to a current convolution window; obtaining an intermediate pooling result corresponding to the current convolution window in the case that the current convolution window is separated from a previous first target convolution window by n convolution windows; determining the current convolution window as a second target convolution window and obtaining a pooling result corresponding to the current convolution window in the case that the current convolution window is separated from a previous second target convolution window by m convolution windows; the current convolution window and the previous first target convolution window are located in the same row, and the current convolution window and the previous second target convolution window are located in the same column; n and m are preset positive integers. The application provides a convolutional neural network acceleration method, device and system, which can perform pooling processing simultaneously when performing convolution calculation, can avoid occupying on-chip cache, and can improve the processing speed of the convolutional neural network.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a convolutional neural network acceleration method, device and system. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, artificial intelligence technology has been widely applied in image recognition, autonomous driving, natural language processing, computer vision and other fields. Convolutional neural network (CNN) is a kind of feedforward neural network, which is an important part of artificial intelligence technology.

[0003] Generally, the more parameters and the more complex the model structure of the convolutional neural network, the stronger the learning ability. However, complex network model, large amount of data movement, large amount of parameters and intensive calculation will slow down the calculation speed of the convolutional neural network, and bring great challenges to the hardware deployment of the convolutional neural network. Therefore, how to improve the processing speed of the convolutional neural network so as to realize the deployment of the convolutional neural network under the condition of low delay and low power consumption is a technical problem to be solved in the field. SUMMARY

[0004] The present application provides a convolutional neural network acceleration method, device and system to solve the defect that the convolutional neural network in the prior art is slow in calculation under the condition of more parameters and more complex model structure, and to realize the improvement of the processing speed of the convolutional neural network under the condition of more parameters and more complex model structure.

[0005] The present application provides a convolutional neural network acceleration method, comprising:

[0006] obtaining a convolution result corresponding to a current convolution window in a target image;

[0007] In the case that the current convolution window is separated from the last first target convolution window by n convolution windows, the current convolution window is determined as the first target convolution window, and based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows separated between the current convolution window and the last first target convolution window, an intermediate pooling result corresponding to the current convolution window is obtained;

[0008] In the case that the current convolution window is separated from the last second target convolution window by m convolution windows, the current convolution window is determined as the second target convolution window, and based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows separated between the current convolution window and the last second target convolution window, a pooling result corresponding to the current convolution window is obtained.

[0009] The current convolution window is located in the same row as the previous first target convolution window, and the current convolution window is located in the same column as the previous second target convolution window; n and m are preset positive integers.

[0010] According to the method, the intermediate pooling result corresponding to the current convolution window is obtained, and the current convolution window is determined as the second target convolution window under the condition that there are m convolution windows between the current convolution window and the previous second target convolution window, and the pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows between the current convolution window and the previous second target convolution window.

[0011] In a case where the current convolution window is an (n+1)th convolution window in a row, the current convolution window is determined as the first target convolution window, and the intermediate pooling result corresponding to the current convolution window is obtained based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the first n convolution windows in the row of the current convolution window.

[0012] According to the method, the intermediate pooling result corresponding to the current convolution window is obtained, and the current convolution window is determined as the second target convolution window under the condition that there are m convolution windows between the current convolution window and the previous second target convolution window, and the pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows between the current convolution window and the previous second target convolution window.

[0013] In a case where the current convolution window is an (m+1)th convolution window in a column, the current convolution window is determined as the second target convolution window, and the pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the first m convolution windows in the column of the current convolution window.

[0014] According to the method, the intermediate pooling result corresponding to the current convolution window is obtained, and the current convolution window is determined as the second target convolution window under the condition that there are m convolution windows between the current convolution window and the previous second target convolution window, and the pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows between the current convolution window and the previous second target convolution window.

[0015] In a case where the current convolution window is not determined as the second target convolution window, the intermediate pooling result corresponding to the current convolution window is cached.

[0016] According to the method, the current convolution window is determined as the second target convolution window under the condition that there are m convolution windows between the current convolution window and the previous second target convolution window, and the pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows between the current convolution window and the previous second target convolution window.

[0017] obtain the intermediate pooling result corresponding to the m convolution windows between the current convolution window and the previous second target convolution window from the cache;

[0018] obtain the pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows between the current convolution window and the previous second target convolution window;

[0019] In the case that the current convolution window is the (m+1)th convolution window in the column, the current convolution window is determined as the second target convolution window, and the pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the first m convolution windows in the column of the current convolution window, comprising:

[0020] obtain the intermediate pooling results corresponding to the first m convolution windows in the column of the current convolution window from the cache;

[0021] obtain the pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows between the current convolution window and the previous second target convolution window.

[0022] According to the convolution neural network acceleration method provided by the application, the intermediate pooling result corresponding to the current convolution window is obtained based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows between the current convolution window and the previous first target convolution window, comprising:

[0023] the maximum value of the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows between the current convolution window and the previous first target convolution window is taken as the intermediate pooling result corresponding to the current convolution window;

[0024] obtain the intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the first n convolution windows in the row of the current convolution window, comprising:

[0025] the maximum value of the convolution result corresponding to the current convolution window and the convolution results corresponding to the first n convolution windows in the row of the current convolution window is taken as the intermediate pooling result corresponding to the current convolution window.

[0026] According to the method for accelerating the convolutional neural network, the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows spaced between the current convolution window and the previous second target convolution window are used to obtain the pooling result corresponding to the current convolution window, including:

[0027] The maximum value in the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows spaced between the current convolution window and the previous second target convolution window is taken as the pooling result corresponding to the current convolution window.

[0028] The intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows in the column where the current convolution window is located are used to obtain the pooling result corresponding to the current convolution window, including:

[0029] The maximum value in the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows in the column where the current convolution window is located is taken as the pooling result corresponding to the current convolution window.

[0030] According to the method for accelerating the convolutional neural network, after the pooling result corresponding to the current convolution window is obtained, the method further includes:

[0031] The pooling result corresponding to the current convolution window is activated to obtain the activated pooling result corresponding to the current convolution window.

[0032] The application further provides a device for accelerating the convolutional neural network, including:

[0033] A data obtaining module is configured to obtain the convolution result corresponding to the current convolution window in a target image.

[0034] A first pooling module is configured to, in the case that the current convolution window is spaced n convolution windows from a previous first target convolution window, determine the current convolution window as the first target convolution window, and obtain the intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows spaced between the current convolution window and the previous first target convolution window.

[0035] a second pooling module configured to, in a case that the current convolution window is spaced apart from a previous second target convolution window by m convolution windows, determine the current convolution window as the second target convolution window, and obtain a pooling result corresponding to the current convolution window based on an intermediate pooling result corresponding to the current convolution window and intermediate pooling results corresponding to the m convolution windows spaced apart from the previous second target convolution window and the current convolution window;

[0036] wherein the current convolution window and the previous first target convolution window are located in a same row, and the current convolution window and the previous second target convolution window are located in a same column; n and m are preset positive integers.

[0037] The application further provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the convolutional neural network acceleration method according to any one of the preceding embodiments when executing the program.

[0038] The application further provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program is executable on a processor to implement the convolutional neural network acceleration method according to any one of the preceding embodiments.

[0039] The application further provides a computer program product comprising a computer program, wherein the computer program is executable on a processor to implement the convolutional neural network acceleration method according to any one of the preceding embodiments.

[0040] The convolutional neural network acceleration method, device, and system provided by the application can obtain a convolution result corresponding to a current convolution window in a target image, determine the current convolution window as a first target convolution window in a case that the current convolution window is spaced apart from a previous first target convolution window by n convolution windows, obtain an intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and convolution results corresponding to the n convolution windows spaced apart from the previous first target convolution window and the current convolution window, determine the current convolution window as a second target convolution window in a case that the current convolution window is spaced apart from a previous second target convolution window by m convolution windows, and obtain a pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and intermediate pooling results corresponding to the m convolution windows spaced apart from the previous second target convolution window and the current convolution window, wherein the current convolution window and the previous first target convolution window are located in a same row, and the current convolution window and the previous second target convolution window are located in a same column; n and m are preset positive integers. The method, device, and system can perform pooling processing simultaneously when performing convolution calculation, can avoid occupying on-chip cache, can reduce calculation power consumption, can reduce the time required for convolution calculation and pooling calculation, and can improve the processing speed of the convolutional neural network. BRIEF DESCRIPTION OF DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings needed in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0042] Figure 1 is a flow diagram of the convolutional neural network acceleration method provided by the present application;

[0043] Figure 2 is one of the schematic diagrams of the target image in the convolutional neural network acceleration method provided by the present application;

[0044] Figure 3 is the second schematic diagram of the target image in the convolutional neural network acceleration method provided by the present application;

[0045] Figure 4 is the third schematic diagram of the target image in the convolutional neural network acceleration method provided by the present application;

[0046] Figure 5 is the fourth schematic diagram of the target image in the convolutional neural network acceleration method provided by the present application;

[0047] Figure 6 is a structural schematic diagram of the convolutional neural network acceleration device provided by the present application;

[0048] Figure 7 is a structural schematic diagram of the convolutional neural network acceleration system provided by the present application;

[0049] Figure 8 is a schematic diagram of the storage hierarchy in the convolutional neural network acceleration system provided by the present application;

[0050] Figure 9 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0052] In the description of the invention, it should be noted that unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integral connection; it can be mechanical connection, or electrical connection; it can be direct connection, or indirect connection through intermediate medium, or internal communication of two elements. For those skilled in the art, the specific meaning of the above terms in the present invention can be understood according to the specific circumstances.

[0053] It should be noted that in the case of image recognition based on convolutional neural network, the traditional convolutional neural network can include multiple cascaded convolutional layers. After the original image is input into the first convolutional layer, the first convolutional layer can obtain and output the feature map of the original image by performing convolution calculation on the original image. By analogy, for any convolutional layer except the first convolutional layer, the input of the convolutional layer is the feature map output by the previous convolutional layer of the convolutional layer.

[0054] Generally, when the convolutional layer performs convolution calculation on the input image, the size of the convolution window is determined based on the size of the predetermined convolution kernel. When the convolutional layer performs convolution calculation on the input image, the convolution result corresponding to each convolution window is obtained as the pixel value of each pixel point in the feature map corresponding to the input image by sliding the convolution window according to the preset step size, and the feature map corresponding to the input image can be obtained. Among them, the sliding direction of the convolution window is usually from left to right along the row direction.

[0055] In order to improve the processing speed of the convolutional neural network, the feature map output by any convolutional layer can also be subjected to pooling processing, and the feature map subjected to the pooling processing is input into the next convolutional layer of the convolutional layer.

[0056] The traditional pooling processing method is to perform pooling processing on the feature map output by the convolutional layer after the convolutional layer completes all convolution calculation. After the convolutional layer outputs the feature map, the feature map needs to be stored in the on-chip cache first, and then the feature map is read from the on-chip cache for pooling processing.

[0057] Among them, if the width of the feature map is COL, the height is ROW, the number of channels is CH, and the bit width of each pixel point in the feature map is B, the capacity of the on-chip cache occupied by the feature map is COLxROWxCHxB.

[0058] In the case that the convolution layer performs convolution calculation based on a preset clock period, the convolution layer can obtain the convolution result corresponding to one convolution window in each clock period, and the convolution layer needs T = COLxROWxCH clock periods to complete all convolution calculation and output the feature map, and it also needs T clock periods to store the feature map to the on-chip cache. It needs at least T clock periods to read the feature map from the on-chip cache for pooling processing. Therefore, after the convolution layer completes all convolution calculation and outputs the feature map, the feature map needs to be processed by the pooling processing, and it needs at least 3T clock periods to complete the convolution calculation and the pooling processing.

[0059] In the case that the convolution neural network has a large number of parameters and a complex model structure, the above process occupies a large amount of on-chip cache, increases the time of forward inference, and affects the processing speed of the convolution neural network.

[0060] To this end, the present application provides a convolution neural network acceleration method. Based on the convolution neural network acceleration method provided by the present application, the pooling processing is not performed on the feature map output by the convolution layer, but the pooling processing is performed synchronously while the input image is subjected to convolution calculation, so that the capacity of the occupied on-chip cache can be reduced, the computing power consumption can be reduced, and the processing speed of the convolution neural network can be improved.

[0061] Figure 1 is a flowchart of the convolution neural network acceleration method provided by the present application. The convolution neural network acceleration method of the present application will be described below with reference to Figure 1 The convolution neural network acceleration method of the present application will be described below with reference to Figure 1 As shown in the figure, the method comprises the following steps: step 101, performing convolution calculation on a current convolution window in a target image to obtain a convolution result corresponding to the current convolution window.

[0062] Specifically, the target image is an image that needs to be subjected to convolution calculation and pooling processing at present. The target image can be an original image or a feature map obtained by convolution calculation.

[0063] In the present application, the size of the convolution window is determined based on the size of the convolution kernel when the target image is subjected to convolution calculation. In the following, the size of the convolution window is 3x3, and the sliding step of the convolution window is 1, which is used to illustrate the convolution neural network acceleration method provided by the present application.

[0064] In the present application, the current convolution window in the target image can be subjected to convolution calculation based on the conventional convolution calculation method, and the convolution result corresponding to the current convolution window can be obtained through data interaction and the like.

[0065] It should be noted that the current convolution window can be located at any position in the target image.

[0066] It should be noted that the convolution window slides from left to right along the direction of the row in the embodiment of the present application, and after the sliding of the current row is completed, it jumps to the leftmost end of the next row of the current row to start the sliding of the next row. The starting position of the sliding of the convolution window is the top left corner of the target image, and the ending position of the sliding of the convolution window is the bottom right corner of the target image.

[0067] In the case that the current convolution window is spaced apart from the last first target convolution window by n convolution windows, the current convolution window is determined as the first target convolution window, and an intermediate pooling result corresponding to the current convolution window is obtained based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows spaced apart from the last first target convolution window.

[0068] In the case that the current convolution window is spaced apart from the last first target convolution window by n convolution windows, the current convolution window is determined as the first target convolution window, and an intermediate pooling result corresponding to the current convolution window is obtained based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows spaced apart from the last first target convolution window.

[0069] It should be noted that the pooling step and the size of the pooling window for performing the pooling processing on the target image can be determined according to actual needs. According to the pooling step and the size of the pooling window, the values of n and m can be determined. For example, in the case that the pooling step is 2 and the size of the pooling window is 2x2, the values of n and m are both 1; in the case that the pooling step is 3 and the size of the pooling window is 3x3, the values of n and m are both 2. Hereinafter, the convolution neural network acceleration method provided by the present application is described by taking the pooling step as 2 and the size of the pooling window as 2x2 as an example.

[0070] Specifically, after obtaining the convolution result corresponding to the current convolution window, it can be judged whether the number of the convolution windows spaced apart from the last first target convolution window in the row where the current convolution window is located is 1.

[0071] Figure 2 is one of the schematic diagrams of the target image in the convolution neural network acceleration method provided by the present application. As shown in Figure 2 As the size of the convolution window is 3x3, 3x3 pixel points are included in the current convolution window. If the current convolution window is spaced apart from the last first target convolution window by 1 convolution window, the current convolution window can be determined as the first target convolution window.

[0072] It should be noted that after the convolution result corresponding to the current convolution window is obtained, the convolution window can be shifted right by 1 step to determine the next convolution window of the current convolution window. Since the current convolution window is determined as the first target convolution window, the next convolution window of the current convolution window is spaced apart from the current convolution window by 0 convolution windows, and the next convolution window of the current convolution window cannot be determined as the first target convolution window. By analogy, in the case that the current convolution window is the 2nd, 4th, 6th, etc. even-numbered convolution window of the row, the current convolution window can be determined as the first target convolution window. Correspondingly, in the case that the current convolution window is the 1st, 3rd, 5th, etc. odd-numbered convolution window of the row, the current convolution window cannot be determined as the first target convolution window.

[0073] After determining that the current convolution window is the first target convolution window, the intermediate pooling result corresponding to the current convolution window can be obtained by numerical calculation based on the convolution result corresponding to the current convolution window and the convolution result corresponding to the 1 convolution window spaced apart between the current convolution window and the previous first target convolution window.

[0074] It should be noted that only in the case that the current convolution window is determined as the first target convolution window, the intermediate pooling result corresponding to the current convolution window is obtained; in the case that the current convolution window is not the first target convolution window, only the convolution result corresponding to the current convolution window needs to be obtained.

[0075] In step 103, in the case that the current convolution window is spaced apart from the previous second target convolution window by m convolution windows, the current convolution window is determined as the second target convolution window, and the pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows spaced apart between the current convolution window and the previous second target convolution window.

[0076] Wherein, the current convolution window and the previous second target convolution window are located in the same column.

[0077] Specifically, after the current convolution window is determined as the first target convolution window and the intermediate pooling result corresponding to the current convolution window is obtained, it can be judged whether the number of convolution windows spaced apart between the current convolution window and the previous second target convolution window in the column where the current convolution window is located is 1.

[0078] Figure 2 is a schematic diagram of a target image in the convolutional neural network acceleration method provided by the application. As shown in Figure 2 If the current convolution window is spaced apart from the previous second target convolution window by 1 convolution window, the current convolution window can be determined as the second target convolution window.

[0079] It should be noted that the current convolution window can be determined as the second target convolution window according to the number of convolution windows spaced between the current convolution window and the previous second target convolution window on the premise that the current convolution window is determined as the first target convolution window. That is, if the current convolution window is the even-numbered convolution window in the even-numbered row, the current convolution window can be determined as the first target convolution window and the second target convolution window. If the current convolution window is the even-numbered convolution window in the odd-numbered row, the current convolution window can be determined as the first target convolution window but cannot be determined as the second target convolution window. If the current convolution window is the odd-numbered convolution window in the odd-numbered row, the current convolution window cannot be determined as the first target convolution window and the second target convolution window.

[0080] After determining the current convolution window as the second target convolution window, the pooling result corresponding to the current convolution window can be obtained through numerical calculation based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the convolution window spaced between the current convolution window and the previous second target convolution window.

[0081] It should be noted that the pooling result corresponding to the current convolution window is obtained only when the current convolution window is determined as the second target convolution window; and the intermediate pooling result corresponding to the current convolution window is obtained only when the current convolution window is not the second target convolution window but is the first target convolution window.

[0082] Optionally, in the case of obtaining the convolution result corresponding to the current convolution window according to the preset clock period, the convolution result corresponding to the current convolution window can be obtained in the current clock period. In the next clock period of the current clock period, the convolution result of the next convolution window of the current convolution window can be obtained, and it is determined whether the current convolution window can be determined as the first target convolution window. In the case that the current convolution window can be determined as the first target convolution window, the intermediate pooling result of the current convolution window is obtained, and it is determined whether the current convolution window can be determined as the second target convolution window. In the case that the current convolution window can be determined as the second target convolution window, the pooling result of the current convolution window is obtained. Thus, the pooling result of the current convolution window is obtained in the next clock period of the current clock period, and the convolution calculation and the pooling processing are simultaneously performed.

[0083] If the current convolution window is the last convolution window in the target image, the convolution calculation and the pooling processing of the target image can be completed in the next clock cycle of the current clock cycle, and the feature map corresponding to the target image after the pooling processing is obtained. Therefore, the convolution calculation and the pooling processing of the target image can be completed in T+1 clock cycles based on the convolution neural network acceleration method provided by the application, which is much less than 3T clock cycles, and the processing efficiency of the convolution neural network is improved.

[0084] It should be noted that in the case that the sliding direction of the convolution window is generally along the column direction, the current convolution window and the last first target convolution window are located in the same column, and the current convolution window and the last second target convolution window are located in the same row.

[0085] In the embodiment of the application, after the convolution result corresponding to the current convolution window in the target image is obtained, the current convolution window is determined as the first target convolution window in the case that the current convolution window and the last first target convolution window are separated by n convolution windows, and the intermediate pooling result corresponding to the current convolution window is obtained based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows separated between the current convolution window and the last first target convolution window, and the current convolution window is determined as the second target convolution window in the case that the current convolution window and the last second target convolution window are separated by m convolution windows, and the pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows separated between the current convolution window and the last second target convolution window, wherein the current convolution window and the last first target convolution window are located in the same row, and the current convolution window and the last second target convolution window are located in the same column; n and m are preset positive integers, which can perform the pooling processing synchronously when the convolution calculation is performed, can avoid the occupation of the on-chip cache, can reduce the calculation power consumption, can reduce the time required for the convolution calculation and the pooling calculation, and can improve the processing speed of the convolution neural network.

[0086] Based on the content of the above embodiments, in the case that the current convolution window is the (n+1)th convolution window in the row, the current convolution window is determined as the first target convolution window, and the intermediate pooling result corresponding to the current convolution window is obtained based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the first n convolution windows in the row of the current convolution window.

[0087] Figure 3 is the second schematic diagram of the target image in the convolution neural network acceleration method provided by the application. As shown in FIG. 2, the target image is divided into a plurality of convolution windows, and the current convolution window is the (n+1)th convolution window in the row.Figure 3 As shown in the figure, if the current convolution window is the second convolution window in the row, it indicates that the row where the current convolution window is located has not determined the first target window, and the current convolution window can be determined as the first first target window in the row.

[0088] After the current convolution window is determined as the first target window, the intermediate pooling result corresponding to the current convolution window can be obtained through numerical calculation based on the convolution result of the current convolution window and the convolution result corresponding to the first convolution window in the row where the current convolution window is located.

[0089] In the case that the current convolution window is the first target convolution window in the row, the intermediate pooling result corresponding to the current convolution window can be more accurately and efficiently obtained by determining the current convolution window as the first first target convolution window in the row in the case that the current convolution window is the n+1th convolution window in the row, and obtaining the intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the first n convolution windows in the row where the current convolution window is located.

[0090] Based on the above-mentioned embodiments, the intermediate pooling result corresponding to the current convolution window is obtained, and the method further comprises: in the case that the current convolution window is the m+1th convolution window in the column, determining the current convolution window as the second target convolution window, and obtaining the pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the first m convolution windows in the column where the current convolution window is located.

[0091] Figure 4 Figure 3 is a schematic diagram of a target image in the method for accelerating a convolutional neural network provided by the application. Figure 4 As shown in the figure, if the current convolution window is the second convolution window in the column, it indicates that the column where the current convolution window is located has not determined the second target window, and the current convolution window can be determined as the first second target window in the column.

[0092] After the current convolution window is determined as the second target window, the pooling result corresponding to the current convolution window can be obtained through numerical calculation based on the intermediate pooling result of the current convolution window and the intermediate pooling result corresponding to the first convolution window in the column where the current convolution window is located.

[0093] Figure 5 Figure 4 is a schematic diagram of a target image in the method for accelerating a convolutional neural network provided by the application. Figure 5As shown, if the current convolution window is the second convolution window in the column and the current convolution window is the second convolution window in the row, it indicates that the row where the current convolution window is located has not determined the first target window, and the column where the current convolution window is located has not determined the second target window, the current convolution window can be determined as the first first target window in the row and the first second target window in the column. The intermediate pooling result corresponding to the current convolution window can be obtained through numerical calculation based on the convolution result of the current convolution window and the convolution result corresponding to the first convolution window in the row where the current convolution window is located. The pooling result corresponding to the current convolution window is obtained through numerical calculation based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the first convolution window in the column where the current convolution window is located.

[0094] In the case that the current convolution window is the first target convolution window in the column, the intermediate pooling result corresponding to the current convolution window is obtained through numerical calculation based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the first convolution window in the column where the current convolution window is located, and the pooling result corresponding to the current convolution window is obtained through numerical calculation based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the first convolution window in the column where the current convolution window is located, which can more accurately and efficiently obtain the pooling result corresponding to the current convolution window in the case that the current convolution window is the first target convolution window in the column.

[0095] Based on the above-mentioned embodiments, after obtaining the intermediate pooling result corresponding to the current convolution window, the intermediate pooling result corresponding to the current convolution window is cached in the case that the current convolution window is not determined as the second target convolution window.

[0096] Specifically, in the case that the number of convolution windows between the current convolution window and the last second target convolution window is less than m, or the current convolution window is the first m convolution windows in the column, the current convolution window is not determined as the second target convolution window.

[0097] In the case that the current convolution window is not determined as the second target convolution window, after obtaining the intermediate pooling result corresponding to the current convolution window, the intermediate pooling result corresponding to the current convolution window can be cached.

[0098] Optionally, in the case that the current convolution window is not determined as the second target convolution window, after obtaining the intermediate pooling result corresponding to the current convolution window, the intermediate pooling result corresponding to the current convolution window can be cached to a FIFO (First Input First Output) memory.

[0099] It should be noted that the FIFO memory is a first-in first-out dual-port buffer, that is, the first data entering the FIFO memory is the first data removed from the FIFO memory, one port of the FIFO memory is an input port of the FIFO memory, and the other port is an output port of the FIFO memory. The FIFO memory gradually obtains more and more extensive applications in high-speed data acquisition, high-speed data processing, high-speed data transmission, and multi-machine processing systems due to its flexibility, convenience, and high efficiency.

[0100] It should be noted that, with the translation of the convolution window, the cache can store the intermediate pooling results corresponding to the convolution windows determined as the first target convolution window but not determined as the second target convolution window.

[0101] Based on the content of the above embodiments, in the case that the current convolution window is separated from the last second target convolution window by m convolution windows, the current convolution window is determined as the second target convolution window, and the pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows separated from the last second target convolution window, including:

[0102] The intermediate pooling results corresponding to the m convolution windows separated from the last second target convolution window are obtained from the cache.

[0103] The pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows separated from the last second target convolution window.

[0104] Specifically, after the current convolution window is determined as the second target convolution window, the intermediate pooling result corresponding to the convolution window separated from the last second target convolution window by 1 can be read from the cache, and the pooling result corresponding to the current convolution window can be obtained by numerical calculation based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the convolution window separated from the last second target convolution window by 1.

[0105] Based on the content of the above embodiments, in the case that the current convolution window is the m+1th convolution window in the column, the current convolution window is determined as the second target convolution window, and the pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the first m convolution windows in the column of the current convolution window, including: the intermediate pooling results corresponding to the first m convolution windows in the column of the current convolution window are obtained from the cache.

[0106] The intermediate pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows between the current convolution window and the last second target convolution window.

[0107] After the current convolution window is determined as the second target convolution window, the intermediate pooling result corresponding to the first convolution window in the column where the current convolution window is located can be read from the cache, and the pooling result corresponding to the current convolution window can be obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the first convolution window in the column where the current convolution window is located.

[0108] Optionally, after the current convolution window is determined as the second target convolution window, the intermediate pooling result corresponding to the one convolution window between the current convolution window and the last second target convolution window can be read from the FIFO memory, or the intermediate pooling result corresponding to the first convolution window in the column where the current convolution window is located can be read from the FIFO memory.

[0109] In the case where the current convolution window is not determined as the second target convolution window, the intermediate pooling result corresponding to the current convolution window is cached, which can reduce the capacity of the on-chip cache, reduce the computing power consumption, and improve the processing speed of the convolutional neural network.

[0110] Based on the convolution result corresponding to the current convolution window and the convolution result corresponding to the n convolution windows between the current convolution window and the last first target convolution window, the intermediate pooling result corresponding to the current convolution window is obtained, including: taking the maximum value in the convolution result corresponding to the current convolution window and the convolution result corresponding to the n convolution windows between the current convolution window and the last first target convolution window as the intermediate pooling result corresponding to the current convolution window.

[0111] Specifically, after the current convolution window is determined as the first target convolution window, the larger one of the convolution result corresponding to the current convolution window and the convolution result corresponding to the one convolution window between the current convolution window and the last first target convolution window is taken as the intermediate pooling result corresponding to the current convolution window.

[0112] Based on the convolution result corresponding to the current convolution window and the convolution result corresponding to the n convolution windows in the row where the current convolution window is located, the intermediate pooling result corresponding to the current convolution window is obtained, including: taking the maximum value in the convolution result corresponding to the current convolution window and the convolution result corresponding to the n convolution windows in the row where the current convolution window is located as the intermediate pooling result corresponding to the current convolution window.

[0113] Specifically, after the current convolution window is determined as the first target convolution window, the larger one of the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the first convolution window in the row where the current convolution window is located can be taken as the intermediate pooling result corresponding to the current convolution window.

[0114] Based on the above embodiments, the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows between the current convolution window and the last second target convolution window are used to obtain the pooling result corresponding to the current convolution window, which includes: taking the maximum value of the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows between the current convolution window and the last second target convolution window as the pooling result corresponding to the current convolution window.

[0115] Specifically, after the current convolution window is determined as the second target convolution window, the larger one of the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the first convolution window in the column where the current convolution window is located can be taken as the pooling result corresponding to the current convolution window.

[0116] Based on the above embodiments, the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows between the current convolution window and the last second target convolution window are used to obtain the pooling result corresponding to the current convolution window, which includes: taking the maximum value of the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows between the current convolution window and the last second target convolution window as the pooling result corresponding to the current convolution window.

[0117] Specifically, after the current convolution window is determined as the second target convolution window, the larger one of the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the first convolution window in the column where the current convolution window is located can be taken as the pooling result corresponding to the current convolution window.

[0118] Based on the above embodiments, after the pooling result corresponding to the current convolution window is obtained, the pooling result corresponding to the current convolution window is activated to obtain the activated pooling result corresponding to the current convolution window.

[0119] Generally, the activation processing is performed before the pooling processing. For a target image with a width of COL, a height of ROW, and a channel number of CH, if the activation processing is performed before the pooling processing, the activation processing needs to be performed on each pixel point in the feature map, i.e., COL×ROW×CH times of activation processing are needed.

[0120] The activation processing in the embodiment of the present application is performed after the pooling processing, and the activation processing is performed on the pooling result corresponding to the current convolution window after the pooling result corresponding to the current convolution window is obtained. Since the size of the feature map after the pooling is reduced to (COL / 2) x (ROW / 2) x CH, if the activation processing is performed after the pooling processing, only (COL / 2) x (ROW / 2) x CH times of activation processing is needed. The number of times of activation processing is only one quarter of the original, that is, only one quarter of the original calculation amount is needed to complete the activation processing of the target image, thereby reducing the calculation power consumption.

[0121] In addition, the traditional activation processing mode is that after the convolution layer completes all convolution calculation and outputs a feature map, the feature map is subjected to the pooling processing to obtain a feature map after the pooling, and then the feature map after the pooling is subjected to the activation processing. In the case that the convolution layer performs the convolution calculation based on a preset clock period, at least T clock periods are needed to complete the activation processing of the feature map after the pooling. Therefore, after the convolution layer completes all convolution calculation and outputs a feature map, the feature map is subjected to the pooling processing and the activation processing, and at least 4T clock periods are needed to complete the convolution calculation, the pooling processing and the activation processing.

[0122] In the embodiment of the present application, after the pooling result corresponding to the current convolution window is obtained, the pooling result corresponding to the current convolution window is activated to obtain the activation pooling result corresponding to the current convolution window, so that the pooling processing and the activation processing can be performed synchronously when the convolution calculation is performed.

[0123] Optionally, in the case that the convolution result corresponding to the current convolution window is obtained according to a preset clock period, in the current clock period, the convolution result corresponding to the current convolution window can be obtained, in the next clock period of the current clock period, the pooling result corresponding to the current convolution window can be obtained, and in the next two clock periods of the current clock period, the activation pooling result corresponding to the current convolution window can be obtained.

[0124] If the current convolution window is the last convolution window in the target image, the convolution calculation, the pooling processing and the activation processing of the target image can be completed in the next two clock periods of the current clock period, and the feature map after the pooling and the activation processing corresponding to the target image is obtained. Therefore, the convolution calculation and the pooling processing of the target image can be completed only by T+2 clock periods based on the convolution neural network acceleration method provided by the present application, which is much smaller than 4T clock periods, thereby improving the processing efficiency of the convolution neural network.

[0125] Optionally, the Relu function can be used to perform the activation processing on the pooling result corresponding to the current convolution window.

[0126] The embodiment of the present application can further reduce the time required for convolution calculation and pooling calculation, and further improve the processing speed of the convolutional neural network, by synchronously performing the pooling processing and the activation processing when performing the convolution calculation, and further reducing the calculation power consumption by performing the activation processing after the pooling processing.

[0127] Figure 6 is a structural schematic diagram of the convolutional neural network acceleration device provided by the present application. The following describes the convolutional neural network acceleration device provided by the present application, which can be correspondingly referred to the convolutional neural network acceleration method provided by the present application described above. As shown in the figure, Figure 6 The convolutional neural network acceleration device provided by the present application is described below, which can be correspondingly referred to the convolutional neural network acceleration method provided by the present application described above. As shown in the figure, Figure 6 The device comprises a data acquisition module 601, a first pooling module 602 and a second pooling module 603.

[0128] The data acquisition module 601 is configured to acquire a convolution result corresponding to a current convolution window in a target image.

[0129] The first pooling module 602 is configured to, in the case that the current convolution window is separated from a previous first target convolution window by n convolution windows, determine the current convolution window as the first target convolution window, and acquire an intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows separated between the current convolution window and the previous first target convolution window.

[0130] The second pooling module 603 is configured to, in the case that the current convolution window is separated from a previous second target convolution window by m convolution windows, determine the current convolution window as the second target convolution window, and acquire a pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows separated between the current convolution window and the previous second target convolution window.

[0131] Wherein, the current convolution window and the previous first target convolution window are located in the same row, and the current convolution window and the previous second target convolution window are located in the same column; n and m are preset positive integers.

[0132] Specifically, the data acquisition module 601, the first pooling module 602 and the second pooling module 603 are electrically connected.

[0133] Optionally, the convolutional neural network acceleration device further comprises an activation module.

[0134] The activation module can be used to activate the pooling result corresponding to the current convolution window, thereby obtaining the activation pooling result corresponding to the current convolution window.

[0135] In this embodiment of the invention, after obtaining the convolution result corresponding to the current convolution window in the target image, if there is an interval of n convolution windows between the current convolution window and the previous first target convolution window, the current convolution window is determined as the first target convolution window. Based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows between the current convolution window and the previous first target convolution window, the intermediate pooling result corresponding to the current convolution window is obtained. If there is an interval of m convolution windows between the current convolution window and the previous second target convolution window, the current convolution window is determined as the second target convolution window. Based on the intermediate pooling results corresponding to the current convolutional window and the intermediate pooling results corresponding to the m convolutional windows between the current convolutional window and the previous second target convolutional window, the pooling result corresponding to the current convolutional window is obtained. Here, the current convolutional window and the previous first target convolutional window are in the same row, and the current convolutional window and the previous second target convolutional window are in the same column. n and m are preset positive integers. Pooling processing can be performed simultaneously during convolution calculation, which can avoid occupying on-chip cache, reduce computing power consumption, reduce the time required for convolution calculation and pooling calculation, and improve the processing speed of convolutional neural networks.

[0136] Figure 7 This is a schematic diagram of the convolutional neural network acceleration system provided by the present invention. The following is in conjunction with… Figure 7 The convolutional neural network acceleration system provided by this invention will be described. For example... Figure 7 As shown, the system includes: a weight parameter cache module 701, an input / output cache module 702, a general computing module 703, and a computing control module 704; the computing control module includes the convolutional neural network acceleration device 705 as described above.

[0137] The weight parameter cache module 701 is connected to the external DDR 706, the general computing module 703, and the computing control module 704, respectively; the input / output cache module 702 is connected to the external DDR 706, the general computing module 703, and the computing control module 704, respectively; the general computing module 703 and the computing control module 704 are connected.

[0138] It should be noted that the traditional general-purpose processor CPU is in a serial computing manner to process data, and cannot efficiently implement parallel computing, so it is very inefficient to deploy the convolutional neural network on the CPU. The graphics processing unit GPU has a highly parallel structure and has a natural advantage in parallel computing of a large amount of data, but the GPU generates extremely high power consumption in the inference process. The field programmable gate array FPGA has the characteristics of programmability, low power consumption, computing parallelism and short development cycle, and has become the mainstream hardware for convolutional neural network acceleration.

[0139] Due to a large number of weight parameters in the convolutional neural network, all of which cannot be stored in the on-chip storage of the FPGA. Usually, the weight parameters are stored in the off-chip memory, and the weight parameters needed in the calculation are read into the FPGA on-chip, but the limitation of the off-chip memory bandwidth will inevitably affect the processing efficiency of the convolutional neural network.

[0140] To this end, the present application provides a convolutional neural network acceleration system. As shown in Figure 7 The convolutional neural network acceleration system provided by the present application comprises four modules.

[0141] The weight parameter cache module 701 is connected with the off-chip DDR 706 through an AXI4 bus master interface. The AXI4 bus master interface can read the weight parameters needed for convolution calculation from the off-chip DDR 706, and store the read weight parameters in the weight parameter cache.

[0142] The weight parameter cache module 701 can store the weight parameters read from the off-chip DDR 706, and send the weight parameters to the weight parameter register unit 711 in the general computing module 703.

[0143] The two cache units in the input-output cache module 702 adopt a ping-pong cache structure, and the above two cache units are used as the input feature cache unit 707 and the output feature cache unit 708 in turn, for storing the image data of the target image read from the off-chip DDR 706 and the convolution result obtained by convolution calculation.

[0144] The general computing module 703 comprises a feature data register unit 709, a multiply-accumulate computing unit 710 and a weight parameter register unit 711, which can be used for multiply-accumulate operation on the image data from the input-output cache module 702 and the weight parameters from the weight parameter cache module 701, and send the convolution result to the computing control module 704.

[0145] The computing control module 704 includes a convolutional neural network acceleration device 705 and a control logic unit 712, which can be used to generate read-write addresses of the input-output cache module 702, perform pooling and activation operations on the convolution results from the general computing module 703, and write the processed data into the input-output cache module 702. In addition, the computing control module 704 can also control the input-output cache module 702 to read the weight parameters from the off-chip DDR 706.

[0146] The workflow of the convolutional neural network acceleration system is as follows:

[0147] Step S1, load the target image from the off-chip DDR 706 to the input feature cache unit, and read the weight parameters for the first round of calculation from the off-chip DDR 706 to the weight parameter cache module 701.

[0148] Step S2, start the current round of convolution calculation, the computing control module 704 sends a read command to the input feature cache unit 707 to store the image data corresponding to the current convolution window to the feature data register unit 709 in the general computing module 703. The computing control module 704 sends a read command to the weight parameter cache module 701 to store the weight parameters to the weight parameter register unit 711 in the general computing module 703.

[0149] Step S3, the multiply-accumulate computing unit 710 reads the image data corresponding to the current convolution window from the feature data register unit 709 and reads the weight parameters from the weight parameter register unit 711, and then performs multiplication and accumulation operation on the image data corresponding to the current convolution window and the weight parameters to obtain the convolution result corresponding to the current convolution window; during the calculation, the feature data register unit 709 needs to be updated as the convolution window moves, but the weight parameter cache module 701 does not need to be updated.

[0150] Step S4, while the general computing module 703 is calculating, read the weight parameters required for the next round of convolution calculation into the weight parameter cache module 701.

[0151] Step S5, the general computing module 703 outputs the convolution result corresponding to the current convolution window to the computing control module 704, and the convolutional neural network acceleration device 705 in the computing control module 704 performs pooling and activation processing on the convolution result corresponding to the current convolution window, and then writes the activation pooling result corresponding to the current convolution window to the output feature cache unit 708.

[0152] Step S6, after the current round of convolution calculation is completed and the weight parameters for the next round of convolution calculation are loaded, the next round of calculation is started.

[0153] Step S7, repeating steps S2 to S6 until the convolution calculation, pooling and activation processing results of the target image are completed, and the forward inference process of the target image is completed.

[0154] It should be noted that the number of rows of the target image can be recorded as ROW+K-1, the number of columns as COL+K-1, and the number of channels as A. A convolution kernel with B rows, K columns, and A channels is used to perform convolution operation on the input feature map. The resulting feature map corresponding to the target image has ROW rows, COL columns, and A channels. In general, in the hardware acceleration of the convolutional neural network, considering the computing resources of the hardware, all B channels of all A convolution kernels are not directly used for calculation, but are divided into several rounds for convolution calculation. In each round of calculation, a(b<B) channels of a(a<A) convolution kernels are used to perform convolution operation with the corresponding b channels of the input feature map.

[0155] It should be noted that the specific structure and working steps of the convolutional neural network acceleration device can refer to the content of the above embodiments, which will not be repeated here.

[0156] Figure 8 is a schematic diagram of the storage hierarchy in the convolutional neural network acceleration system provided by the present application. As shown in Figure 8 The weight parameter cache module 701 is used to store the weight parameters read from the off-chip DDR 706 memory. When the weight parameters used for the current convolution operation are read, the weight parameters are read into the weight parameter register unit 701. When all the weight parameters are read into the weight parameter register unit 701, the convolution operation starts, and the weight parameters used for the next round of convolution operation are read from the off-chip DDR 706 memory into the weight parameter cache module 701.

[0157] The convolutional neural network acceleration system in the embodiment of the present application hides the limitation of memory bandwidth on acceleration performance by designing the storage hierarchy structure. Through the convolutional neural network acceleration device, pooling and activation processing can be performed simultaneously during convolution calculation. By exchanging the order of activation processing and pooling processing, the number of activation processing can be reduced, the calculation power consumption can be reduced, the time required for convolution calculation and pooling calculation can be reduced, and the processing speed of the convolutional neural network can be improved.

[0158] Figure 9 An example of an electronic device entity structure schematic diagram is shown in Figure 9As shown, the electronic device can include a processor 910, a communications interface 920, a memory 930, and a communications bus 940, wherein the processor 910, the communications interface 920, and the memory 930 complete mutual communication through the communications bus 940. The processor 910 can invoke a logical instruction in the memory 930 to execute a convolutional neural network acceleration method, which includes: obtaining a convolution result corresponding to a current convolution window in a target image; in the case that the current convolution window is separated from a previous first target convolution window by n convolution windows, determining the current convolution window as the first target convolution window, and obtaining an intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows separated between the current convolution window and the previous first target convolution window; in the case that the current convolution window is separated from a previous second target convolution window by m convolution windows, determining the current convolution window as the second target convolution window, and obtaining a pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows separated between the current convolution window and the previous second target convolution window; wherein the current convolution window and the previous first target convolution window are located in the same row, and the current convolution window and the previous second target convolution window are located in the same column; n and m are preset positive integers.

[0159] In addition, the logical instruction in the memory 930 described above can be implemented in the form of a software function unit and sold or used as an independent product, which can be stored in a computer-readable storage medium. Based on such understanding, the technical solutions of the present application essentially or partly or parts of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0160] In another aspect, the present application also provides a computer program product comprising a computer program, which can be stored on a non-transitory computer readable storage medium, and the computer program, when executed by a processor, enables a computer to perform the convolutional neural network acceleration method provided by any of the above methods, which comprises: obtaining a convolution result corresponding to a current convolution window in a target image; in the case that the current convolution window is separated from a previous first target convolution window by n convolution windows, determining the current convolution window as the first target convolution window, and obtaining an intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows separated from the previous first target convolution window; in the case that the current convolution window is separated from a previous second target convolution window by m convolution windows, determining the current convolution window as the second target convolution window, and obtaining a pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows separated from the previous second target convolution window; wherein the current convolution window and the previous first target convolution window are located in the same row, and the current convolution window and the previous second target convolution window are located in the same column; n and m are preset positive integers.

[0161] In another aspect, the present application also provides a non-transitory computer readable storage medium having a computer program stored thereon, and the computer program, when executed by a processor, implements the convolutional neural network acceleration method provided by any of the above methods, which comprises: obtaining a convolution result corresponding to a current convolution window in a target image; in the case that the current convolution window is separated from a previous first target convolution window by n convolution windows, determining the current convolution window as the first target convolution window, and obtaining an intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows separated from the previous first target convolution window; in the case that the current convolution window is separated from a previous second target convolution window by m convolution windows, determining the current convolution window as the second target convolution window, and obtaining a pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows separated from the previous second target convolution window; wherein the current convolution window and the previous first target convolution window are located in the same row, and the current convolution window and the previous second target convolution window are located in the same column; n and m are preset positive integers.

[0162] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0163] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0164] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for accelerating a convolutional neural network, the method comprising: The method comprises the following steps: obtaining a convolution result corresponding to a current convolution window in a target image; in a case where the current convolution window is separated from a previous first target convolution window by n convolution windows, determining the current convolution window as the first target convolution window, and obtaining an intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows separated between the current convolution window and the previous first target convolution window; in a case where the current convolution window is separated from a previous second target convolution window by m convolution windows, determining the current convolution window as the second target convolution window, and obtaining a pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows separated between the current convolution window and the previous second target convolution window; wherein the current convolution window and the previous first target convolution window are located in the same row, and the current convolution window and the previous second target convolution window are located in the same column; n and m are preset positive integers.

2. The convolutional neural network acceleration method of claim 1, wherein, The method further comprises the following steps in addition to the steps of obtaining the convolution result corresponding to the current convolution window in the target image and determining the current convolution window as the second target convolution window in the case where the current convolution window is separated from the previous second target convolution window by m convolution windows: in a case where the current convolution window is the (n+1)th convolution window in the row, determining the current convolution window as the first target convolution window, and obtaining the intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the first n convolution windows in the row of the current convolution window.

3. The convolutional neural network acceleration method of claim 1, wherein, The method further comprises the following steps in addition to the step of obtaining the intermediate pooling result corresponding to the current convolution window: in a case where the current convolution window is the (m+1)th convolution window in the column, determining the current convolution window as the second target convolution window, and obtaining the pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the first m convolution windows in the column of the current convolution window.

4. The convolutional neural network acceleration method of claim 3, wherein, The method further comprises the following steps in addition to the step of obtaining the intermediate pooling result corresponding to the current convolution window: in a case where the current convolution window is not determined as the second target convolution window, caching the intermediate pooling result corresponding to the current convolution window.

5. The convolutional neural network acceleration method of claim 4, wherein, The method further comprises the following steps in addition to the steps of determining the current convolution window as the second target convolution window in the case where the current convolution window is separated from the previous second target convolution window by m convolution windows and obtaining the pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows separated between the current convolution window and the previous second target convolution window: obtaining the intermediate pooling results corresponding to the m convolution windows separated between the current convolution window and the previous second target convolution window from the cache; obtaining the intermediate pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows between the current convolution window and the previous second target convolution window; the current convolution window is determined as the second target convolution window, and the intermediate pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows in the column where the current convolution window is located. the intermediate pooling result corresponding to the m convolution windows in the column where the current convolution window is located is obtained from the cache; the intermediate pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows between the current convolution window and the previous second target convolution window.

6. The convolutional neural network acceleration method of claim 2, wherein, the intermediate pooling result corresponding to the current convolution window is obtained based on the convolution result corresponding to the current convolution window and the convolution result corresponding to the n convolution windows between the current convolution window and the previous first target convolution window, including: the maximum value of the convolution result corresponding to the current convolution window and the convolution result corresponding to the n convolution windows between the current convolution window and the previous first target convolution window is taken as the intermediate pooling result corresponding to the current convolution window; the intermediate pooling result corresponding to the current convolution window is obtained based on the convolution result corresponding to the current convolution window and the convolution result corresponding to the n convolution windows in the row where the current convolution window is located, including: the maximum value of the convolution result corresponding to the current convolution window and the convolution result corresponding to the n convolution windows in the row where the current convolution window is located is taken as the intermediate pooling result corresponding to the current convolution window.

7. The convolutional neural network acceleration method of claim 3, wherein, the intermediate pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows between the current convolution window and the previous second target convolution window, including: the maximum value of the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows between the current convolution window and the previous second target convolution window is taken as the pooling result corresponding to the current convolution window; the intermediate pooling result corresponding to the current convolution window is obtained based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows in the column where the current convolution window is located, including: the maximum value of the intermediate pooling result corresponding to the current convolution window and the intermediate pooling result corresponding to the m convolution windows in the column where the current convolution window is located is taken as the pooling result corresponding to the current convolution window.

8. The convolutional neural network acceleration method of any one of claims 1 to 7, wherein, after the pooling result corresponding to the current convolution window is obtained, the method further includes: The pooling result corresponding to the current convolution window is activated to obtain an activated pooling result corresponding to the current convolution window.

9. A convolutional neural network acceleration apparatus, characterized by, Comprise: The data acquisition module is used for acquiring the convolution result corresponding to the current convolution window in the target image. The first pooling module is used for determining the current convolution window as a first target convolution window when the current convolution window is spaced apart from a previous first target convolution window by n convolution windows, and acquiring an intermediate pooling result corresponding to the current convolution window based on the convolution result corresponding to the current convolution window and the convolution results corresponding to the n convolution windows spaced apart between the current convolution window and the previous first target convolution window. The second pooling module is used for determining the current convolution window as a second target convolution window when the current convolution window is spaced apart from a previous second target convolution window by m convolution windows, and acquiring a pooling result corresponding to the current convolution window based on the intermediate pooling result corresponding to the current convolution window and the intermediate pooling results corresponding to the m convolution windows spaced apart between the current convolution window and the previous second target convolution window. Wherein, the current convolution window and the previous first target convolution window are located in the same row, and the current convolution window and the previous second target convolution window are located in the same column; n and m are preset positive integers.

10. A convolutional neural network acceleration system, comprising: Also include: The weight parameter cache module, the input and output cache module, the general computing module and the computing control module; the computing control module comprises the convolutional neural network acceleration device as claimed in claim 9; The weight parameter cache module is connected with off-chip DDR, general computing module and computing control module respectively; the input and output cache module is connected with off-chip DDR, general computing module and computing control module respectively; the general computing module and the computing control module are connected.

Images