Calculation method of convolutional neural network and device for executing method

Abstract

The invention provides a calculation method of a convolutional neural network and a device for executing the method. The method comprises the steps that a first device performs convolution operation on data based on a first convolution layer of the convolutional neural network to obtain a first convolution matrix; writing, by the first device, the first convolution matrix into a high bandwidth memory (HBM) of a second device and controlling the second device to perform an activation operation on the first convolution matrix in the HBM based on a first activation layer corresponding to the first convolution layer to obtain a first activation matrix; in response to the fact that a first activation matrix is written into the HBM, the first device controls a second device to execute pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to a first convolutional layer to obtain a first pooling matrix; and in response to the first pooling matrix being written into the HBM, performing, by the first device, a convolution operation on the first pooling matrix based on a second convolutional layer of the convolutional neural network.

Description

Technical Field

[0001] The present application relates to the field of data computing technology, and more specifically, to a convolutional neural network computing method and a device for executing the method. Background Art

[0002] When a processor (e.g., CPU, GPU, etc.) performs convolutional neural network calculations, it needs to store some of the intermediate result data in high bandwidth memory (HBM) because its storage space is too small to store all the intermediate result data. This causes the processor to frequently interact with HBM to write and / or read the intermediate result data, thereby slowing down the convolutional neural network calculation speed (e.g., training speed, inference speed, etc.). In addition, the computational complexity of different layers in a convolutional neural network is different. The convolutional layer is a computationally intensive layer, while the activation layer and pooling layer are layers with simple computational logic. The computationally intensive layers are generally slowed down by the layers with simple computational logic.

[0003] Therefore, how to improve the computing speed of convolutional neural networks is a problem to be solved urgently in the present invention. Summary of the invention

[0004] The object of the present invention is to provide a convolutional neural network calculation method and a device for executing the method, so as to at least solve the problems in the above-mentioned related technologies, or not solve any of the above-mentioned problems.

[0005] According to one aspect of an embodiment of the present invention, a computing method of a convolutional neural network executed by a first device is provided, the method comprising: performing a convolution operation on data based on a first convolutional layer of the convolutional neural network to obtain a first convolution matrix; writing the first convolutional matrix to an HBM of a second device and controlling the second device to perform an activation operation on the first convolutional matrix in the HBM based on a first activation layer corresponding to the first convolutional layer to obtain a first activation matrix; in response to the first activation matrix being written to the HBM, controlling the second device to perform a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer to obtain a first pooling matrix; and in response to the first pooling matrix being written to the HBM, performing a convolution operation on the first pooling matrix based on the second convolutional layer of the convolutional neural network, wherein the first convolutional layer, the first activation layer, the first pooling layer, and the second convolutional layer are sequentially cascaded in the convolutional neural network.

[0006] According to an embodiment of the present disclosure, by offloading the calculation of the activation layer and the pooling layer of the convolutional neural network to the second device, the data reading interaction between the first device and the second device can be reduced, thereby improving the calculation speed of the convolutional neural network.

[0007] Optionally, the method also includes: dividing the first convolution matrix into a plurality of partitions and / or dividing the first activation matrix into a plurality of partitions.

[0008] Optionally, the step of controlling the second device to perform an activation operation on the first convolution matrix in the HBM based on the first activation layer corresponding to the first convolution layer includes: controlling the second device to perform an activation operation on the first convolution matrix according to partitions, and / or, the step of controlling the second device to perform a pooling operation on the first activation matrix in the HBM based on the first pooling layer corresponding to the first convolution layer includes: performing a pooling operation on the first activation matrix according to partitions.

[0009] According to an embodiment of the present disclosure, activation operations and pooling operations are performed according to partitions, which can improve activation and pooling processing speeds.

[0010] Optionally, the size of the partition is determined based on the size of the convolution kernel of the first convolution layer of the convolutional neural network.

[0011] Optionally, in response to the first pooling matrix being written into the HBM, the step of performing a convolution operation on the first pooling matrix based on the second convolution layer of the convolutional neural network includes: in response to the data of the first pooling matrix having the convolution kernel size of the second convolution layer being written into the HBM, performing a convolution operation on the first pooling matrix based on the second convolution layer of the neural network.

[0012] According to an embodiment of the present disclosure, the calculation of the second convolutional layer is performed when the pooling matrix is ​​not completely acquired, so as to speed up the calculation speed of the convolutional neural network.

[0013] Optionally, the step of controlling the second device to perform a pooling operation on the first activation matrix in the HBM based on the first pooling layer corresponding to the first convolution layer includes: while controlling the second device to perform an activation operation on the first convolution matrix, controlling the second device to perform a pooling operation on the data in the first activation matrix that has been written into the HBM.

[0014] According to an embodiment of the present disclosure, the activation operation and the pooling operation are performed simultaneously, which can speed up the calculation speed of the convolutional neural network.

[0015] Optionally, the step of controlling the second device to perform an activation operation on the first convolution matrix according to partitions includes: performing parallel activation operations on multiple partitions of the first convolution matrix, and / or the step of controlling the second device to perform a pooling operation on the first activation matrix according to partitions includes: performing parallel pooling operations on multiple partitions of the first activation matrix.

[0016] According to an embodiment of the present disclosure, parallel processing can improve the computing speed of a convolutional neural network.

[0017] Optionally, the first device is a GPU, and the second device is a HBM-PIM device.

[0018] According to another aspect of an embodiment of the present disclosure, a calculation method of a convolutional neural network performed by a second device is provided, the method comprising: receiving from a first device a first convolution matrix obtained by performing a convolution operation on data based on a first convolution layer of the convolutional neural network by the first device to store in an HBM of the second device; receiving a first control message from the first device and in response to receiving the first control message, performing an activation operation on the first convolution matrix in the HBM based on a first activation layer corresponding to the first convolution layer to obtain a first activation matrix, wherein the first control message is generated by the first device based on the first convolution matrix being written into the HBM; A second control message is received from a first device, and in response to receiving the second control message, a pooling operation is performed on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer to obtain a first pooling matrix, wherein the second control message is generated by the first device in response to the first activation matrix being written into the HBM; and the first pooling matrix is ​​written into the HBM, wherein the first pooling matrix in the HBM is used by the first device to perform a convolution operation on the first pooling matrix based on the second convolutional layer of the convolutional neural network, and the first convolutional layer, the first activation layer, the first pooling layer, the second convolutional layer are sequentially cascaded in the neural network.

[0019] Optionally, the first convolution matrix is ​​divided into a plurality of partitions by the first device and / or the first activation matrix is ​​divided into a plurality of partitions by the first device.

[0020] Optionally, the step of performing an activation operation on the first convolution matrix in the HBM based on the first activation layer corresponding to the first convolution layer includes: performing an activation operation on the first convolution matrix according to partitions, and / or the step of performing a pooling operation on the first activation matrix in the HBM based on the first pooling layer corresponding to the first convolution layer includes: performing a pooling operation on the first activation matrix according to partitions.

[0021] Optionally, the size of the partition is determined based on the size of the convolution kernel of the first convolution layer of the convolutional neural network.

[0022] Optionally, when data of the first pooling matrix having the convolution kernel size of the second convolution layer is written into the HBM, the convolution operation on the first pooling matrix is ​​performed by the first device based on the second convolution layer of the convolutional neural network.

[0023] Optionally, the step of performing a pooling operation on a first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer includes: while performing an activation operation on the first convolutional matrix, performing a pooling operation on data in the first activation matrix that has been written into the HBM.

[0024] Optionally, the step of performing an activation operation on the first convolution matrix according to partitions includes: performing parallel activation operations on multiple partitions of the first convolution matrix, and / or the step of performing a pooling operation on the first activation matrix according to partitions includes: performing parallel pooling operations on multiple partitions of the first activation matrix.

[0025] Optionally, the first device is a GPU, and the second device is a HBM-PIM device.

[0026] According to another aspect of an embodiment of the present invention, a first device for executing a computing method of a convolutional neural network is provided, comprising: a first convolution unit, configured to: perform a convolution operation on data based on a first convolution layer of the convolutional neural network to obtain a first convolution matrix; a writing unit, configured to write the first convolution matrix to a high-bandwidth memory HBM of a second device; a control unit, configured to: control the second device to perform an activation operation on the first convolution matrix in the HBM based on a first activation layer corresponding to the first convolution layer to obtain a first activation matrix, and in response to the first activation matrix being written into the HBM, control the second device to perform a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolution layer to obtain a first pooling matrix; and a second convolution unit, configured to perform a convolution operation on the first pooling matrix based on the second convolution layer of the convolutional neural network in response to the first pooling matrix being written into the HBM, wherein the first convolution layer, the first activation layer, the first pooling layer, the second convolution layer are sequentially cascaded in the convolutional neural network.

[0027] Optionally, the first device also includes: a division unit configured to: divide the first convolution matrix into multiple partitions and / or divide the first activation matrix into multiple partitions.

[0028] Optionally, the control unit is configured to: control the second device to perform an activation operation on the first convolution matrix according to partitions, and / or to perform a pooling operation on the first activation matrix according to partitions.

[0029] Optionally, the size of the partition is determined based on the size of the convolution kernel of the first convolution layer of the convolutional neural network.

[0030] Optionally, the second convolution unit is configured to: in response to data of the first pooling matrix having a convolution kernel size of the second convolution layer being written into the HBM, perform a convolution operation on the first pooling matrix based on the second convolution layer of the neural network.

[0031] Optionally, the control unit is configured to: while controlling the second device to perform an activation operation on the first convolution matrix, control the second device to perform a pooling operation on the data in the first activation matrix that has been written into the HBM.

[0032] Optionally, the control unit is configured to: control the second device to perform parallel activation operations on multiple partitions of the first convolution matrix based on a first activation layer corresponding to the first convolution layer, and / or control the second device to perform parallel pooling operations on multiple partitions of the first activation matrix based on a first pooling layer corresponding to the first convolution layer.

[0033] Optionally, the first device is a GPU, and the second device is a HBM-PIM device.

[0034] According to another aspect of an embodiment of the present invention, a second device for executing a computing method of a convolutional neural network is provided, comprising: a receiving unit, configured to: receive from a first device a first convolution matrix, a first control message, and a second control message obtained by the first device performing a convolution operation based on a first convolution layer of the convolutional neural network; an activation unit, configured to perform an activation operation on the first convolution matrix in a high bandwidth memory HBM of the second device based on a first activation layer corresponding to the first convolution layer in response to receiving the first control message to obtain a first activation matrix; a pooling unit, configured to perform a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolution layer in response to receiving the second control message to obtain a first pooling matrix; and a writing unit, configured to: store the first convolution matrix in the HBM and write the first pooling matrix into the HBM, wherein the first pooling matrix in the HBM is used by the first device to perform a convolution operation on the first pooling matrix based on the second convolution layer of the convolutional neural network, and the first convolution layer, the first activation layer, the first pooling layer, the second convolution layer are sequentially cascaded in the convolutional neural network.

[0035] Optionally, the first convolution matrix is ​​divided into a plurality of partitions by the first device and / or the first activation matrix is ​​divided into a plurality of partitions by the first device.

[0036] Optionally, the activation unit is configured to: perform an activation operation on the first convolution matrix according to partitions, and / or the pooling unit is configured to: perform a pooling operation on the first activation matrix according to partitions.

[0037] Optionally, the size of the partition is determined based on the size of the convolution kernel of the first convolution layer of the convolutional neural network.

[0038] Optionally, when data of the first pooling matrix having the convolution kernel size of the second convolution layer is written into the HBM, the convolution operation on the first pooling matrix is ​​performed by the first device based on the second convolution layer of the convolutional neural network.

[0039] Optionally, the pooling unit is configured to: perform a pooling operation on the data in the first activation matrix that has been written into the HBM while performing an activation operation on the first convolution matrix.

[0040] Optionally, the activation unit is configured to perform parallel activation operations on multiple partitions of the first convolution matrix, and / or the pooling unit is configured to perform parallel pooling operations on multiple partitions of the first activation matrix.

[0041] Optionally, the first device is a GPU, and the second device is a HBM-PIM device.

[0042] According to another aspect of an embodiment of the present disclosure, a computer-readable storage medium storing a computer program is provided, wherein when the computer program is executed by a processor, the processor implements the calculation method of the convolutional neural network as described herein. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The above and other objects and features of the present invention will become more apparent through the following description in conjunction with the accompanying drawings which exemplarily illustrate an example, in which:

[0044] Figure 1 A schematic diagram showing a process of recognizing the number 4 in an image;

[0045] Figure 2 A diagram showing an example of a method of performing calculations of a convolutional neural network in the related art;

[0046] Figure 3 A flow chart of a convolutional neural network calculation method according to an embodiment of the present disclosure is shown;

[0047] Figure 4 is a diagram showing an example of partitioning of a convolution matrix, activation operation of an activation layer, and pooling operation of a pooling layer;

[0048] Figure 5 A comparative diagram showing activation operations of the related art and an embodiment of the present disclosure;

[0049] Figure 6 A flow chart of a convolutional neural network calculation method according to an embodiment of the present disclosure is shown;

[0050] Figure 7 A control flow diagram of a convolutional neural network calculation method according to an embodiment of the present disclosure is shown;

[0051] Figure 8 Shown with Figure 7 A schematic diagram of the data flow corresponding to the embodiment in FIG.

[0052] Fig. 9 is a block diagram showing the structure of a first device for executing a calculation method of a convolutional neural network according to an embodiment of the present disclosure; and

[0053] Fig.10 is a block diagram showing the structure of a second device for executing a calculation method of a convolutional neural network according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0054] Hereinafter, various embodiments of the present disclosure are described with reference to the accompanying drawings, wherein the same reference numerals are used to represent the same or similar elements, features and structures. However, it is not intended that the present disclosure be limited to specific embodiments by the various embodiments described herein, and it is intended that: the present disclosure covers all modifications, equivalents and / or substitutes of the present disclosure, as long as they are within the scope of the attached claims and their equivalents. The terms and words used in the following specification and claims are not limited to their dictionary meanings, but are only used to enable a clear and consistent understanding of the present disclosure. Therefore, it should be apparent to those skilled in the art that the following description of various embodiments of the present disclosure is provided for illustrative purposes only, and not for the purpose of limiting the present disclosure defined by the attached claims and their equivalents.

[0055] It should be understood that the singular includes the plural unless the context clearly indicates otherwise.The terms "include", "comprising" and "having" used herein indicate the presence of disclosed functions, operations or elements, but do not exclude other functions, operations or elements.

[0056] For example, the expression "A or B" or "at least one of A and / or B" may indicate A and B, A or B. For example, the expression "A or B" or "at least one of A and / or B" may indicate (1) A, (2) B, or (3) both A and B.

[0057] In various embodiments of the present disclosure, it is intended that when a component (e.g., a first component) is referred to as being "coupled" or "connected" to another component (e.g., a second component) or being "coupled" or "connected" to another component (e.g., the second component), the component may be directly connected to the other component or may be connected through another component (e.g., a third component). In contrast, when a component (e.g., a first component) is referred to as being "directly coupled" or "directly connected" to another component (e.g., the second component) or being directly coupled to or directly connected to another component (e.g., the second component), there is no other component (e.g., the third component) between the component and the other component.

[0058] The expression "configured to" used in describing various embodiments of the present disclosure may be used interchangeably with expressions such as "suitable for", "having the ability to", "designed to", "suitable for", "manufactured to", and "capable of", for example, depending on the circumstances. The term "configured to" may not necessarily indicate that it is "specially designed to" in terms of hardware. On the contrary, the expression "a device configured to..." in some cases may indicate that the device and another device or part "can..." For example, the expression "a processor configured to perform A, B, and C" may indicate a dedicated processor (e.g., an embedded processor) for performing the corresponding operations or a general-purpose processor (e.g., a central processing unit CPU or an application processor (AP)) for performing the corresponding operations by executing at least one software program stored in a memory device.

[0059] The terms used herein are to describe certain embodiments of the present disclosure, but are not intended to limit the scope of other embodiments. Unless otherwise noted herein, all terms used herein (including technical or scientific terms) may have the same meaning as those generally understood by those skilled in the art. Typically, the terms defined in the dictionary should be considered to have the same meaning as the contextual meaning in the relevant field, and, unless clearly defined herein, should not be understood differently or understood to have an overly formal meaning. In any case, the terms defined in the present disclosure are not intended to be interpreted as excluding embodiments of the present disclosure.

[0060] In order to facilitate the explanation of the present disclosure, the calculation process of the convolutional neural network is first explained. For the sake of ease of description, image recognition is taken as an example.

[0061] Figure 1 A schematic diagram showing the process of recognizing the number 4 in an image.

[0062] Reference Figure 1 In a), assuming that the image can be represented by 8*8 pixels, the value of white pixels (i.e. the corresponding part of the number 4) is 1, and the value of black pixels is 0, the values ​​corresponding to different areas of the image can be obtained, and these values ​​can be represented in the form of a matrix.

[0063] Reference Figure 1 b), a convolution operation can be performed on the matrix shown in a) based on the convolution layer. For example, a convolution operation is performed on the 3*3 submatrix in the upper left corner of the matrix to obtain a value of 0.

[0064] Reference Figure 1 c), it can move to the left with the step size set by the user to perform the next convolution operation, and finally obtain Figure 1 The convolution result shown in d) in FIG. 5 (may also be referred to as the convolution matrix in this article).

[0065] Then, based on the activation layer, the activation function can be used to activate all elements in the convolution matrix to obtain the activation matrix. Convolutional neural networks can enhance the ability to learn complex features by using nonlinear functions as activation functions. Among them, the most commonly used activation function is the Relu function, which can be expressed as follows:

[0066]

[0067] like Figure 1 As shown in e), the Relu function is used to obtain the activation matrix corresponding to the convolution matrix.

[0068] After obtaining the activation matrix, you can perform a pooling operation on the activation matrix based on the pooling layer. The pooling operation is used to prevent the model from overfitting. The maximum pooling method is the most commonly used pooling method. Figure 1 f), can be obtained based on the maximum pooling as follows Figure 1 For example, the result corresponding to the first maximum pooling is 2, and the result corresponding to the fourth maximum pooling is 3.

[0069] Figure 2 A diagram showing an example of a method of performing calculations of a convolutional neural network in the related art.

[0070] Reference Figure 2 , the GPU or GPU chip writes the calculation result of the convolution layer to HBM, reads the calculation result of the convolution layer from HBM to perform the activation operation based on the activation function, the calculation result of the activation operation is written to HBM, the calculation result of the activation operation is read to the GPU chip to perform the pooling operation of the pooling layer, and the calculation result of the pooling operation is written to HBM. In addition, although Figure 2 Not shown in the figure, but when the calculation result of the pooling layer is used for the next layer of the pooling layer (for example, another convolutional layer or a fully connected layer), it needs to be read from the HBM to the GPU chip to perform the corresponding calculation.

[0071] It can be seen that during the calculation process of the convolutional neural network, the GPU chip needs to frequently interact with the HBM for data reading, which will obviously slow down the calculation speed of the convolutional neural network, thus leading to a bottleneck in the calculation speed.

[0072] The convolutional neural network calculation method proposed in the present disclosure can offload the simple layers of the convolutional neural network calculation logic to other computing devices to reduce the interaction between the processor and the HBM and improve the calculation speed of the convolutional neural network.

[0073] Figure 3 A flowchart of a convolutional neural network calculation method performed by a first device according to an embodiment of the present disclosure is shown.

[0074] Reference Figure 3 In step S301, a first device performs a convolution operation on the data based on a first convolution layer of a convolutional neural network to obtain a first convolution matrix.

[0075] As understood by those skilled in the art, performing a convolution operation on data based on the first convolution layer of the convolutional neural network may represent using the first convolution layer to perform a convolution operation on a matrix (e.g., Figure 1 A convolution operation is performed on the matrix shown in a) in FIG. 5 to obtain a convolution matrix.

[0076] As an example, the data may be a matrix representing an image or other matrix that needs to be processed by a convolutional neural network.

[0077] As an example, the first device may be a GPU, a CPU, etc. As will be appreciated by those skilled in the art, the first device may include a computing core and a memory, wherein the computing core performs a computing function. For example, a GPU chip or a GPU may include registers, a shared memory, and a local memory as a memory.

[0078] In step S302, the first convolution matrix is ​​written into the HBM of the second device and the second device is controlled to perform an activation operation on the first convolution matrix in the HBM based on a first activation layer corresponding to the first convolution layer to obtain a first activation matrix.

[0079] As understood by those skilled in the art, the first activation layer is a layer next to the first convolutional layer connected to the first convolutional layer.

[0080] In the related art (for example, see Figure 2 In the example described above, the activation layer calculation is performed at the first device (i.e., the GPU), while the embodiment of the present disclosure offloads the activation layer of the convolutional neural network to the second device to perform the calculation, thereby reducing the data reading interaction between the first device and the second device.

[0081] As an example, the activation function corresponding to the first activation layer may be a Relu function.

[0082] As an example, the second device may be an HBM-PIM device. The HBM-PIM device integrates an AI-specific semiconductor in its HBM, a technology known as Processing in Memory (PIM). This technology integrates a dedicated data processor directly into DRAM to transfer part of the data computing work from the host processor to the memory, which can reduce data movement to improve energy efficiency and data processing efficiency.

[0083] In step S303, in response to the first activation matrix being written into the HBM, the second device is controlled to perform a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer to obtain a first pooling matrix.

[0084] As understood by those skilled in the art, the pooling layer is a layer below the activation layer connected to the activation layer.

[0085] In the related art (for example, see Figure 2 In the example described above, the calculation of the pooling layer is performed at the first device (i.e., the GPU), while the embodiment of the present disclosure offloads the pooling layer of the convolutional neural network to the second device to perform the calculation, thereby reducing the data reading interaction between the first device and the second device. As an example, the pooling layer can use maximum pooling processing.

[0086] As an example, Figure 3 The illustrated method may also include partitioning the first convolution matrix into a plurality of partitions and / or partitioning the first activation matrix into a plurality of partitions.

[0087] As an example, the step of controlling the second device to perform an activation operation on the first convolution matrix in the HBM based on the first activation layer corresponding to the first convolution layer includes: controlling the second device to perform an activation operation on the first convolution matrix according to partitions, and / or, the step of controlling the second device to perform a pooling operation on the first activation matrix in the HBM based on the first pooling layer corresponding to the first convolution layer includes: performing a pooling operation on the first activation matrix according to partitions.

[0088] Figure 4 It is a diagram showing examples of partitioning of a convolution matrix, activation operation of an activation layer, and pooling operation of a pooling layer.

[0089] Reference Figure 4 , the convolution matrix can be divided into four partitions R1, R2, R3 and R4.

[0090] In the related art, a GPU adopts a row-based activation strategy when performing an activation operation on a convolution matrix.

[0091] According to the calculation method of an embodiment of the present disclosure, the second device performs activation on the convolution matrix according to partitions.

[0092] For example, refer to Figure 4 , the activation operation can be performed on R1, R2, R3 and R4 in sequence. Obviously, compared with the activation by row, the activation operation by partition can improve the calculation speed of the activation layer.

[0093] As an example, the step of controlling the second device to perform an activation operation on the first convolution matrix according to the partitions includes: performing parallel activation operations on multiple partitions of the first convolution matrix, and / or, the step of controlling the second device to perform a pooling operation on the first activation matrix according to the partitions includes: performing parallel pooling operations on multiple partitions of the first activation matrix. That is, when performing activation or pooling operations, parallel processing is performed on the partitions. Parallel processing can obviously increase the calculation speed.

[0094] Figure 5 A comparative diagram showing activation operations of the related art and an embodiment of the present disclosure.

[0095] Reference Figure 5 When the GPU performs an activation operation on the convolution matrix, a row-based activation strategy is adopted, while the embodiment of the present disclosure activates multiple elements in the convolution matrix in parallel. Obviously, compared with the activation operation based on the GPU, the activation operation according to the embodiment of the present disclosure requires less activation processing time.

[0096] As an example, the step of controlling the second device to perform a pooling operation on the first activation matrix in the HBM based on the first pooling layer corresponding to the first convolution layer includes: while controlling the second device to perform an activation operation on the first convolution matrix, controlling the second device to perform a pooling operation on the data in the first activation matrix that has been written into the HBM.

[0097] According to an embodiment of the present disclosure, when the activation operation is not completed, the pooling operation of the pooling layer can be performed. Figure 4 , after completing the activation of R1 and executing the activation of R2, the pooling corresponding to R1 can be executed, that is, the pooling operation is performed on the activation elements corresponding to R1. Obviously, since the pooling operation does not need to wait for the activation operation to be completed, the efficiency of activation and pooling can be improved.

[0098] In step S304, in response to the first pooling matrix being written into the HBM, a convolution operation is performed on the first pooling matrix based on the second convolutional layer of the convolutional neural network, wherein the first convolutional layer, the first activation layer, the first pooling layer, and the second convolutional layer are sequentially cascaded in the convolutional neural network.

[0099] As an example, in response to the first pooling matrix being written into the HBM, the step of performing a convolution operation on the first pooling matrix based on the second convolution layer of the neural network includes: in response to the data of the first pooling matrix having the convolution kernel size of the second convolution layer being written into the HBM, performing a convolution operation on the first pooling matrix based on the second convolution layer of the convolutional neural network.

[0100] According to an embodiment of the present disclosure, the convolution operation of the second convolutional layer is performed after partial pooling is completed, so that there is no need to wait until the pooling operation is fully completed before performing the calculation of the second convolutional layer, which can obviously improve the calculation speed of the convolutional neural network.

[0101] As an example, the size of the partition may be determined based on the size of the convolution kernel of the first convolution layer of the convolutional neural network.

[0102] Figure 6 A flowchart of a convolutional neural network calculation method performed by a second device according to an embodiment of the present disclosure is shown.

[0103] Reference Figure 6 In step S601, a first convolution matrix obtained by performing a convolution operation on data based on a first convolution layer of the convolutional neural network is received from the first device to be stored in the HBM of the second device.

[0104] In step S602, a first control message is received from a first device and in response to receiving the first control message, an activation operation is performed on a first convolution matrix in the HBM based on a first activation layer corresponding to the first convolution layer to obtain a first activation matrix, wherein the first control message is generated by the first device based on the first convolution matrix being written into the HBM.

[0105] In step S603, a second control message is received from the first device, and in response to receiving the second control message, a pooling operation is performed on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer to obtain a first pooling matrix, wherein the second control message is generated by the first device in response to the first activation matrix being written to the HBM.

[0106] In step S604, a first pooling matrix is ​​written into the HBM, wherein the first pooling matrix in the HBM is used by a first device to perform a convolution operation on the first pooling matrix based on a second convolutional layer of the convolutional neural network, and the first convolutional layer, the first activation layer, the first pooling layer, and the second convolutional layer are sequentially cascaded in the neural network.

[0107] As an example, the first convolution matrix is ​​divided into a plurality of partitions by the first device and / or the first activation matrix is ​​divided into a plurality of partitions by the first device.

[0108] As an example, the step of performing an activation operation on the first convolution matrix in the HBM based on the first activation layer corresponding to the first convolution layer includes: performing an activation operation on the first convolution matrix according to partitions, and / or the step of performing a pooling operation on the first activation matrix in the HBM based on the first pooling layer corresponding to the first convolution layer includes: performing a pooling operation on the first activation matrix according to partitions.

[0109] As an example, the size of the partition is determined based on the size of the convolution sum of the first convolutional layer of the convolutional neural network.

[0110] As an example, when data of a first pooling matrix having a convolution kernel size of a second convolution layer is written into the HBM, a convolution operation on the first pooling matrix is ​​performed by the first device based on the second convolution layer of the convolutional neural network.

[0111] As an example, the step of performing a pooling operation on the first activation matrix in the HBM based on the first pooling layer corresponding to the first convolution layer includes: while performing an activation operation on the first convolution matrix, performing a pooling operation on the data in the first activation matrix that has been written into the HBM.

[0112] As an example, the step of performing an activation operation on the first convolution matrix according to partitions includes: performing parallel activation operations on multiple partitions of the first convolution matrix, and / or the step of performing a pooling operation on the first activation matrix according to partitions includes: performing parallel pooling operations on multiple partitions of the first activation matrix.

[0113] As an example, the first device is a GPU and the second device is a HBM-PIM device.

[0114] Figure 7 A control flow diagram of a convolutional neural network calculation method according to an embodiment of the present disclosure is shown.

[0115] Reference Figure 7 , the GPU may correspond to the first device described above, and the HBM-PIM device may correspond to the second device described above.

[0116] Reference Figure 7 , the GPU can control the storage and computing operations in HBM-PIM based on the process supervision module.

[0117] Specifically, in step 1.1, the process monitoring module may monitor whether the convolution matrix is ​​written into the HBM.

[0118] In step 1.2, if the convolution matrix is ​​written to HBM, the adaptive activation layer is notified to start working. The adaptive activation layer indicates the activation layer of the convolutional neural network that is unloaded to the HBM-PIM device. In other words, the calculation of the activation layer of the convolutional neural network is performed in the HBM-PIM device.

[0119] In step 2.1, the process monitoring module may monitor whether the activation matrix is ​​written into the HBM.

[0120] In step 2.2, if the activation matrix is ​​written to HBM, the adaptive pooling layer is notified to start working. The adaptive pooling layer described in this article indicates the pooling layer of the convolutional neural network that is offloaded to the HBM-PIM device. In other words, the calculation of the pooling layer of the convolutional neural network is performed in the HBM-PIM device.

[0121] In step 3.1, the process monitoring module may monitor whether the pooling matrix is ​​written into the HBM.

[0122] In step 3.2, when part of the data of the pooling matrix (for example, data of a convolution kernel size) is written to the HBM, the CNN is notified to perform the calculation of the next layer of the pooling layer.

[0123] According to an embodiment of the present disclosure, when the CNN model is initially loaded into the GPU, the process monitoring module scans the CNN model and records relevant parameters, such as input data size, convolution kernel size, number of network layers, etc. After obtaining the size of the convolution kernel, matrix partitioning can be performed according to the size of the convolution kernel, and the calculation of the adaptive layer is based on this matrix partitioning.

[0124] As an example, the sizes of convolution kernels of different convolution layers can be the same or different. In a CNN model, the sizes of convolution kernels of all convolution layers are generally the same. If they are different, they can be clearly defined in the model. In other words, each convolution layer has its own defined convolution kernel size.

[0125] As an example, the size of the convolution kernel of the first convolutional layer is S1 (for example, 5*5), then the size of the partition corresponding to the first convolutional layer can be determined as S1, that is, when performing activation on the calculation result of the first convolutional layer, activation is performed according to the partition of size 5*5.

[0126] As an example, the size of the convolution kernel of the second convolutional layer (to be described below) is S2 (for example, 3*3), then the size of the partition corresponding to the second convolutional layer is determined to be S2, that is, when performing, for example, activation on the calculation result of the second convolutional layer, the activation is performed according to the partition of size 3*3.

[0127] As understood by those skilled in the art, in the calculation of convolutional neural networks, the convolution kernel is the smallest unit of GPU calculation. Dividing the partitions according to the size of the convolution kernel is more conducive to GPU calculation, thereby speeding up the training or calculation speed of the model.

[0128] Figure 8 Shown with Figure 7 Schematic diagram of the corresponding data flow in the embodiment.

[0129] As understood by those skilled in the art, data flow refers to read and write operations under control flow.

[0130] Specifically, in step 1.1, CNN writes the intermediate result matrix (e.g., convolution matrix) into HBM.

[0131] In step 1.2, the adaptive activation layer reads the convolution matrix from the HBM.

[0132] In step 2.1, the adaptive activation layer writes the activation matrix into HBM.

[0133] In step 2.2, the adaptive pooling layer reads the activation matrix from the HBM.

[0134] In step 3.1, the adaptive pooling layer writes the pooling matrix into HBM.

[0135] In step 3.2, the GPU reads the pooling matrix from the HBM to perform the corresponding calculation of the next layer of the pooling layer.

[0136] Reference above Figures 1 to 8 The calculation method of the convolutional neural network according to the embodiment of the present disclosure is described below. Figure 9-10 A device for performing calculations of a convolutional neural network according to an embodiment of the present disclosure is described.

[0137] Fig. 9 is a block diagram showing the structure of a first device 900 for executing a calculation method of a convolutional neural network according to an embodiment of the present disclosure.

[0138] Reference Fig. 9 , the first device 900 may include: a first convolution unit 901, a writing unit 902, a control unit 903 and a second convolution unit 904.

[0139] Those skilled in the art should understand that the first device 900 may additionally include other components, and at least one of the components included in the first device 900 may be combined or split.

[0140] As an example, the first convolution unit 901 may be configured to: perform a convolution operation on the data based on the first convolution layer of the convolutional neural network to obtain a first convolution matrix.

[0141] As an example, the writing unit 902 may be configured to write the first convolution matrix to a high bandwidth memory HBM of the second device.

[0142] As an example, the control unit 903 may be configured to: control the second device to perform an activation operation on the first convolution matrix in the HBM based on a first activation layer corresponding to the first convolution layer to obtain a first activation matrix, and in response to the first activation matrix being written into the HBM, control the second device to perform a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolution layer to obtain a first pooling matrix.

[0143] As an example, the second convolution unit 904 may be configured to perform a convolution operation on the first pooling matrix based on a second convolution layer of the convolutional neural network in response to the first pooling matrix being written into the HBM, wherein the first convolution layer, the first activation layer, the first pooling layer, and the second convolution layer are sequentially cascaded in the convolutional neural network.

[0144] As an example, the first device 900 may also include: a division unit (not shown) configured to: divide the first convolution matrix into a plurality of partitions and / or divide the first activation matrix into a plurality of partitions.

[0145] As an example, the control unit 903 may be configured to: control the second device to perform an activation operation on the first convolution matrix according to partitions, and / or to perform a pooling operation on the first activation matrix according to partitions.

[0146] Optionally, the size of the partition is determined based on the size of the convolution kernel of the first convolution layer of the convolutional neural network.

[0147] Optionally, the second convolution unit 904 may be configured to: in response to data of the first pooling matrix having a convolution kernel size of the second convolution layer being written into the HBM, perform a convolution operation on the first pooling matrix based on the second convolution layer of the neural network.

[0148] Optionally, the control unit 903 may be configured to: while controlling the second device to perform an activation operation on the first convolution matrix, control the second device to perform a pooling operation on the data in the first activation matrix that has been written into the HBM.

[0149] Optionally, the control unit 903 may be configured to: control the second device to perform parallel activation operations on multiple partitions of the first convolution matrix based on a first activation layer corresponding to the first convolution layer, and / or control the second device to perform parallel pooling operations on multiple partitions of the first activation matrix based on a first pooling layer corresponding to the first convolution layer.

[0150] Optionally, the first device is a GPU, and the second device is a HBM-PIM device.

[0151] Fig.10 is a block diagram illustrating a structure of a second device 1000 for performing a calculation method of a convolutional neural network according to an embodiment of the present disclosure.

[0152] Reference Fig.10 , the second device 1000 may include: a receiving unit 1001, an activation unit 1002, a pooling unit 1003 and a writing unit 1004.

[0153] Those skilled in the art should understand that the second device 1000 may additionally include other components, and at least one of the components included in the second device 1000 may be combined or split.

[0154] As an example, the receiving unit 1001 may be configured to: receive from the first device a first convolution matrix, a first control message, and a second control message obtained by the first device performing a convolution operation on data based on the first convolution layer of the convolutional neural network.

[0155] As an example, the activation unit 1002 may be configured to perform an activation operation on a first convolution matrix in a high bandwidth memory HBM of the second device based on a first activation layer corresponding to the first convolution layer in response to receiving the first control message to obtain a first activation matrix.

[0156] As an example, the pooling unit 1003 may be configured to perform a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer in response to receiving the second control message to obtain a first pooling matrix.

[0157] As an example, the writing unit may be configured to: store a first convolution matrix in the HBM, and write a first pooling matrix into the HBM, wherein the first pooling matrix in the HBM is used by a first device to perform a convolution operation on the first pooling matrix based on a second convolution layer of the convolutional neural network, and the first convolution layer, the first activation layer, the first pooling layer, and the second convolution layer are sequentially cascaded in the convolutional neural network.

[0158] As an example, the first convolution matrix may be divided into a plurality of partitions by the first device and / or the first activation matrix may be divided into a plurality of partitions by the first device.

[0159] As an example, the activation unit 1002 may be configured to perform an activation operation on the first convolution matrix according to partitions, and / or the pooling unit 1003 may be configured to perform a pooling operation on the first activation matrix according to partitions.

[0160] As an example, the size of the partition is determined based on the size of the convolution kernel of the first convolution layer of the convolutional neural network.

[0161] As an example, when data of a first pooling matrix having a convolution kernel size of a second convolution layer is written into the HBM, a convolution operation on the first pooling matrix is ​​performed by the first device based on the second convolution layer of the neural network.

[0162] As an example, the pooling unit 1003 may be configured to: perform a pooling operation on the data in the first activation matrix that has been written into the HBM while performing an activation operation on the first convolution matrix.

[0163] As an example, the activation unit 1002 may be configured to perform parallel activation operations on multiple partitions of the first convolution matrix, and / or the pooling unit 1003 may be configured to perform parallel pooling operations on multiple partitions of the first activation matrix.

[0164] As an example, the first device is a GPU and the second device is a HBM-PIM device.

[0165] According to an embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein when the instructions are executed by at least one processor, the at least one processor performs the calculation method of the convolutional neural network according to the embodiment of the present disclosure. Examples of computer-readable storage media here include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as, multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device, any other device is configured to store computer programs and any associated data, data files and data structures in a non-transitory manner and provide the computer programs and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be run in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system, so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0166] According to an embodiment of the present disclosure, a computer program product may also be provided, and instructions in the computer program product may be executed by a processor of a computer device to complete the convolutional neural network computing method described herein.

[0167] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

Claims

1. A convolutional neural network computing method performed by a first device, the method comprising: Performing a convolution operation on the data based on the first convolution layer of the convolutional neural network to obtain a first convolution matrix; Writing the first convolution matrix into a high bandwidth memory HBM of the second device and controlling the second device to perform an activation operation on the first convolution matrix in the HBM based on a first activation layer corresponding to the first convolution layer to obtain a first activation matrix; In response to the first activation matrix being written into the HBM, controlling the second device to perform a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer to obtain a first pooling matrix; as well as In response to the first pooling matrix being written into the HBM, performing a convolution operation on the first pooling matrix based on a second convolutional layer of the convolutional neural network, Among them, the first convolutional layer, the first activation layer, the first pooling layer and the second convolutional layer are sequentially cascaded in the convolutional neural network.

2. The method of claim 1, further comprising: The first convolution matrix is ​​divided into a plurality of partitions and / or the first activation matrix is ​​divided into a plurality of partitions.

3. The method of claim 2, wherein: The step of controlling the second device to perform an activation operation on the first convolution matrix in the HBM based on the first activation layer corresponding to the first convolution layer includes: controlling the second device to perform an activation operation on the first convolution matrix according to the partitions, And / or, the step of controlling the second device to perform a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer includes: A pooling operation is performed on the first activation matrix according to the partitions.

4. The method of claim 2, wherein: The size of the partition is determined based on the size of the convolution kernel of the first convolution layer of the convolutional neural network.

5. The method of claim 1, in response to the first pooling matrix being written into the HBM, the step of performing a convolution operation on the first pooling matrix based on the second convolutional layer of the convolutional neural network comprises: In response to data of the first pooling matrix having a convolution kernel size of a second convolution layer being written into the HBM, a convolution operation is performed on the first pooling matrix based on the second convolution layer of the neural network.

6. The method of claim 1, wherein: The step of controlling the second device to perform a pooling operation on the first activation matrix in the HBM based on the first pooling layer corresponding to the first convolutional layer includes: While controlling the second device to perform an activation operation on the first convolution matrix, the second device is controlled to perform a pooling operation on the data in the first activation matrix that has been written into the HBM.

7. The method of claim 3, wherein: The step of controlling the second device to perform an activation operation on the first convolution matrix according to the partitions includes: performing parallel activation operations on the plurality of partitions of the first convolution matrix, And / or, the step of controlling the second device to perform a pooling operation on the first activation matrix according to the partitions includes: A parallel pooling operation is performed on the plurality of partitions of the first activation matrix.

8. The method of claim 1, wherein: The first device is a GPU, and the second device is a HBM-PIM device.

9. A convolutional neural network computing method performed by a second device, the method comprising: Receiving from the first device a first convolution matrix obtained by the first device performing a convolution operation on the data based on the first convolution layer of the convolutional neural network, and storing the first convolution matrix in the high bandwidth memory HBM of the second device; receiving a first control message from a first device and in response to receiving the first control message, performing an activation operation on a first convolution matrix in the HBM based on a first activation layer corresponding to the first convolution layer to obtain a first activation matrix, wherein the first control message is generated by the first device based on the first convolution matrix being written into the HBM; receiving a second control message from the first device, and in response to receiving the second control message, performing a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer to obtain a first pooling matrix, wherein the second control message is generated by the first device in response to the first activation matrix being written into the HBM; and A first pooling matrix is ​​written into the HBM, wherein the first pooling matrix in the HBM is used by a first device to perform a convolution operation on the first pooling matrix based on a second convolutional layer of the convolutional neural network, and the first convolutional layer, the first activation layer, the first pooling layer, and the second convolutional layer are sequentially cascaded in the neural network.

10. The method of claim 9, wherein: The first convolution matrix is ​​divided into a plurality of partitions by the first device and / or the first activation matrix is ​​divided into a plurality of partitions by the first device.

11. The method of claim 10, wherein: The step of performing an activation operation on a first convolution matrix in the HBM based on a first activation layer corresponding to the first convolution layer comprises: Perform activation operations on the first convolution matrix according to the partition, And / or, the step of performing a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer includes: A pooling operation is performed on the first activation matrix according to the partitions.

12. The method of claim 10, wherein: The size of the partition is determined based on the size of the convolution kernel of the first convolution layer of the convolutional neural network.

13. The method of claim 9, wherein: When data of the first pooling matrix having the convolution kernel size of the second convolution layer is written into the HBM, the convolution operation on the first pooling matrix is ​​performed by the first device based on the second convolution layer of the convolutional neural network.

14. The method of claim 9, wherein: The step of performing a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer comprises: While performing an activation operation on the first convolution matrix, a pooling operation is performed on the data in the first activation matrix that has been written into the HBM.

15. The method of claim 11, wherein: The step of performing an activation operation on the first convolution matrix according to the partition includes: performing parallel activation operations on the plurality of partitions of the first convolution matrix, And / or, the step of performing a pooling operation on the first activation matrix according to partitions includes: performing a parallel pooling operation on the multiple partitions of the first activation matrix.

16. The method of claim 9, wherein the first device is a GPU, and the second device is a HBM-PIM device.

17. A first device for executing a convolutional neural network computing method, comprising: A first convolution unit is configured to: perform a convolution operation on the data based on the first convolution layer of the convolutional neural network to obtain a first convolution matrix; A writing unit configured to write the first convolution matrix to a high bandwidth memory HBM of the second device; A control unit, configured to: control the second device to perform an activation operation on the first convolution matrix in the HBM based on a first activation layer corresponding to the first convolution layer to obtain a first activation matrix, and in response to the first activation matrix being written into the HBM, control the second device to perform a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolution layer to obtain a first pooling matrix; as well as a second convolution unit configured to, in response to the first pooling matrix being written into the HBM, perform a convolution operation on the first pooling matrix based on a second convolution layer of the convolutional neural network, Among them, the first convolutional layer, the first activation layer, the first pooling layer and the second convolutional layer are sequentially cascaded in the convolutional neural network.

18. A second device for executing a convolutional neural network computing method, comprising: A receiving unit is configured to: receive from the first device a first convolution matrix, a first control message, and a second control message obtained by the first device performing a convolution operation on the data based on the first convolution layer of the convolutional neural network; an activation unit configured to, in response to receiving the first control message, perform an activation operation on a first convolution matrix in a high bandwidth memory HBM of the second device based on a first activation layer corresponding to the first convolution layer to obtain a first activation matrix; a pooling unit configured to perform a pooling operation on the first activation matrix in the HBM based on a first pooling layer corresponding to the first convolutional layer in response to receiving the second control message to obtain a first pooling matrix; as well as A writing unit is configured to: store the first convolution matrix in the HBM, and write the first pooling matrix into the HBM, Among them, the first pooling matrix in the HBM is used by the first device to perform a convolution operation on the first pooling matrix based on the second convolution layer of the convolutional neural network, and the first convolution layer, the first activation layer, the first pooling layer and the second convolution layer are sequentially cascaded in the convolutional neural network.

19. A computer-readable storage medium storing a computer program, wherein: When the computer program is executed by a processor, the processor is caused to implement the method according to any one of claims 1 to 16.

Images