Acceleration method and device of neural network model, electronic equipment and storage medium

Abstract

The application discloses a neural network model acceleration method and device, electronic equipment and a storage medium, and comprises the following steps: in response to a neural network model acceleration event being triggered, a trained convolutional neural network model is acquired; for each two-dimensional convolution operator of the convolutional neural network model, the two-dimensional convolution operator is split into at least two one-dimensional convolution operators; wherein the width of the input feature map of the one-dimensional convolution operator is the same as the width of the initial convolution kernel of the two-dimensional convolution operator; during the sliding process of the initial convolution kernel in the height direction of the input feature map, the initial convolution kernel in the height dimension is expanded at each sliding time to generate a target convolution kernel; the target convolution kernel is unfolded into two-dimensional convolution kernel data, and the two-dimensional convolution kernel data and the bias data corresponding to the two-dimensional convolution operator are mapped to a calculation array; and the input feature map is processed based on the at least two one-dimensional convolution operators and the calculation array. The scheme can improve the calculation performance of the convolutional neural network model.

Description

Technical Field

[0001] This invention relates to the field of machine learning technology, and in particular to methods, apparatuses, electronic devices, and storage media for accelerating neural network models. Background Technology

[0002] Currently, convolutional neural networks (CNNs) have achieved tremendous success in many fields such as computer vision, speech processing, natural language processing, and intelligent transportation, attracting increasing attention from researchers and engineers. To better solve complex problems in various fields, CNNs are generating more and more network parameters and computational demands. While traditional chips based on the John von Neumann architecture have achieved breakthrough performance, they often result in significant power consumption. This is especially true for low-power terminal devices such as headphones, hearing aids, watches, and glasses, posing a greater challenge to balancing chip performance and power consumption. Therefore, many researchers have begun exploring an in-memory computing architecture that breaks away from the John von Neumann computer architecture. This architecture stores the weights and biases of the neural network in the conductance of the in-memory array nodes. Simultaneously, when the input data feature map, represented by voltage, is fed into the array, Ohm's law indicates that the output current is the product of the voltage and the conductance, thus completing the matrix multiplication operation between the input data feature map and the network weights.

[0003] In related technologies, GPU-based parallel computing accelerates the computation of convolutional processing. The general process of GPU parallel computing typically involves dividing a large-scale computational task into smaller, parallel computational tasks. Input data or weight data is distributed to different GPU cores for parallel computation, allowing each GPU core to perform only a portion of the input data or weight data computation. Finally, the intermediate feature maps are integrated based on their relationships to generate a complete feature map.

[0004] However, when using GPUs for convolutional computations, data, weights, and bias parameters need to be loaded into GPU memory. When the computational task is large, the memory consumption of the GPU runtime is also significant, potentially leading to memory shortages or requiring substantial memory swapping. This further increases the computation time of convolutional operations. Furthermore, when using multiple GPUs for convolutional computations, high-speed networks are required for communication to exchange data or parameters between different GPUs. Network latency and bandwidth can become performance bottlenecks, affecting the efficiency and scalability of parallel computing. Moreover, while GPUs possess powerful parallel computing capabilities, they also come with high energy consumption. The large number of computing units and high-frequency operations of a GPU consume significant amounts of power, resulting in high power consumption. This can increase the electrical load on the device and incur additional electricity costs. Additionally, implementing GPU parallel computing requires a deep understanding and mastery of GPU architecture and parallel programming, necessitating adjustments and optimizations for different hardware platforms and deep learning frameworks. This can present a significant barrier for ordinary programmers.

[0005] In related technologies, the Img2Col method is used to convert the input feature map into a two-dimensional matrix and the weight data into a two-dimensional matrix, thus transforming the convolution operation into a vectorized computation of two matrices. This conversion can significantly reduce the number of loops in the convolution operation and improve computational efficiency. However, the optimization method of Img2Col increases the amount of data in the input feature map during the unfolding process. Especially when the input convolution kernel is large, it will require a large amount of additional memory space to store the input feature map, resulting in a significant additional computational load. Therefore, it is not suitable for convolution optimization in scenarios such as embedded devices with limited computing resources. Summary of the Invention

[0006] This invention provides a method, apparatus, electronic device, and storage medium for accelerating neural network models, which can effectively improve the computational performance of convolutional neural network models without introducing too much additional memory overhead.

[0007] According to one aspect of the present invention, a method for accelerating a neural network model is provided, comprising:

[0008] In response to the acceleration event of the neural network model being triggered, the trained convolutional neural network model is obtained;

[0009] For each two-dimensional convolution operator in the convolutional neural network model, the two-dimensional convolution operator is split into at least two one-dimensional convolution operators; wherein the width of the input feature map of the one-dimensional convolution operator is the same as the width of the initial convolution kernel of the two-dimensional convolution operator;

[0010] Based on the initial convolutional kernel, during the sliding process of the input feature map in the height direction, the initial convolutional kernel is expanded in the height dimension at each sliding time to generate the target convolutional kernel;

[0011] The target convolution kernel is unfolded into two-dimensional convolution kernel data, and the two-dimensional convolution kernel data and the bias data corresponding to the two-dimensional convolution operator are mapped to the storage array.

[0012] The input feature map is processed based on the at least two one-dimensional convolution operators and the storage array.

[0013] According to another aspect of the present invention, an acceleration device for a neural network model is provided, comprising:

[0014] The neural network model acquisition module is used to acquire the trained convolutional neural network model in response to the acceleration event of the neural network model being triggered.

[0015] A two-dimensional convolution operator splitting module is used to split each two-dimensional convolution operator of the convolutional neural network model into at least two one-dimensional convolution operators; wherein the width of the input feature map of the one-dimensional convolution operator is the same as the width of the initial convolution kernel of the two-dimensional convolution operator.

[0016] The convolutional kernel expansion module is used to expand the initial convolutional kernel in the height dimension during each slide of the input feature map based on the initial convolutional kernel, thereby generating the target convolutional kernel;

[0017] The convolutional data mapping module is used to unfold the target convolutional kernel into two-dimensional convolutional kernel data, and map the two-dimensional convolutional kernel data and the bias data corresponding to the two-dimensional convolutional operator to the storage array.

[0018] The feature map processing module is used to process the input feature map based on the at least two one-dimensional convolution operators and the storage array.

[0019] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0020] At least one processor; and

[0021] A memory communicatively connected to the at least one processor; wherein,

[0022] The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to execute the acceleration method of the neural network model according to any embodiment of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the acceleration method of the neural network model according to any embodiment of the present invention.

[0024] An acceleration scheme for a neural network model according to an embodiment of the present invention includes: in response to an acceleration event of the neural network model being triggered, acquiring a trained convolutional neural network model; for each two-dimensional convolution operator of the convolutional neural network model, splitting the two-dimensional convolution operator into at least two one-dimensional convolution operators; wherein the width of the input feature map of the one-dimensional convolution operator is the same as the width of the initial convolution kernel of the two-dimensional convolution operator; based on the initial convolution kernel sliding in the height direction of the input feature map, expanding the initial convolution kernel in the height dimension at each sliding to generate a target convolution kernel; unfolding the target convolution kernel into two-dimensional convolution kernel data, and mapping the two-dimensional convolution kernel data and the bias data corresponding to the two-dimensional convolution operator to a storage array; and processing the input feature map based on the at least two one-dimensional convolution operators and the storage array. Through the technical solution provided by this embodiment of the invention, the computational performance of the convolutional neural network model can be effectively improved without introducing too much additional memory overhead, meeting the requirements of convolutional neural network models in low-power terminal device application scenarios.

[0025] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0026] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0027] Figure 1 This is a flowchart of a method for accelerating a neural network model according to Embodiment 1 of the present invention;

[0028] Figure 2 This is a schematic diagram of the sliding window calculation process of the Conv2d operator provided in an embodiment of the present invention;

[0029] Figure 3 This is a schematic diagram illustrating the parallel sliding window process of an input feature map in the height H direction, as provided in an embodiment of the present invention.

[0030] Figure 4This is a schematic diagram of convolution kernel expansion provided in an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram illustrating the processing procedure of an input feature map according to an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of the structure of an acceleration device for a neural network model according to Embodiment 2 of the present invention;

[0033] Figure 7 This is a schematic diagram of the structure of an electronic device that implements the acceleration method for the neural network model in the embodiments of the present invention. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] Example 1

[0037] Figure 1 This is a flowchart illustrating a method for accelerating a neural network model according to Embodiment 1 of the present invention. This embodiment is applicable to accelerating the processing of input feature maps in a neural network model. The method can be executed by an acceleration device for the neural network model, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes:

[0038] S110. In response to the acceleration event of the neural network model being triggered, the trained convolutional neural network model is obtained.

[0039] In this embodiment of the invention, when an acceleration instruction for the neural network model is received, it is determined that an acceleration event for the neural network model has been triggered. Optionally, it is determined that an acceleration event for the neural network model has been triggered when it is detected that the user has enabled Conv2D optimization and multi-point optimization switches in a custom Protobuf optimization configuration file. In response to the triggering of the acceleration event for the neural network model, a trained convolutional neural network model is obtained. The convolutional neural network model can be a quantized neural network model trained using an open-source deep learning framework such as PyTorch. Optionally, the convolutional neural network model structure can be serialized to ONNX format using Protobuf tools; that is, the convolutional neural network model can be stored in ONNX mode.

[0040] S120. For each two-dimensional convolution operator of the convolutional neural network model, the two-dimensional convolution operator is split into at least two one-dimensional convolution operators; wherein the width of the input feature map of the one-dimensional convolution operator is the same as the width of the initial convolution kernel of the two-dimensional convolution operator.

[0041] In this embodiment of the invention, the convolutional neural network model may contain one or more two-dimensional convolution operators (i.e., Conv2D operators). It should be noted that this embodiment of the invention does not limit the number of two-dimensional convolution operators contained in the convolutional neural network model. Figure 2 This is a schematic diagram illustrating the sliding window calculation process of the Conv2d operator provided in an embodiment of the present invention. Figure 2 As shown, the convolution kernel performs sliding window computation in the width dimension (i.e., W dimension) of the input feature map. According to the sliding window computation principle of the Conv2d operator, each two-dimensional convolution operator of the convolutional neural network model is split into at least two one-dimensional convolution operators (i.e., Conv1d operators). The splitting principle is to make the size of the width dimension (i.e., W dimension) of the input feature map of each Conv1d operator after splitting equal to the width K_W of the convolution kernel of the two-dimensional convolution operator. This can meet the data volume requirements of a single window computation in the W direction.

[0042] Optionally, if padding[pad_top,pad_left,pad_bottom,pad_right] is required in the W-dimensional direction of the input feature map of the 2D convolution operator (i.e., padding with zeros, where pad_top,pad_left,pad_bottom, and pad_right represent the number of zeros padded at the top, left, bottom, and right of the input feature map, respectively), then for the first sliding window split in the W-dimensional direction, the left side padding with zeros is pad_left_first = pad_left, and the right side padding with zeros is pad_right_first = 0; for the last sliding window split, the left side padding with zeros is pad_left_last = 0, and the right side padding with zeros is pad_right_last = pad_right; and for the middle sliding window splits, the left side padding with zeros is pad_left and the right side padding with zeros is pad_right. Optionally, since the input feature map of the split one-dimensional convolution operator does not require sliding window calculation in the W-dimensional direction, the stride_w of the split one-dimensional convolution operator is updated to 0 in the width direction. Optionally, the number of one-dimensional convolution operators Conv1d obtained after splitting the two-dimensional convolution operator Conv2d is the same as the width of the output feature map of the two-dimensional convolution operator, that is, the number of one-dimensional convolution operators Conv1d obtained after splitting the two-dimensional convolution operator Conv2d is equal to the size of the output feature map of the two-dimensional convolution operator in the W-dimensional direction.

[0043] Optionally, the channel order of the input feature map is [N, W, H, C]. This ensures that after Conv2d is converted to Conv1d, the input feature map maintains the continuity of memory storage even after being split in the W dimension.

[0044] S130. During the sliding process of the input feature map based on the initial convolutional kernel, the initial convolutional kernel is expanded in the height dimension at each sliding to generate the target convolutional kernel.

[0045] After splitting the Conv2d operator into multiple Conv1d operators through S120, for the computation of the input feature map of each Conv1d operator, due to the continuity of memory storage of the input feature map of the Conv1d operator when sliding windowing in the H-dimensional direction, it is advisable to expand the convolution kernel so that a single sliding window can complete the computation of multiple windows in the H-dimensional direction, thereby improving the parallelism of computation. Figure 3This is a schematic diagram illustrating the parallel sliding window process of an input feature map in the height H direction, as provided in an embodiment of the present invention. Assuming the initial convolution kernel size of the two-dimensional convolution operator is 5*2, and the convolution stride_h = 2 in the height direction, the sliding process of the initial convolution kernel in the H direction of the input feature map is as follows: Figure 3 As shown. Assuming the multi-point parallelism in the H direction is 4, meaning that the data volume of four sliding windows is calculated in a single operation, the corresponding new single sliding window feature map data is as follows: Figure 3 The rightmost image shows an 11*2 sliding window feature map. To improve the processing speed of the convolutional neural network model on the input feature map, during the sliding motion of the initial convolutional kernel along the height direction of the input feature map, the initial convolutional kernel is expanded in the height dimension at each slide to generate a target convolutional kernel. In other words, each time the initial convolutional kernel slides along the height direction of the input feature map, a corresponding target convolutional kernel is generated; that is, the number of target convolutional kernels is the same as the number of times the initial convolutional kernel slides along the height direction of the input feature map.

[0046] Optionally, based on the initial convolutional kernel sliding along the height direction of the input feature map, the initial convolutional kernel is expanded in the height dimension for each slide to generate a target convolutional kernel. This includes: obtaining the height-direction convolutional stride and a pre-set multi-point parallelism corresponding to the initial convolutional kernel of the two-dimensional convolution operator; based on the initial convolutional kernel sliding along the height direction of the input feature map, expanding the initial convolutional kernel in the height dimension for each slide according to the height-direction convolutional stride and the multi-point parallelism to generate the target convolutional kernel. Wherein, the height-direction convolutional stride is the step size of each slide when the initial convolutional kernel slides along the height direction of the input feature map, such as stride_h = 2. The multi-point parallelism is the amount of data completed in a single sliding window, such as multi_point = 4. The height-direction convolution stride_h and multi-point parallelism multi_point are obtained. Based on the height-direction convolution stride_h and multi-point parallelism multi_point, the dimension of the expanded target convolutional kernel in the height direction and the number of zeros to be padded at the top and bottom of the initial convolutional kernel are determined. This is used to expand the initial convolutional kernel for each slide, generating the corresponding target convolutional kernel. Optionally, the dimension of the target convolutional kernel in the height dimension is the sum of the product of the height-direction convolution stride and the multi-point parallelism multi_point minus 1, and the dimension of the initial convolutional kernel in the height dimension. For example, the initial convolution kernel is a 5*2 convolution kernel, that is, the size of the initial convolution kernel in the height dimension is k_h = 5. If the convolution stride in the height direction is stride_h = 2 and the multipoint parallelism is multi_point = 4, then the size of the target convolution kernel in the height dimension is: k_h + (multi_point - 1) * stride_h = 5 + (4 - 1) * 2 = 11, that is, the target convolution kernel is an 11*2 convolution kernel.

[0047] Optionally, during the sliding process of the initial convolutional kernel in the height direction of the input feature map, the initial convolutional kernel is expanded in the height dimension at each slide according to the height direction convolution stride and the multi-point parallelism to generate a target convolutional kernel. This includes: determining the current sliding number of the initial convolutional kernel during the sliding process of the initial convolutional kernel in the height direction of the input feature map; determining a first number of zeros to fill the top of the initial convolutional kernel in the height dimension according to the current sliding number and the height direction convolution stride; determining a second number of zeros to fill the bottom of the initial convolutional kernel in the height dimension according to the current sliding number, the height direction convolution stride, and the multi-point parallelism; and filling the top of the initial convolutional kernel with the first number of zeros and the bottom of the initial convolutional kernel with the second number of zeros in the height dimension to generate the target convolutional kernel corresponding to the current sliding number.

[0048] Figure 4 This is a schematic diagram of a convolution kernel expansion provided in an embodiment of the present invention. Figure 4 As shown, based on the calculation relationship of the input feature map kernel and the convolution kernel, Figure 4 The process of expanding the convolutional kernel is shown. By expanding the convolutional kernel in the out_channel dimension, the computation results of multiple points are obtained in a single sliding window. During the sliding process of the initial convolutional kernel in the height direction of the input feature map, the current sliding number idx of the initial convolutional kernel is determined, where idx = 1, 2, ..., multi_point. Then, based on the current sliding number idx and the height direction convolution stride_h, the first number of zeros to be padded at the top of the initial convolutional kernel in the height dimension is determined, where the first number is stride_h*(idx-1). Based on the current sliding number idx, the height direction convolution stride_h, and the multi-point parallelism multi_point, the second number of zeros to be padded at the bottom of the initial convolutional kernel in the height dimension is determined, where the second number is stride_h*(multi_point-idx). In the height dimension, the first number of zeros is padded at the top of the initial convolutional kernel, and the second number of zeros is padded at the bottom of the initial convolutional kernel, generating the target convolutional kernel corresponding to the current sliding number. For example, the target convolutional kernels that slide four times along the height direction of the input feature map as shown below. Figure 4 The four convolution kernels on the right are shown.

[0049] S140. Unfold the target convolution kernel into two-dimensional convolution kernel data, and map the two-dimensional convolution kernel data and the bias data corresponding to the two-dimensional convolution operator to the storage array.

[0050] In this embodiment of the invention, the target convolution kernel is unfolded into two-dimensional convolution kernel data, wherein the two-dimensional convolution kernel data is [c_in*k_w*k_h,c_out], that is, the vertical dimension data of the two-dimensional convolution kernel data is the product of the C-dimensional data, H-dimensional data and W-dimensional data of the target convolution kernel, and the horizontal dimension data of the two-dimensional convolution kernel data is the N-dimensional data of the target convolution kernel. The two-dimensional convolution kernel data and the bias data corresponding to the two-dimensional convolution operator are mapped into an in-memory array.

[0051] S150. The input feature map is processed based on the at least two one-dimensional convolution operators and the storage array.

[0052] In this embodiment of the invention, the input feature map is processed based on at least two one-dimensional convolution operators and an in-memory array. Optionally, the processing of the input feature map based on the at least two one-dimensional convolution operators and the in-memory array includes: transmitting the input feature map of each of the at least two one-dimensional convolution operators to the in-memory array to perform parallel operations on the input feature map and each column of data in the in-memory array. Figure 5 This is a schematic diagram illustrating the processing procedure of an input feature map provided in an embodiment of the present invention. For example... Figure 5 As shown, after the target convolutional kernel is unfolded into two-dimensional convolutional kernel data and arranged in the in-memory array, the input feature map ifm (i.e., the data of each sliding window of the input feature map of the two-dimensional convolutional operator) of at least two one-dimensional convolutional operators is transmitted to the in-memory array to perform parallel operations on the input feature map and each column of data in the in-memory array. That is, the in-memory array can simultaneously complete the multiplication and addition calculations of the input feature map, multiple columns of convolutional kernel data, and corresponding bias data, and the multiplication and addition operations of each column of data in the in-memory array are also parallel, which can greatly improve the parallelism of the convolutional neural network model computation.

[0053] In some embodiments, after mapping the two-dimensional convolution kernel data and the bias data corresponding to the two-dimensional convolution operator to the in-memory array, the method further includes: determining the first memory address of the input feature map and the second memory address of the output feature map of the at least two one-dimensional convolution operators; determining the mapping space position of the two-dimensional convolution kernel data and the corresponding bias data in the in-memory array; generating a configuration file of the convolutional neural network model based on the first memory address, the second memory address, and the mapping space position, and storing the configuration file. Specifically, the memory address of the input feature map and the memory address of the output feature map of each of the at least two one-dimensional convolution operators are determined, and the mapping space position of the two-dimensional convolution kernel data and the corresponding bias data in the in-memory array is determined. Then, a configuration file of the convolutional neural network model is generated based on the memory address of the input feature map, the memory address of the output feature map, and the mapping space position of each one-dimensional convolution operator, and the configuration file is stored. The configuration file of the convolutional neural network model can be a binary file. The configuration file and the convolutional neural network model are loaded onto the terminal device. When an application event of the neural network model is detected, parallel computation is performed on the data to be processed based on the configuration file of the convolutional neural network model, thereby improving the speed of data processing by the convolutional neural network model.

[0054] An acceleration method for a neural network model according to an embodiment of the present invention includes: in response to an acceleration event of the neural network model being triggered, acquiring a trained convolutional neural network model; for each two-dimensional convolution operator of the convolutional neural network model, splitting the two-dimensional convolution operator into at least two one-dimensional convolution operators; wherein the width of the input feature map of the one-dimensional convolution operator is the same as the width of the initial convolution kernel of the two-dimensional convolution operator; based on the initial convolution kernel sliding in the height direction of the input feature map, expanding the initial convolution kernel in the height dimension at each sliding to generate a target convolution kernel; unfolding the target convolution kernel into two-dimensional convolution kernel data, and mapping the two-dimensional convolution kernel data and the bias data corresponding to the two-dimensional convolution operator to an in-memory array; and processing the input feature map based on the at least two one-dimensional convolution operators and the in-memory array. The technical solution provided by this invention reduces the number of sliding window operations for the convolution kernel by converting the Conv2d operator to the Conv1d operator. This overcomes the technical challenge of increased data storage and computation caused by the expansion of the input feature map in the Img2Col optimization method of convolutional neural networks. In addition, based on multi-point optimization enabled by the in-memory array, it overcomes the technical challenge of high power consumption when deploying convolutional neural networks on GPU hardware, which makes them unsuitable for embedded low-power terminal devices, while ensuring parallel computing performance. Without introducing too much additional memory overhead, it can effectively improve the computational performance of the convolutional neural network model and meet the requirements of convolutional neural network models in low-power terminal device application scenarios.

[0055] Example 2

[0056] Figure 6 This is a schematic diagram of the structure of an acceleration device for a neural network model provided in Embodiment 2 of the present invention. Figure 6 As shown, the device includes:

[0057] The neural network model acquisition module 610 is used to acquire the trained convolutional neural network model in response to the acceleration event of the neural network model being triggered.

[0058] The two-dimensional convolution operator splitting module 620 is used to split each two-dimensional convolution operator of the convolutional neural network model into at least two one-dimensional convolution operators; wherein the width of the input feature map of the one-dimensional convolution operator is the same as the width of the initial convolution kernel of the two-dimensional convolution operator.

[0059] The convolution kernel expansion module 630 is used to expand the initial convolution kernel in the height dimension during each sliding process of the input feature map based on the initial convolution kernel to generate the target convolution kernel.

[0060] The convolutional data mapping module 640 is used to unfold the target convolutional kernel into two-dimensional convolutional kernel data, and map the two-dimensional convolutional kernel data and the bias data corresponding to the two-dimensional convolutional operator to the storage array.

[0061] The feature map processing module 650 is used to process the input feature map based on the at least two one-dimensional convolution operators and the storage array.

[0062] Optionally, the convolution kernel expansion module includes:

[0063] A multi-point parallelism acquisition unit is used to acquire the height direction convolution stride and the pre-set multi-point parallelism of the initial convolution kernel of the two-dimensional convolution operator.

[0064] The convolution kernel expansion unit is used to expand the initial convolution kernel in the height dimension during the sliding process of the input feature map based on the initial convolution kernel, according to the convolution stride in the height direction and the multi-point parallelism, to generate the target convolution kernel.

[0065] Optionally, the convolution kernel expansion unit is used for:

[0066] Based on the initial convolutional kernel sliding in the height direction of the input feature map, the current number of slides of the initial convolutional kernel is determined;

[0067] Based on the current number of slides and the convolution stride in the height direction, determine the first number of zeros to be padded at the top of the initial convolution kernel in the height dimension;

[0068] Based on the current number of slides, the height-direction convolution stride, and the multi-point parallelism, determine a second number of zeros to pad the bottom of the initial convolution kernel in the height dimension;

[0069] In the height dimension, the first number of 0s is filled at the top of the initial convolutional kernel, and the second number of 0s is filled at the bottom of the initial convolutional kernel to generate the target convolutional kernel corresponding to the current number of slides.

[0070] Optionally, the size of the target convolutional kernel in the height dimension is the sum of the product of the height-direction convolution stride and the multi-point parallelism minus 1, and the size of the initial convolutional kernel in the height dimension.

[0071] Optionally, the number of one-dimensional convolution operators is the same as the width of the output feature map of the two-dimensional convolution operator.

[0072] Optionally, the feature map processing module is used for:

[0073] The input feature map of each of the at least two one-dimensional convolution operators is transmitted to the in-memory array to perform parallel operations on the input feature map and each column of data in the in-memory array.

[0074] Optionally, the device further includes:

[0075] The memory address determination module is used to determine the first memory address of the input feature map and the second memory address of the output feature map of the at least two one-dimensional convolution operators after mapping the two-dimensional convolution kernel data and the bias data corresponding to the two-dimensional convolution operator to the memory array.

[0076] The mapping space location determination module is used to determine the mapping space location of the two-dimensional convolution kernel data and the corresponding bias data in the in-memory array;

[0077] The configuration file generation module is used to generate a configuration file for the convolutional neural network model based on the first memory address, the second memory address, and the mapping space location, and to store the configuration file.

[0078] The neural network model acceleration device provided in the embodiments of the present invention can execute the neural network model acceleration method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0079] Example 3

[0080] Figure 7 A schematic diagram of an electronic device 10 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0081] like Figure 7As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0082] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0083] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as methods for accelerating neural network models.

[0084] In some embodiments, the method for accelerating the neural network model can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for accelerating the neural network model described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to execute the method for accelerating the neural network model by any other suitable means (e.g., by means of firmware).

[0085] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0086] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0087] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0088] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0089] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or computing systems that include middleware components (e.g., application servers), or computing systems that include frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0090] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0091] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0092] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method for accelerating a neural network model, characterized in that, include: In response to the acceleration event of the neural network model being triggered, the trained convolutional neural network model is obtained; For each two-dimensional convolution operator in the convolutional neural network model, the two-dimensional convolution operator is split into at least two one-dimensional convolution operators; wherein the width of the input feature map of the one-dimensional convolution operator is the same as the width of the initial convolution kernel of the two-dimensional convolution operator; Based on the initial convolutional kernel, during the sliding process of the input feature map in the height direction, the initial convolutional kernel is expanded in the height dimension at each sliding time to generate the target convolutional kernel; The target convolution kernel is unfolded into two-dimensional convolution kernel data, and the two-dimensional convolution kernel data and the bias data corresponding to the two-dimensional convolution operator are mapped to the storage array. The input feature map is processed based on the at least two one-dimensional convolution operators and the storage array.

2. The method according to claim 1, characterized in that, Based on the initial convolutional kernel, during the sliding process of the input feature map in the height direction, the initial convolutional kernel is expanded in the height dimension at each sliding step to generate the target convolutional kernel, including: Obtain the height direction convolution stride and the pre-set multi-point parallelism of the initial convolution kernel of the two-dimensional convolution operator; During the sliding process of the input feature map based on the initial convolutional kernel, the initial convolutional kernel is expanded in the height dimension according to the height direction convolution stride and the multi-point parallelism to generate the target convolutional kernel.

3. The method according to claim 2, characterized in that, During the sliding process of the input feature map based on the initial convolutional kernel, the target convolutional kernel is expanded in the height dimension at each sliding step according to the height direction convolution stride and the multi-point parallelism, including: Based on the initial convolutional kernel sliding in the height direction of the input feature map, the current number of slides of the initial convolutional kernel is determined; Based on the current number of slides and the convolution stride in the height direction, determine the first number of zeros to be padded at the top of the initial convolution kernel in the height dimension; Based on the current number of slides, the height-direction convolution stride, and the multi-point parallelism, determine the second number of zeros to be padded at the bottom of the initial convolution kernel in the height dimension; In the height dimension, the first number of 0s is filled at the top of the initial convolutional kernel, and the second number of 0s is filled at the bottom of the initial convolutional kernel to generate the target convolutional kernel corresponding to the current number of slides.

4. The method according to claim 2, characterized in that, The size of the target convolutional kernel in the height dimension is the sum of the product of the height-direction convolution stride and the multi-point parallelism minus 1, and the size of the initial convolutional kernel in the height dimension.

5. The method according to claim 1, characterized in that, The number of one-dimensional convolution operators is the same as the width of the output feature map of the two-dimensional convolution operator.

6. The method according to claim 1, characterized in that, The input feature map is processed based on the at least two one-dimensional convolution operators and the in-memory array, including: The input feature map of each of the at least two one-dimensional convolution operators is transmitted to the in-memory array to perform parallel operations on the input feature map and each column of data in the in-memory array.

7. The method according to claim 1, characterized in that, After mapping the two-dimensional convolution kernel data and the bias data corresponding to the two-dimensional convolution operator to the in-memory array, the method further includes: Determine the first memory address of the input feature map and the second memory address of the output feature map of the at least two one-dimensional convolution operators respectively; Determine the location of the two-dimensional convolution kernel data and the corresponding bias data in the mapping space of the in-memory array; The configuration file of the convolutional neural network model is generated based on the first memory address, the second memory address, and the mapping space location, and the configuration file is stored.

8. An acceleration device for a neural network model, characterized in that, include: The neural network model acquisition module is used to acquire the trained convolutional neural network model in response to the acceleration event of the neural network model being triggered. A two-dimensional convolution operator splitting module is used to split each two-dimensional convolution operator of the convolutional neural network model into at least two one-dimensional convolution operators; wherein the width of the input feature map of the one-dimensional convolution operator is the same as the width of the initial convolution kernel of the two-dimensional convolution operator. The convolutional kernel expansion module is used to expand the initial convolutional kernel in the height dimension during each slide of the input feature map based on the initial convolutional kernel, thereby generating the target convolutional kernel; The convolutional data mapping module is used to unfold the target convolutional kernel into two-dimensional convolutional kernel data, and map the two-dimensional convolutional kernel data and the bias data corresponding to the two-dimensional convolutional operator to the storage array. The feature map processing module is used to process the input feature map based on the at least two one-dimensional convolution operators and the storage array.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method for accelerating the neural network model according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the method for accelerating the neural network model according to any one of claims 1-7.

Images