Neural network processing device and method

Abstract

The present application relates to the technical field of neural networks, and provides a neural network processing device and method, used for avoiding the memory wall problem and power consumption waste, thereby improving the processing efficiency of neural networks. The device is used for processing a neural network comprising a plurality of convolutional layers. The device comprises a buffer, a scheduler, and a plurality of computing units. The buffer comprises a plurality of cache spaces in one-to-one correspondence with the plurality of convolutional layers. The scheduler is used for enabling, when a first convolutional layer among the plurality of convolutional layers meets a preset condition, at least one computing unit corresponding to the first convolutional layer in a first period among a plurality of computing periods. The at least one computing unit corresponding to the first convolutional layer is used for reading at least one sub-input feature map from a first cache space in the first period, computing the at least one sub-input feature map to obtain a sub-output feature map of the first convolutional layer, and caching the sub-output feature map into a second cache space as a sub-input feature map of the next convolutional layer of the first convolutional layer.

Description

A neural network processing device and method

[0001] The present application claims priority from the Chinese patent application No. 202410642854.3 filed on May 22, 2024, and entitled "A neural network processing device and method", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the technical field of neural networks, and in particular to a neural network processing device and method. BACKGROUND

[0003] The core of artificial intelligence (AI) technology includes neural network algorithms, computing units providing hardware computing power, and hardware schedulers coordinating the processing of complex data.

[0004] At present, in the traditional scheduling mode of neural networks, hardware computing resources need to be allocated in the order of convolution layers in the neural network. Specifically, for each convolution layer, when scheduling in the traditional way, the input feature map of the convolution layer needs to be cut into multiple rows and multiple columns of cut data, and each cut data of the input feature map of the convolution layer needs to be repeatedly transported between the internal memory and the external memory. However, with the upgrading of deep learning neural network algorithms, the amount of data that needs to be processed by the hardware scheduler increases exponentially, so the above scheduling mode will cause the memory wall problem and result in a large waste of power consumption, which seriously restricts the performance of the neural network system.

[0005] Therefore, how to avoid the memory wall problem and power consumption waste, and improve the energy efficiency of the neural network processing device, has become a technical problem to be solved. SUMMARY

[0006] The present application provides a neural network processing device and method for avoiding the memory wall problem and power consumption waste, improving the processing efficiency of the neural network, and further improving the energy efficiency of the neural network processing device.

[0007] To achieve the above-mentioned purpose, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, a neural network processing apparatus is provided for processing a neural network comprising a plurality of convolution layers, the apparatus comprising: a buffer, a scheduler and a plurality of computing units; each of the plurality of convolution layers corresponds to at least one computing unit of the plurality of computing units in any of a plurality of computing periods; the buffer comprises a plurality of buffer spaces corresponding to the plurality of convolution layers one by one, each buffer space being configured to buffer at least one sub-input feature map of a corresponding convolution layer, each sub-input feature map being a part of an input feature map; the scheduler is configured to: enable the at least one computing unit corresponding to a first convolution layer of the plurality of convolution layers in a first period of the plurality of computing periods when the first convolution layer satisfies a preset condition; wherein the first convolution layer satisfies the preset condition comprises: a first buffer space corresponding to the first convolution layer is in a readable state in the first period (e.g., the first buffer space corresponding to the first convolution layer buffers a certain number of sub-input feature maps of the first convolution layer), a second buffer space corresponding to a next convolution layer of the first convolution layer is in a writable state in the first period (e.g., a capacity of a remaining space of the second buffer space is greater than or equal to a data amount of a sub-output feature map of the first convolution layer), and the at least one computing unit corresponding to the first convolution layer is in an idle state in the first period; and the at least one computing unit corresponding to the first convolution layer is configured to: read the at least one sub-input feature map from the first buffer space in the first period, perform computation on the at least one sub-input feature map to obtain a sub-output feature map of the first convolution layer, and buffer the sub-output feature map as a sub-input feature map of the next convolution layer of the first convolution layer in the second buffer space.

[0009] In the above technical solution, taking the computation of the first convolution layer as an example, the operation of reading the at least one input feature map from the first buffer space by the at least one computing unit can be regarded as a data-consuming process, and the operation of buffering the sub-output feature map obtained by computation as a sub-input feature map of the next convolution layer of the first convolution layer in the second buffer space can be regarded as a data-generating process. When different convolution layers of the plurality of convolution layers are computed in parallel, the data of the sub-output feature map of a previous convolution layer as a sub-input feature map of a next convolution layer is gradually consumed in the process of gradually generating the sub-output feature map of the previous convolution layer, so that the inter-layer data (i.e., the sub-input feature maps and the sub-output feature maps of the plurality of convolution layers) can be consumed in time after being generated, and a large amount of inter-layer data does not need to be buffered in the buffer, thereby saving the buffer space of the buffer. In addition, during the entire computation process, the inter-layer data of the corresponding convolution layer is generated and consumed between the plurality of computing units and the buffer, thereby avoiding the process of moving the inter-layer data between the external memory and the internal buffer of the neural network, reducing the power consumption caused by data movement, and thereby improving the energy efficiency of the neural network system.

[0010] In any possible implementation form of the first aspect, the scheduler is further configured to determine, according to the resource mapping information, a first cache space corresponding to the first convolution layer, a second cache space corresponding to a next convolution layer of the first convolution layer, and at least one computing unit corresponding to the first convolution layer in the first period; wherein the resource mapping information is used to indicate a cache space corresponding to each convolution layer of the plurality of convolution layers, and at least one computing unit corresponding to each convolution layer in each period of the plurality of computing periods. Optionally, the resource mapping information can be generated offline and stored in the device, and the scheduler can obtain it when in use, which can reduce the consumption of hardware resources. The above possible implementation form enables the scheduler to accurately enable the computing units corresponding to different convolution layers according to the resource mapping relationship, so as to maximize the guarantee that all computing units are in a working state, thereby avoiding waste of computing resources and improving computing efficiency.

[0011] In any possible implementation form of the first aspect, each convolution layer of the plurality of convolution layers corresponds to a read pointer and a write pointer, the read pointer is used to read at least one sub-input feature map of each convolution layer in the cache space corresponding to each convolution layer, and the write pointer is used to write a sub-output feature map of each convolution layer in the cache space corresponding to a next convolution layer of each convolution layer; the scheduler is further configured to update the read pointer corresponding to the first convolution layer when the at least one computing unit corresponding to the first convolution layer reads the at least one sub-input feature map from the first cache space, and update the write pointer corresponding to the first convolution layer when the at least one computing unit corresponding to the first convolution layer caches the sub-output feature map of the first convolution layer into the second cache space. The above possible implementation form updates the read pointer and the write pointer corresponding to the first convolution layer in a timely manner, which is conducive to the scheduler to determine in a timely manner whether the corresponding convolution layer meets the preset condition, so as to enable the plurality of computing units to perform computation on different convolution layers in a timely manner.

[0012] In any possible implementation form of the first aspect, each computing unit of the at least one computing unit corresponds to a computing flag bit, which is used to indicate a state of each computing unit; the state of each computing unit includes a working state or an idle state. The above possible implementation form enables the scheduler to obtain the state of each computing unit according to the computing flag bit of the computing unit, so as to accurately enable the computing unit in the idle state and to avoid waste of computing resources to a certain extent.

[0013] In any possible implementation form of the first aspect, when there are at least two convolutional layers in the plurality of convolutional layers satisfying the preset condition, and the at least two convolutional layers correspond to a same computing unit in the plurality of computing units; the scheduler is further configured to: select one convolutional layer from the at least two convolutional layers as the first convolutional layer according to a priority order of the at least two convolutional layers. The above possible implementation form enables the scheduler to solve the problem of computing resource conflict in time, and consume the inter-layer data generated by parallel computation of the plurality of convolutional layers as early as possible, so as to further reduce the required cache space of the inter-layer data.

[0014] In any possible implementation form of the first aspect, in the at least two convolutional layers, the convolutional layer with a larger number of layers has a higher priority. The above possible implementation form enables the scheduler to enable the computing unit to process the convolutional layer with a larger number of layers preferentially, so that the scheduler can solve the problem of computing resource conflict in time, and consume the inter-layer data generated by parallel computation of the plurality of convolutional layers as early as possible, so as to further reduce the required cache space of the inter-layer data.

[0015] In any possible implementation form of the first aspect, the cache, the scheduler and the plurality of computing units can be integrated in one chip. For example, the neural network processing apparatus is a processor or a processing chip, and the cache, the scheduler and the plurality of computing units are integrated in the processor or the processing chip. Optionally, the cache can be an on-chip cache.

[0016] In a second aspect, a neural network processing method is provided. The method is applied to a neural network processing apparatus. The neural network includes a plurality of convolution layers. The apparatus includes a buffer, a scheduler and a plurality of computing units. Each of the plurality of convolution layers corresponds to at least one computing unit of the plurality of computing units in any of a plurality of computing periods. The buffer includes a plurality of buffer spaces corresponding to the plurality of convolution layers one by one. Each buffer space is configured to buffer at least one sub-input feature map of a corresponding convolution layer. Each sub-input feature map is a part of an input feature map. The method includes: when a first convolution layer of the plurality of convolution layers satisfies a preset condition, enabling, by the scheduler, the at least one computing unit corresponding to the first convolution layer in a first period of the plurality of computing periods. The first convolution layer satisfies the preset condition includes that: a first buffer space corresponding to the first convolution layer is in a readable state in the first period, a second buffer space corresponding to a next convolution layer of the first convolution layer is in a writable state in the first period, and the at least one computing unit corresponding to the first convolution layer in the first period is in an idle state. In the first period, the at least one computing unit corresponding to the first convolution layer reads the at least one sub-input feature map from the first buffer space, performs computation on the at least one sub-input feature map to obtain a sub-output feature map of the first convolution layer, and caches the sub-output feature map as a sub-input feature map of the next convolution layer of the first convolution layer in the second buffer space.

[0017] In any possible implementation form of the second aspect, the method further includes: determining, by the scheduler, the first buffer space corresponding to the first convolution layer, the second buffer space corresponding to the next convolution layer of the first convolution layer, and the at least one computing unit corresponding to the first convolution layer in the first period according to resource mapping information. The resource mapping information is configured to indicate the buffer space corresponding to each of the plurality of convolution layers and the at least one computing unit corresponding to each of the plurality of convolution layers in each of the plurality of computing periods.

[0018] In any possible implementation form of the second aspect, each of the plurality of convolution layers corresponds to a read pointer and a write pointer. The read pointer is configured to read the at least one sub-input feature map of each of the plurality of convolution layers in the buffer space corresponding to each of the plurality of convolution layers. The write pointer is configured to write a sub-output feature map of each of the plurality of convolution layers in the buffer space corresponding to a next convolution layer of each of the plurality of convolution layers. The method further includes: updating, by the scheduler, the read pointer corresponding to the first convolution layer when the at least one computing unit corresponding to the first convolution layer reads the at least one sub-input feature map from the first buffer space. Updating, by the scheduler, the write pointer corresponding to the first convolution layer when the at least one computing unit corresponding to the first convolution layer caches the sub-output feature map of the first convolution layer in the second buffer space.

[0019] In any possible implementation manner of the second aspect, each of the at least one computing unit corresponds to a computing flag bit, used to indicate a state of each computing unit; the state of each computing unit comprises a working state or an idle state.

[0020] In any possible implementation manner of the second aspect, when there are at least two convolution layers in the plurality of convolution layers satisfying the preset condition, and the at least two convolution layers correspond to a same computing unit in the plurality of computing units, the method further comprises: the scheduler selects one convolution layer as the first convolution layer from the at least two convolution layers according to a priority order of the at least two convolution layers.

[0021] In any possible implementation manner of the second aspect, in the at least two convolution layers, a convolution layer with a larger layer number has a higher priority.

[0022] In a third aspect, an electronic device is provided, which comprises a processor and a memory. The processor can be the neural network processing apparatus provided in the first aspect or any possible implementation manner of the first aspect, and the memory can be an external memory. The memory stores instructions, and when the instructions are executed by the processor, the neural network processing method provided in the second aspect or any possible implementation manner of the second aspect is implemented.

[0023] In a fourth aspect, a computer readable storage medium is provided, which stores a computer program or instructions, and when the computer program or instructions are executed, the neural network processing method provided in the second aspect or any possible implementation manner of the second aspect is implemented.

[0024] In a fifth aspect, a computer program product is provided, which comprises a computer program (also referred to as code or instructions), and when the computer program is executed, the computer program makes the computer execute the neural network processing method provided in the second aspect or any possible implementation manner of the second aspect.

[0025] It can be understood that the beneficial effects achieved by the second aspect to the fifth aspect can correspond to the beneficial effects of the neural network processing method provided in the first aspect or any possible implementation manner of the first aspect, which will not be repeated here. BRIEF DESCRIPTION OF DRAWINGS

[0026] FIG. 1 is a schematic diagram of a neural network convolution layer computing characteristic provided by an embodiment of the present application;

[0027] FIG. 2 is a structural schematic diagram of input feature map tile data provided by an embodiment of the present application;

[0028] FIG. 3 is a schematic diagram of a calculation process of input feature map cut block data according to an embodiment of the present application;

[0029] FIG. 4 is a schematic diagram of data flow of serial calculation between neural network convolution layers according to an embodiment of the present application;

[0030] FIG. 5 is a schematic diagram of a structure of an electronic device according to an embodiment of the present application;

[0031] FIG. 6 is a schematic diagram of a structure of a neural network processing apparatus according to an embodiment of the present application;

[0032] FIG. 7 is a schematic diagram of another structure of input feature map cut block data according to an embodiment of the present application;

[0033] FIG. 8 is a schematic diagram of a flow of a neural network processing method according to an embodiment of the present application;

[0034] FIG. 9 is a schematic diagram of a mapping relationship between a convolution layer and a buffer and a calculation unit according to an embodiment of the present application;

[0035] FIG. 10 is a schematic diagram of a calculation process of multiple convolution layers according to an embodiment of the present application;

[0036] FIG. 11 is a schematic diagram of another flow of a neural network processing method according to an embodiment of the present application;

[0037] FIG. 12 is a schematic diagram of data flow of parallel calculation of neural network convolution layers according to an embodiment of the present application;

[0038] FIG. 13 is a schematic diagram of data flow of buffer management of neural network convolution layers according to an embodiment of the present application;

[0039] FIG. 14 is a schematic diagram of another data flow of parallel calculation of neural network convolution layers according to an embodiment of the present application;

[0040] FIG. 15 is a schematic diagram of another data flow of buffer management of neural network convolution layers according to an embodiment of the present application;

[0041] FIG. 16 is a schematic diagram of a structure of a scheduler according to an embodiment of the present application;

[0042] FIG. 17 is a schematic diagram of another structure of a scheduler according to an embodiment of the present application;

[0043] FIG. 18 is a schematic diagram of data flow of parallel scheduling performed by a scheduler according to an embodiment of the present application;

[0044] FIG. 19 is a schematic diagram of another data flow of parallel scheduling performed by a scheduler according to an embodiment of the present application;

[0045] FIG. 20 is a flow diagram of a scheduler enabling a computing unit according to an embodiment of the present application;

[0046] FIG. 21 is a flow diagram of another scheduler enabling a computing unit according to an embodiment of the present application. DETAILED DESCRIPTION

[0047] The making and using of various embodiments are discussed in detail below. It should be appreciated that the specific application provided herein is applicable as implemented in a variety of particular contexts. The particular embodiments discussed are merely illustrative of specific ways to make and use the application and the technology.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs.

[0049] Circuits or other components can be described as or said to be "configured to" perform a task or tasks, in instances of this usage, "configured to" is used to convey structuring of a system (e.g., a circuit) that performs tasks during operation. Accordingly, a circuit or component can be said to be configured to perform tasks even if a specified circuit / component is not currently operational (e.g., is not on). Circuits / components used in association with the term "configured to" include hardware-only circuits / components as well as a combination of hardware and software.

[0050] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the present application, "at least one" means one or more, and "multiple" means two or more. The association relationship of the associated objects is described by "and / or", which means that there can be three relationships, for example, A and / or B can represent the following cases: A exists alone, A and B exist together, and B exists alone, where A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single item or multiple items. For example, at least one of a, b or c can represent a, b, c, a and b, a and c, b and c, a, b and c; where a, b and c can be single or multiple.

[0051] The embodiments of the present application use "first" and "second" and the like to distinguish objects or functions or roles with similar names or functions or roles, and those skilled in the art can understand that "first" and "second" and the like do not limit the quantity and execution order. The word "coupled" is used to represent electrical connection, including direct connection through wires or connection terminals or indirect connection through other devices. Therefore, "coupled" should be regarded as a broad sense of electronic communication connection.

[0052] It should be noted that, in this application, the terms "exemplary" or "for example" are used to indicate that something is being described as an example, illustration, or illustration. Any embodiment or design described as "exemplary" or "for example" in this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0053] Before introducing the embodiments of this application, the application scenarios involved in this application will be described first.

[0054] The core of artificial intelligence (AI) technology includes neural network algorithms, computing units that provide hardware computing power, and hardware schedulers that coordinate the processing of massive amounts of data.

[0055] Neural networks require a large amount of convolutional computation. Using Figure 1 as an example, we will introduce the characteristics of convolutional computation in neural networks. In this paper, the input data of each convolutional layer in a neural network is called the input feature map, and the output data of each convolutional layer is called the output feature map. The output feature map of each convolutional layer is obtained by performing a convolution operation between the input feature map and the convolutional kernel of that convolutional layer. The input feature map or output feature map can also be simply referred to as a feature map. As shown in Figure 1, taking three adjacent convolutional layers as an example, assume that the feature map 1 of the first convolutional layer includes multiple rows (row, R) in the height direction H, which are, in order from top to bottom as shown in the figure, row 1 R0, row 2 R1, row 3 R2, and the last row Rn. Simultaneously, this feature map 1 includes multiple columns in the width direction W, which are, in order from left to right as shown in the figure, column 1, column 2, and so on until the last column. When performing convolution calculations in a neural network, taking the data of the first 3x3 grid (rows 1 to 3 and columns 1 to 3) in feature map 1 and the data of convolution kernel 1 of the first convolutional layer as an example, the convolution calculation result is 204 = 0×0 + 1×1 + 2×2 + 3×3 + 4×4 + 5×5 + 6×6 + 7×7 + 8×8. Then, "204" is used as the data of the first row and first column of feature map 2 in the second convolutional layer, and so on. The convolution calculation result of the first 3x3 grid in feature map 2 and the convolution kernel 2 of the second convolutional layer is the data of the first row and first column of feature map 3 in the third convolutional layer. In other words, the calculation result of the previous convolutional layer, or output feature map, is the input feature map of the current convolutional layer. It should be noted that the above-described segmentation method of feature map 1 is only for illustration and does not constitute a limitation on the embodiments of this application. This represents the convolution operation.

[0056] The processing procedure of the neural network will be explained below with reference to Figures 2 to 4.

[0057] For example, FIG. 2 shows a way of splitting the input feature map. As shown in FIG. 2, for each convolutional layer, when scheduled in a conventional manner, the input feature map of the convolutional layer needs to be split into multiple rows and multiple columns of patch data along the height direction H, the width direction W and the input channel direction Ci, such as patch 0, patch 1, …, patch n, wherein each patch data is stored in an external memory. Specifically, when calculating the patch 0 of the current convolutional layer, the patch 0 is first read from the external memory, and after the calculation is completed, the patch 0 of the next convolutional layer is obtained and output to the external memory. Each patch data of the current convolutional layer is calculated and processed in a similar manner, so that each patch data needs to be transported between the neural network and the external memory. After the calculation of each patch data of the current convolutional layer is completed, the calculation of the next convolutional layer can be started, and the calculation of the next convolutional layer is independent of the calculation of the current convolutional layer, and the patch data of the current convolutional layer cannot be reused in the calculation of the next convolutional layer.

[0058] FIG. 3 is a neural network processing device for processing a neural network. The neural network processing device is connected with an external memory, and the neural network processing device includes a plurality of internal buffer areas and a calculation unit, and each internal buffer area is connected with the external memory. Optionally, the neural network processing device can be a processor or a processing chip. The plurality of internal buffer areas can be on-chip cache, Cache or static random-access memory (SRAM), and the external memory can be dynamic random-access memory (DRAM) or flash memory, which are not limited in the embodiment.

[0059] The calculation process of the neural network is explained below by taking the device as an example. For example, taking the calculation process of the patch n in the input feature map of the i-th convolutional layer as an example, when the calculation of the patch n is started, the data of the patch n needs to be transported from the external memory to the internal buffer area 1 in the neural network, the calculation unit obtains the data of the patch n from the internal buffer area 1, and after the calculation is completed, the patch n of the input feature map of the i+1-th convolutional layer is output to the internal buffer area 2; then, the data of the patch n in the input feature map of the i+1-th convolutional layer is further transported to the corresponding storage space in the external buffer, and the storage space is used to store the patch n of the input feature map of the i+1-th convolutional layer. Wherein, i is greater than or equal to 1 and less than or equal to M, M represents the number of convolutional layers in the neural network, the i-th convolutional layer represents any one of the M convolutional layers, and the i+1-th convolutional layer represents the next convolutional layer of the i-th convolutional layer.

[0060] For example, FIG. 4 shows the processing of multiple convolution layers in a neural network by a neural network processing device in combination with a time axis. As shown in FIG. 4, in a conventional scheduling manner of a neural network, hardware computing resources need to be allocated in sequence according to the order of convolution layers in the neural network. Specifically, the 1st convolution layer (the first convolution layer), the 2nd convolution layer, and the Mth convolution layer (the last convolution layer) need to be calculated in sequence in the computing unit according to the time sequence. For example, the 1st convolution layer is calculated at T0, and the 2nd convolution layer is calculated at T1. For example, the tile j of the i-th convolution layer, the tile data of the i-th convolution layer is calculated in sequence according to the arrangement order of tile 0, tile 1, tile 2, and tile n. Each tile data of each convolution layer is transferred between the external memory and the internal cache of the neural network. Wherein, j is greater than or equal to 0 and less than or equal to n, n represents the number of tile data in the input feature map of the i-th convolution layer, and tile j represents any one of the n tile data.

[0061] For example, when two different neural networks are processed by using the conventional scheduling manner, the data amount of the intermediate layer feature map (fm) of neural network 1 needs to be transferred is 126531.5 megabytes (MB), and the corresponding power consumption is 222.58 milliwatts (mW). The data amount of the intermediate layer of neural network 2 needs to be transferred is 8016.62 MB, and the corresponding power consumption is 71.77 mW. As can be seen, with the upgrading of deep learning neural network algorithms, when the data amount to be processed by the hardware scheduler increases exponentially, the above scheduling manner will cause a large waste of power consumption, and there will also be a memory wall problem, which seriously restricts the performance of the neural network system.

[0062] Therefore, the embodiments of the present application provide a neural network processing method for avoiding the memory wall problem and the waste of power consumption, improving the processing efficiency of the neural network, and further improving the energy efficiency of the neural network processing device. The method can be applied to various electronic devices with a neural network processing device.

[0063] Optionally, the electronic device can include, but is not limited to, a mobile phone, a tablet computer, a notebook computer, a palm computer, a mobile internet device (MID), a camera, a wearable device (for example, a smart watch, a smart bracelet, a pedometer, etc.), a sound equipment, an audio and video player, a set-top box, a game console, a printer, a mouse, a keyboard, a vehicle-mounted device (for example, a device on a vehicle such as a car, an airplane, a ship, a train, and a high-speed rail), a wireless terminal in virtual reality (VR), a wireless terminal in augmented reality (AR), a wireless terminal in industrial control, a smart home device (for example, a refrigerator, a television, an air conditioner, an electricity meter, etc.), a smart robot, a workshop device, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in a smart grid, a wireless terminal in transportation safety, a wireless terminal in a smart city, or a wireless terminal in a smart home, a flight device (for example, a smart robot, a hot air balloon, a drone, an airplane), and the like.

[0064] The structure of the electronic device will be described below by taking the electronic device as a mobile phone as an example. As shown in FIG. 5, the electronic device can include radio frequency (RF) circuit 110, memory 120, input unit 130, display unit 140, sensor 150, audio circuit 160, processor 170, and power supply 180, and the like. Optionally, the processor 170 can be the processing apparatus in the above.

[0065] The RF circuit 110 can be used to transceive information, or receive or send a signal in a call process. In particular, after receiving the downlink information of a base station, the processor 170 processes it; in addition, the uplink data is sent to the base station. Generally, the RF circuit 110 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, and the like. In addition, the RF circuit 110 can also communicate with a network and other devices through a wireless communication mode.

[0066] The memory 120 can be used to store data, software programs, and modules; and mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system and at least one application program required by a function, such as a sound playing function, an image playing function, etc.; and the data storage area can store data created according to the use of the electronic device, such as audio data, image data, a phone book, etc. In addition, the electronic device can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device. In the embodiments of the present application, the memory 120 can be an external memory, for example, the external memory can be a DRAM.

[0067] The input unit 130 can be used to receive inputted digital or character information, and to generate signal input related to the user settings and function control of the electronic device. Specifically, the input unit 130 can include a touch screen 131 and other input devices 132. The touch screen 131 can collect the touch operation of a user on or near it (such as the operation of a user using a finger, a stylus, or any suitable object or accessory on or near the touch screen), and drive the corresponding connection device according to the pre-set program. Optionally, the other input devices 132 can include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, power on-off keys, etc.), a trackball, a mouse, a joystick, etc.

[0068] The display unit 140 can be used to display the information input by the user or provided to the user, and various menus of the electronic device, etc. In one example, the display unit 140 can include a display screen 141, which can be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch screen 131 can cover the display screen 141, and when the touch screen 131 detects a touch operation on or near it, it transmits to the processor 170 to determine the type of touch event, and then the processor 170 provides corresponding visual output on the display screen 141 according to the type of touch event. Although in the figure, the touch screen 131 and the display screen 141 are realized as two independent components to realize the input and output functions of the electronic device, in some embodiments, the touch screen 131 and the display screen 141 can be integrated to realize the input and output functions of the electronic device.

[0069] The sensor 150 can include one or more sensors for providing various aspects of state assessment for the electronic device. Among them, the sensor 150 can include a light sensor that can be used in imaging applications, i.e., as a component of a camera or a camera. In addition, the sensor 150 can also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor, through which the acceleration / deceleration of the electronic device, the orientation, the open / close state, the relative positioning of components, or the temperature change of the electronic device, etc. can be detected.

[0070] The audio circuit 160, the speaker, and the microphone can provide an audio interface between the user and the electronic device. The audio circuit 160 can convert the received audio data into an electrical signal and transmit it to the speaker, which converts it into a sound signal output. On the other hand, the microphone collects sound signals and converts them into electrical signals, which are received by the audio circuit 160 and converted into audio data, which is then output to the RF circuit 110 for transmission to, for example, another phone, or to the memory 120 for further processing.

[0071] The processor 170 is the control center of the electronic device, connects all parts of the electronic device through various interfaces and lines, performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 120 and calling data stored in the memory 120, and thus monitors the entire electronic device. Optionally, the processor 170 can include one or more processing units, which can include but are not limited to a central processing unit (CPU), a network processing unit (NPU), a graphic processing unit (GPU), an image signal processor (ISP), a tensor processing unit (TPU), a data processing unit (DPU), a digital signal processor (DSP), a microcontroller or a microprocessor, etc. Further, the processor 170 can also include other hardware circuits or accelerators, such as an application specific integrated circuit (ASIC), a complex programmable logic device (CPLD) or other programmable logic device, a transistor logic device, a hardware component, or any combination thereof. Optionally, the processor 170 can also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc.

[0072] The electronic device can also include a power supply 180 (such as a battery) for powering various components. The power supply 180 can be logically connected to the processor 170 through a power management system, so as to realize the functions of managing charging, discharging, and power consumption management through the power management system. Optionally, the power management system can support both fast charging technology and non-fast charging technology, and in actual application, the power management system can charge the battery in the power supply 180 through fast charging technology, or can charge the battery in the power supply 180 through non-fast charging technology.

[0073] Although not shown, the electronic device can also include a wireless fidelity (WiFi) module, a Bluetooth module, and the like, which will not be described herein. Those skilled in the art can understand that the structure of the electronic device shown in FIG. 5 does not constitute a limitation on the electronic device, and can include more or fewer components than FIG. 5, or combine certain components, or different component arrangements.

[0074] FIG. 6 is a structural schematic diagram of a neural network processing apparatus provided by an embodiment of the present application. The apparatus is used for processing a neural network, which includes a plurality of convolution layers. The apparatus includes a cache, a scheduler, and a plurality of computing units. Optionally, the apparatus can be the processor 170 in the electronic device described above, the external memory can be the memory 120 in the electronic device described above, and the cache can be an on-chip cache, Cache, or SRAM integrated in the processor 170. The cache is used to cache an input feature map of a convolution layer or at least one sub-input feature map, each sub-input feature map being a part of the input feature map. The scheduler is used to enable the plurality of computing units, each of which is used to calculate at least one sub-input feature map of the plurality of convolution layers to obtain a sub-output feature map of a corresponding convolution layer, the sub-output feature map being a part of an output feature map of the corresponding convolution layer.

[0075] For example, as shown in FIG. 7, if the input feature map of the first convolution layer is divided in the height direction H, a row of data in the diagram can be regarded as one sub-input feature map. If the input feature map is divided in the width direction W, a corresponding group of column data can be regarded as one sub-input feature map. In practice, other division manners can also be used to divide the input feature map, for example, one patch of data in FIG. 2 can be regarded as one sub-input feature map, which is not limited in the embodiments of the present application.

[0076] FIG. 8 is a flowchart of a neural network processing method provided by an embodiment of the present application. The method includes the following steps. The method is applied to the neural network processing apparatus described above. The neural network includes a plurality of convolution layers, and the plurality of convolution layers have a plurality of calculation periods. In any one calculation period, any one convolution layer corresponds to one computing unit or a plurality of computing units. The cache includes a plurality of cache spaces, each cache space corresponding to each convolution layer in a one-to-one manner, and the plurality of cache spaces are used to cache at least one sub-input feature map of a corresponding convolution layer, each sub-input feature map being a part of an input feature map.

[0077] For example, FIG. 9 shows a schematic diagram of the correspondence between a plurality of convolution layers, a plurality of cache spaces, and a plurality of calculation units. In FIG. 9, the number of the plurality of convolution layers is M, the number of the plurality of calculation periods is n, and the number of the plurality of calculation units is M. The first convolution layer of the M convolution layers corresponds to the cache space 1. The first convolution layer corresponds to the calculation unit 1 in the calculation period T1, the calculation unit 2 in the calculation period T2, and the calculation unit 1 in the calculation period Tn. The second convolution layer of the M convolution layers corresponds to the cache space 2. The second convolution layer corresponds to the calculation unit 2 in the calculation period T1, the calculation unit 1 in the calculation period T2, and the calculation unit 2 in the calculation period Tn. The third convolution layer of the M convolution layers corresponds to the cache space 3. The third convolution layer corresponds to the calculation unit 3 in the calculation period T1, the calculation unit 1 in the calculation period T2, and the calculation unit 1 and the calculation unit 3 in the calculation period Tn. The Mth convolution layer corresponds to the cache space M. The Mth convolution layer corresponds to the calculation unit M in the calculation period T1, the calculation unit M in the calculation period T2, and the calculation unit 2 and the calculation unit M in the calculation period Tn.

[0078] In S201, when a first convolution layer of the plurality of convolution layers satisfies a preset condition, the scheduler enables at least one calculation unit corresponding to the first convolution layer in a first period of the plurality of calculation periods. The first convolution layer satisfies the preset condition, which includes that a first cache space corresponding to the first convolution layer is in a readable state in the first period, a second cache space corresponding to a next convolution layer of the first convolution layer is in a writable state in the first period, and at least one calculation unit corresponding to the first convolution layer in the first period is in an idle state.

[0079] The first convolution layer is any one convolution layer of the plurality of convolution layers except the last convolution layer, and the first period is any one calculation period of the plurality of calculation periods of the first convolution layer. For example, as shown in FIG. 10, the plurality of convolution layers include a first convolution layer (a first layer convolution layer) to an Mth convolution layer (a last layer convolution layer). The first convolution layer can be any one convolution layer of the first convolution layer to the M-1th convolution layer.

[0080] In addition, the first cache space is in the readable state in the first period, which means that a certain number of sub-input feature maps of the first convolution layer are cached in the first cache space in the first period. The second cache space is in the writable state in the first period, which means that the capacity of the remaining space in the second cache space is greater than or equal to the data amount of the sub-output feature map of the first convolution layer, which is used as the sub-input feature map of the next convolution layer of the first convolution layer.

[0081] For example, as shown in FIG. 10, the certain number of sub-input feature maps can be K rows. The K can be the number of rows of the convolution kernel corresponding to the first convolution layer, or the number of columns of the convolution kernel, and the specific value of K is related to the arrangement and calculation manner of the convolution layer. For example, when K = 1, in the case where the first convolution layer satisfies the preset condition, the certain number of sub-input feature maps indicates that the first cache space has cached 1 sub-input feature map of the first convolution layer in the first period, and the 1 sub-input feature map can be 1 row of data. For another example, when K = 3, in the case where the first convolution layer satisfies the preset condition, the certain number of sub-input feature maps indicates that the first cache space has cached 3 sub-input feature maps of the first convolution layer in the first period, and the 3 sub-input feature maps can be 3 rows of data.

[0082] In a possible embodiment, each convolution layer of the plurality of convolution layers corresponds to a read pointer and a write pointer. The read pointer is used to read at least one sub-input feature map of the each convolution layer in the cache space corresponding to the each convolution layer. The write pointer is used to write the sub-output feature map of the each convolution layer in the cache space corresponding to the next convolution layer of the each convolution layer. The state of the cache space of the first convolution layer can be determined by the read pointer and the write pointer of the corresponding convolution layer. When the first cache space of the first convolution layer is in the readable state in the first period, the read pointer of the first convolution layer is less than the write pointer of the previous convolution layer of the first convolution layer. When the second cache space corresponding to the next convolution layer of the first convolution layer is in the writable state in the first period, the read pointer of the next convolution layer of the first convolution layer is less than the write pointer of the first convolution layer.

[0083] Further, the at least one calculation unit corresponding to the first convolution layer in the first period is in the idle state, which indicates that there is one or more idle calculation units in the at least one calculation unit corresponding to the first convolution layer in the first period. When there is one idle calculation unit, the scheduler can enable the calculation unit. When there are multiple idle calculation units, the scheduler can enable any one of the multiple idle calculation units.

[0084] Further, each calculation unit of the at least one calculation unit corresponds to a calculation flag bit, and the calculation flag bit is used to indicate the state of the each calculation unit. The state of the each calculation unit includes a working state or an idle state. Based on this, the scheduler can know the state of the calculation unit according to the calculation flag bit of the calculation unit, so as to accurately enable the calculation unit in the idle state, and to avoid waste of calculation resources to a certain extent.

[0085] For the plurality of convolutional layers, when only one convolutional layer (such as only the first convolutional layer at the beginning of the calculation) meets the preset condition, the scheduler can enable the calculation unit corresponding to the first convolutional layer to process the first convolutional layer; after a period of calculation, when at least two convolutional layers in the plurality of convolutional layers meet the preset condition, the scheduler can enable the calculation units corresponding to the at least two convolutional layers to process the at least two convolutional layers respectively, that is, to realize parallel calculation of the at least two convolutional layers.

[0086] For ease of understanding, as shown in FIG. 10, the row data of the first convolutional layer is stored in the external memory, the buffer can read the row data of the first convolutional layer from the external memory, the certain number of sub-input feature maps is K rows, and the plurality of calculation periods include T0, T1, … Tn. Taking the above step S201 as an example, the above step S201 is described.

[0087] For example, if in the calculation period T0, the buffer space of the first convolutional layer in the buffer stores K rows of data of the first convolutional layer, the buffer space of the second convolutional layer is in a writable state, and the calculation unit 1 corresponding to the first convolutional layer is in an idle state, the scheduler enables the calculation unit 1; if in the calculation period T1, the buffer space of the second convolutional layer in the buffer stores K rows of data of the second convolutional layer, the buffer space of the third convolutional layer is in a writable state, and the calculation unit 2 corresponding to the second convolutional layer is in an idle state, the scheduler enables the calculation unit 2; and so on. If in the calculation period Ti, the buffer space of the i-th convolutional layer in the buffer stores K rows of data of the i-th convolutional layer, the buffer space of the i+1-th convolutional layer is in a writable state, and the calculation unit i corresponding to the i-th convolutional layer is in an idle state, the scheduler enables the calculation unit i. In a certain calculation period in the above calculation periods T0 to Tn, when at least two convolutional layers in the first convolutional layer to the M-th convolutional layer meet the preset condition, the scheduler enables the calculation units corresponding to the at least two convolutional layers. For example, when the first convolutional layer and the second convolutional layer meet the preset condition in the calculation period T1, the scheduler enables the calculation unit 1 and the calculation unit 2.

[0088] Optionally, when any convolutional layer in the plurality of convolutional layers does not meet the preset condition in a certain calculation period, the convolutional layer can also not be provided with a corresponding calculation unit in the calculation period. For example, at the beginning of the calculation and only when the first convolutional layer meets the preset condition, the first convolutional layer is provided with a calculation unit in the current calculation period, and the other convolutional layers in the plurality of convolutional layers can not be provided with corresponding calculation units in the current calculation period.

[0089] S202: At least one calculation unit corresponding to the first convolution layer reads at least one sub-input feature map from the first cache space in the first period, performs calculation on the at least one sub-input feature map to obtain a sub-output feature map of the first convolution layer, and caches the sub-output feature map as a sub-input feature map of a next convolution layer of the first convolution layer in the second cache space.

[0090] In a possible example, as shown in FIG. 10, in a calculation period T0, calculation unit 1 corresponding to the first convolution layer calculates K rows of data of the first convolution layer, and stores one row of calculated data in the cache space of the second convolution layer; in a calculation period T1, calculation unit 2 corresponding to the second convolution layer calculates K rows of data of the second convolution layer, and stores one row of calculated data in the cache space of the third convolution layer; and so on and so forth, in a calculation period Ti, calculation unit i corresponding to the i th convolution layer calculates K rows of data of the i th convolution layer, and stores one row of calculated data in the cache space of the i+1 th convolution layer.

[0091] Further, the scheduler enables at least one calculation unit corresponding to the first convolution layer, and in the process of calculating the at least one sub-input feature map, the state of the at least one calculation unit changes, and the read pointer and the write pointer corresponding to the first convolution layer also change, which are illustrated below.

[0092] For ease of description, the at least one calculation unit corresponding to the first convolution layer in the first period is referred to as a calculation unit of the first convolution layer below. Before the scheduler enables the calculation unit of the first convolution layer, the calculation flag bit of the calculation unit of the first convolution layer is in an idle state, that is, the state of the calculation unit of the first convolution layer is in an idle state. After the scheduler enables the calculation unit of the first convolution layer, the calculation unit of the first convolution layer performs calculation on the first convolution layer in the first period, at this time, the calculation unit of the first convolution layer is in a working state, and the scheduler updates the calculation flag bit of the calculation unit of the first convolution layer to the working state. After the calculation unit of the first convolution layer completes the calculation in the first period, the scheduler updates the calculation flag bit of the calculation unit of the first convolution layer to the idle state. For example, taking calculation unit 1 corresponding to the first convolution layer in the calculation period T1 in FIG. 10 as an example, before the scheduler enables the calculation unit 1, the calculation flag bit of the calculation unit 1 is in an idle state, that is, the state of the calculation unit 1 is in an idle state. After the scheduler enables the calculation unit 1, the calculation unit 1 performs calculation on the first convolution layer in the calculation period T1, at this time, the calculation unit 1 is in a working state, and the scheduler updates the calculation flag bit of the calculation unit 1 to the working state. After the calculation unit 1 completes the calculation in the calculation period T1, the scheduler updates the calculation flag bit of the calculation unit 1 to the idle state.

[0093] In addition, the scheduler can update the read pointer and the write pointer corresponding to the first convolutional layer during the calculation unit of the first convolutional layer calculates the first convolutional layer in the first period. In one possible embodiment, the scheduler can update the read pointer corresponding to the first convolutional layer when the calculation unit of the first convolutional layer reads the at least one sub-input feature map from the first cache space; and the scheduler can update the write pointer corresponding to the first convolutional layer when the calculation unit of the first convolutional layer caches the sub-output feature map of the first convolutional layer into the second cache space.

[0094] For example, when the calculation unit of the first convolutional layer reads one sub-input feature map, the read pointer of the first convolutional layer is increased by 1; when the calculation unit of the first convolutional layer reads two sub-input feature maps, the read pointer of the first convolutional layer is increased by 2; and when the calculation unit of the first convolutional layer reads three sub-input feature maps, the read pointer of the first convolutional layer is increased by 3. Similarly, when the calculation unit of the first convolutional layer writes one sub-output feature map of the first convolutional layer, the write pointer of the first convolutional layer is increased by 1; and when the calculation unit of the first convolutional layer writes two sub-output feature maps of the first convolutional layer, the write pointer of the first convolutional layer is increased by 2.

[0095] The update mode of the read pointer described above can be related to the splitting mode of the input feature map, and the update mode of the read pointer can be different when different splitting modes are used to split the input feature map. The embodiments of the present application do not limit this. Similarly, the update mode of the write pointer described above is only for example and does not limit the constitution of the embodiments of the present application.

[0096] Further, when the number of the plurality of calculation units is less than the number of the plurality of convolutional layers, if there are at least two convolutional layers in the plurality of convolutional layers that satisfy the preset condition, and there are different convolutional layers in the at least two convolutional layers that share one or more calculation units, the scheduler can select part of the convolutional layers from the at least two convolutional layers for calculation according to the priority order of the at least two convolutional layers. Optionally, in the at least two convolutional layers, the convolutional layer with a larger layer number has a higher priority. In this way, the problem of calculation resource conflict can be solved, and the inter-layer data generated by the parallel calculation of the plurality of convolutional layers can be consumed as early as possible, so as to further reduce the cache space required by the inter-layer data.

[0097] In one possible example, when the first convolution layer and the second convolution layer in the plurality of convolution layers satisfy the preset condition, and the first convolution layer and the second convolution layer both correspond to the first calculation unit, if the priority of the first convolution layer is higher than the priority of the second convolution layer, the scheduler can select the first convolution layer with higher priority according to the priority order of the first convolution layer and the second convolution layer, i.e., the scheduler can enable the first calculation unit to calculate the first convolution layer in the first period. For example, as shown in FIG. 9, when the second convolution layer and the third convolution layer satisfy the preset condition in the calculation period T2 and both correspond to the calculation unit 1, if the priority of the third convolution layer is higher than the priority of the second convolution layer, the scheduler can enable the calculation unit 1 to calculate the third convolution layer in the calculation period T2.

[0098] In another possible example, when the first convolution layer, the second convolution layer and the third convolution layer in the plurality of convolution layers satisfy the preset condition, and the first convolution layer to the third convolution layer all correspond to the first calculation unit and the second calculation unit, if the priority of the first convolution layer is higher than the priority of the second convolution layer, and the priority of the second convolution layer is higher than the priority of the third convolution layer, the scheduler can select the first convolution layer and the second convolution layer according to the priority order of the convolution layers, i.e., the scheduler can enable the first calculation unit and the second calculation unit to calculate the first convolution layer and the second convolution layer, respectively, in the first period.

[0099] In another possible example, when the first convolution layer, the second convolution layer, the third convolution layer and the fourth convolution layer in the plurality of convolution layers satisfy the preset condition, and the first convolution layer to the fourth convolution layer all correspond to the first calculation unit and the second calculation unit, if the priority of the first convolution layer is higher than the priority of the second convolution layer, the priority of the second convolution layer is higher than the priority of the third convolution layer, and the priority of the third convolution layer is higher than the priority of the fourth convolution layer, the scheduler can divide the first convolution layer and the second convolution layer into a first group and correspond to the first calculation unit, and divide the third convolution layer and the fourth convolution layer into a second group and correspond to the second calculation unit; for the first group and the second group, the scheduler can select the first convolution layer and the third convolution layer with higher priority, i.e., enable the first calculation unit to calculate the first convolution layer in the first period, and enable the second calculation unit to calculate the second convolution layer in the first period.

[0100] Further, for each convolution layer in the plurality of convolution layers, the scheduler can determine the cache space corresponding to each convolution layer and the calculation unit corresponding to each convolution layer in any calculation period in the following manner. Hereinafter, the first convolution layer in the plurality of convolution layers is taken as an example for illustration.

[0101] In a possible embodiment, as shown in FIG. 11, the method further comprises S200, which can be performed before or after S201-S202. In FIG. 11, S200 is performed before S201.

[0102] S200: The scheduler determines, according to the resource mapping information, a first cache space corresponding to the first convolution layer, a second cache space corresponding to a next convolution layer of the first convolution layer, and at least one computing unit corresponding to the first convolution layer in a first period; wherein the resource mapping information is used to indicate a cache space corresponding to each convolution layer in the plurality of convolution layers, and at least one computing unit corresponding to each convolution layer in each period in the plurality of computing periods.

[0103] In the resource mapping information, each cache space in the cache is one-to-one corresponding to each convolution layer. Taking the first convolution layer as an example, the scheduler can determine, according to the resource mapping information, the first cache space corresponding to the first convolution layer and the second cache space corresponding to the next convolution layer of the first convolution layer after determining the first convolution layer. Meanwhile, based on the resource mapping information, the scheduler can obtain the at least one computing unit corresponding to the first convolution layer in the first period. For example, as shown in FIG. 9, taking the first convolution layer as an example, the resource mapping information indicates that the first convolution layer corresponds to the cache space 1, the second convolution layer (the next convolution layer of the first convolution layer) corresponds to the cache space 2, and the computing unit corresponding to the first convolution layer in the computing period T1 is the computing unit 1. When the first convolution layer meets the preset condition, the scheduler enables the computing unit 1.

[0104] Optionally, the resource mapping information can be generated offline and stored in the device, and the scheduler can obtain the resource mapping information when needed, so as to reduce the consumption of hardware resources.

[0105] In order to facilitate understanding of the scheme of the present application, the following examples are used to illustrate the calculation process of the plurality of computing units on the plurality of convolution layers in the above-mentioned embodiments in combination with FIG. 12 and FIG. 13.

[0106] As shown in FIGS. 12 and 13, for example, taking the foregoing K = 3 as an example, the M convolutional layers correspond to a plurality of calculation cycles (C), such as C0-C8. Specifically, in the calculation cycles C0-C2, the external memory sequentially stores the first three sub-input feature maps R0-R2 of the first convolutional layer in the buffer space of the first convolutional layer; in the calculation cycle C2, the number of sub-input feature maps of the first convolutional layer has reached 3, the buffer space of the first convolutional layer is in a readable state, the buffer space of the second convolutional layer has not yet written the sub-input feature maps of the second convolutional layer, the buffer space of the second convolutional layer is in a writable state, and the calculation unit 1 corresponding to the first convolutional layer is in an idle state. The scheduler enables the calculation unit 1 to make the calculation unit 1 calculate the first three sub-input feature maps R0-R2 in the first convolutional layer to obtain a sub-output feature map of the first convolutional layer, and cache the sub-output feature map as the first sub-input feature map R0 of the second convolutional layer in the buffer space of the second convolutional layer. Similarly, in the calculation cycles C3 and C4, the external memory sequentially stores the sub-input feature maps R3 and R4 of the first convolutional layer in the buffer space of the first convolutional layer, and the first convolutional layer satisfies the preset condition in the calculation cycles C3 and C4. The calculation unit 1 calculates the sub-input feature maps R1-R3 and R2-R4 in the first convolutional layer in the calculation cycles C3 and C4, respectively, and caches the sub-output feature maps calculated respectively as the sub-input feature maps R1 and R2 of the second convolutional layer in the buffer space of the second convolutional layer. In the calculation cycle C5, the first convolutional layer still calculates in the above similar manner, and at this time, the number of sub-input feature maps of the second convolutional layer has reached 3, the buffer space of the second convolutional layer is in a readable state, the buffer space of the third convolutional layer is in a writable state, and the calculation unit 2 corresponding to the second convolutional layer is in an idle state. The scheduler enables the calculation unit 2 to make the calculation unit 2 calculate the first three sub-input feature maps R0-R2 in the second convolutional layer to obtain a sub-output feature map of the second convolutional layer, and cache the sub-output feature map as the first sub-input feature map R0 of the third convolutional layer in the buffer space of the third convolutional layer. In this way, at the calculation cycle C5, the first convolutional layer and the second convolutional layer achieve parallel calculation. Similarly, at the calculation cycle C8, the first convolutional layer, the second convolutional layer, and the third convolutional layer achieve parallel calculation. Further, when the M convolutional layers include a larger number of convolutional layers, and each convolutional layer corresponds to a calculation unit, from a certain cycle in the plurality of calculation cycles corresponding to the M convolutional layers, the M convolutional layers can achieve parallel calculation. Wherein, after the calculation of the Mth convolutional layer is completed, the output feature map of the Mth convolutional layer is obtained, and the buffer can output the output feature map of the Mth convolutional layer to the external memory.

[0107] In FIG. 12 and FIG. 13, the calculation units 1 to M correspond to the first convolutional layer to the Mth convolutional layer, respectively. In FIG. 12, only the first three sub-input feature maps R0 to R2 of each convolutional layer are stored in the cache space of the convolutional layer.

[0108] It can be understood that the parallel computing scheme of the M convolutional layers provided by the embodiments of the present application can also be applied to the scenario in which the neural network processing apparatus includes only one calculation unit. In the following, the neural network processing apparatus includes one calculation unit as an example, and the parallel computing process of the M convolutional layers is illustrated by means of FIG. 14 and FIG. 15.

[0109] For example, as shown in FIG. 14 and FIG. 15, taking K = 3 as an example, the plurality of calculation periods are divided into a plurality of calculation steps (step, S), such as S0-S2. Specifically, in the calculation step S0, the external memory stores one sub-input feature map R11 of the first convolutional layer into the cache space of the first convolutional layer. The number of sub-input feature maps of the first convolutional layer has reached 3, the cache space of the first convolutional layer is in a readable state, the cache space of the second convolutional layer is in a writable state, and the calculation unit is in an idle state. The scheduler enables the calculation unit to make the calculation unit calculate the three sub-input feature maps R9-R11 in the first convolutional layer to obtain one sub-output feature map of the first convolutional layer, and store the sub-output feature map as a sub-input feature map R7 of the second convolutional layer into the cache space of the second convolutional layer. Then, the number of sub-input feature maps of the second convolutional layer has reached 3, the cache space of the second convolutional layer is in a readable state, the cache space of the third convolutional layer is in a writable state, and the calculation unit is in an idle state. The scheduler enables the calculation unit to make the calculation unit calculate the three sub-input feature maps R5-R7 in the second convolutional layer to obtain one sub-output feature map of the second convolutional layer, and store the sub-output feature map as a sub-input feature map of the third convolutional layer into the cache space of the third convolutional layer. The remaining convolutional layers are calculated in a similar manner, until the number of sub-input feature maps of the Mth convolutional layer has reached 3, the cache space of the Mth convolutional layer is in a readable state, and the calculation unit is in an idle state. The scheduler enables the calculation unit to make the calculation unit calculate the three sub-input feature maps R1-R3 in the Mth convolutional layer to obtain one sub-output feature map of the Mth convolutional layer, and store the sub-output feature map into the cache space of the output feature map of the Mth convolutional layer. Similarly, in the calculation steps S1 and S2, the scheduler can enable the calculation unit to calculate the M convolutional layers when each convolutional layer in the M convolutional layers meets the preset condition. After the calculation of the Mth convolutional layer is completed, the output feature map of the Mth convolutional layer is obtained, and the buffer can output the output feature map of the Mth convolutional layer to the external memory.

[0110] The calculation unit performs calculation on the M convolutional layers in time sequence of outputting sub-input feature maps of each convolutional layer in a smaller time granularity (for example, taking the calculation period of each convolutional layer as a reference), but can perform parallel calculation on the M convolutional layers in a larger time granularity (taking the calculation stage in the figure as a reference). The calculation unit can perform calculation on the M convolutional layers simultaneously in each calculation stage (for example, S0, S1 and S2).

[0111] In the above FIG. 14 and FIG. 15, the calculation unit corresponding to the first convolutional layer to the Mth convolutional layer is taken as an example for illustration.

[0112] In the embodiments of the present application, the operation of reading K input feature maps from the corresponding cache space by each calculation unit can be regarded as a data consumption process, and the operation of caching the calculated sub-output feature map as the sub-input feature map of the next convolutional layer of the corresponding convolutional layer to the corresponding cache space can be regarded as a data generation process. In this way, when different convolutional layers in the plurality of convolutional layers are calculated in parallel, the sub-output feature map of the previous convolutional layer is gradually generated, and the data of the sub-output feature map of the previous convolutional layer as the sub-input feature map of the next convolutional layer is gradually consumed, so that the inter-layer data (i.e., the sub-input feature map and the sub-output feature map of the plurality of convolutional layers) can be consumed in time after generation, and a large amount of inter-layer data does not need to be cached in the cache, thereby saving the cache space of the cache. In addition, in the entire calculation process, the cache only needs to obtain the first layer input feature map from the external memory and output the output feature map of the Mth convolutional layer finally calculated to the external memory, thereby avoiding the process of moving the inter-layer data between the external memory and the internal cache of the neural network, reducing the power consumption caused by data movement, and thereby improving the energy efficiency of the neural network system.

[0113] The scheduler in the above method embodiments can have many different structures. The structure of the scheduler is exemplified below by taking FIG. 16 to FIG. 19 as examples. In one embodiment of the present application, as shown in FIG. 16, the scheduler can include a detector, an arbitrator and a loader. In another embodiment of the present application, in combination with FIG. 16, as shown in FIG. 17, the arbitrator in the scheduler can include a first-level arbitrator and a second-level arbitrator.

[0114] In an example, the detector includes an input state detection circuit, an output state detection circuit, and a calculation state detection circuit. For each of the plurality of convolution layers, the input state detection circuit is configured to detect whether a buffer space corresponding to each convolution layer is in a readable state in a first period. The output state detection circuit is configured to detect whether a next convolution layer of each convolution layer is in a writable state in the first period. The calculation state detection circuit is configured to detect whether a corresponding calculation unit is in an idle state.

[0115] The arbiter is configured to enable the calculation unit corresponding to a convolution layer in the plurality of convolution layers when the convolution layer satisfies a preset condition. When the arbiter includes a first-level arbiter and a second-level arbiter, the first-level arbiter is configured to determine the convolution layer satisfying the condition in the plurality of convolution layers. The second-level arbiter is configured to enable the calculation unit according to a priority order of the convolution layer.

[0116] The loader is configured to receive the indication information sent by the arbiter to load the configuration information of the corresponding convolution layer. Specifically, as shown in FIGS. 18 and 19, the loader can include a parallel loading circuit and a parallel instruction circuit. As shown in FIG. 18, the parallel loading circuit performs preloading processing on the configuration information of the corresponding convolution layer based on a parallel loading completion signal sent by the arbiter, and outputs an instruction execution enable signal to the parallel instruction circuit. The parallel instruction circuit performs loading processing on the configuration information of the corresponding convolution layer based on the instruction execution enable signal, and outputs an instruction execution completion signal to the parallel loading circuit after the loading is completed. The parallel loading circuit outputs a parallel loading completion signal to the arbiter based on the instruction execution completion signal. As shown in FIG. 19, when the second-level arbiter is included, the parallel loading circuit performs preloading processing on the configuration information of the corresponding convolution layer based on a parallel loading completion signal sent by the second-level arbiter, and outputs an instruction execution enable signal to the parallel instruction circuit. The parallel instruction circuit performs loading processing on the configuration information of the corresponding convolution layer based on the instruction execution enable signal, and outputs an instruction execution completion signal to the parallel loading circuit after the loading is completed. The parallel loading circuit outputs a parallel loading completion signal to the second-level arbiter based on the instruction execution completion signal. The above-mentioned configuration information includes but is not limited to: a calculation type (such as convolution calculation, deconvolution calculation, matrix multiplication calculation, etc.) of the corresponding convolution layer, a bit width of a weight, a bit width of a feature map, a padding, a stride, an activation function (relu), a channel in and out, a bias, and an address, and the like.

[0117] Those skilled in the art can understand that the structure of the scheduler shown in the figure does not constitute a limitation on the scheduler, and can include more or fewer components than shown, or combine certain components, or different component arrangements.

[0118] The following describes the process of enabling the plurality of computing units by the scheduler when the plurality of convolution layers meet the condition, taking the first convolution layer in the plurality of convolution layers as an example.

[0119] In one possible example, as shown in FIG. 20, the process of enabling the computing unit corresponding to the first convolution layer by the scheduler can include S301 to S305.

[0120] S301: Network initialization.

[0121] Specifically, the neural network processing apparatus is initialized, including the initialization of the buffer, the scheduler, and the plurality of computing units. After the initialization is completed, the scheduling judgment of the scheduler is started.

[0122] S302: The input state detection circuit compares the read pointer of the first convolution layer with the write pointer of the previous convolution layer.

[0123] Specifically, when the write pointer of the previous convolution layer is greater than the read pointer of the first convolution layer, the write pointer of the previous convolution layer writes faster, and the input state detection circuit outputs the information that the first buffer space is readable to the arbitrator.

[0124] S303: The output state detection circuit compares the write pointer of the first convolution layer with the read pointer of the next convolution layer.

[0125] Specifically, when the write pointer of the first convolution layer is greater than the read pointer of the next convolution layer, the write pointer of the first convolution layer writes faster, and the output state detection circuit outputs the information that the second buffer space is writable to the arbitrator.

[0126] S304: The computing state detection circuit judges whether the computing unit of the first convolution layer is in an idle state.

[0127] Specifically, the computing state detection circuit determines the state of the computing unit according to the computing flag. When the computing flag of the computing unit of the first convolution layer is in an idle state, the computing state detection circuit outputs the information that the computing unit of the first convolution layer is in an idle state to the arbitrator.

[0128] S305: When the first convolution layer meets the preset condition, the arbitrator enables the computing unit of the first convolution layer to perform computation on the first convolution layer.

[0129] Specifically, when the first convolutional layer satisfies the above three conditions, the arbitrator can enable the calculation unit of the first convolutional layer to perform calculation on the first convolutional layer. When there are multiple convolutional layers satisfying the above preset conditions, the arbitrator enables the corresponding calculation units to perform calculation on the multiple convolutional layers, respectively, from all the convolutional layers satisfying the conditions. When there are at least two convolutional layers in the multiple convolutional layers that have a calculation resource conflict, the arbitrator needs to enable the corresponding calculation units according to the priority order of the at least two convolutional layers to timely solve the problem of calculation resource conflict.

[0130] It can be understood that when the scheduler performs scheduling judgment, it can be predicted in advance whether each convolutional layer satisfies the preset condition, and the convolutional layer satisfying the preset condition is referred to as a to-be-calculated convolutional layer. The to-be-calculated convolutional layer needs to wait for resource release, and after the to-be-calculated convolutional layer performs parallel loading completion signal enabling this time, the calculation on the to-be-calculated convolutional layer is started, so as to avoid resource conflict between the calculation unit and the cache space of the to-be-calculated convolutional layer and the calculation unit and the cache space of the convolutional layer being executed, thereby ensuring the highest utilization of the multiple calculation units.

[0131] In another possible example, as shown in FIG. 21, the process in which the scheduler enables the calculation unit corresponding to the first convolutional layer can include S401 to S406.

[0132] S401: Network initialization.

[0133] Specifically, the neural network processing device is initialized, including the initialization of the buffer, the scheduler, and the calculation unit. After the initialization is completed, the scheduler performs scheduling judgment.

[0134] S402: The input state detection circuit compares the read pointer of the first convolutional layer with the write pointer of the previous convolutional layer.

[0135] Specifically, when the write pointer of the previous convolutional layer is greater than the read pointer of the first convolutional layer, the write pointer of the previous convolutional layer writes faster, and the input state detection circuit outputs information that the first cache space is readable to the first-level arbitrator.

[0136] S403: The output state detection circuit compares the write pointer of the first convolutional layer with the read pointer of the next convolutional layer.

[0137] Specifically, when the write pointer of the first convolutional layer is greater than the read pointer of the next convolutional layer, the write pointer of the first convolutional layer writes faster, and the output state detection circuit outputs information that the second cache space is writable to the first-level arbitrator.

[0138] S404: The calculation state detection circuit judges whether the calculation unit is in an idle state.

[0139] Specifically, the calculation state detection circuit determines the state of the calculation unit according to the calculation flag. When the calculation unit is in an idle state in the first period of the first convolutional layer, the calculation state detection circuit outputs information that the calculation unit is in the idle state to the first-level arbiter.

[0140] S405: The first-level arbiter determines a convolutional layer satisfying a preset condition.

[0141] Specifically, when the first convolutional layer satisfies the preset condition, the first-level arbiter determines a corresponding layer number of the first convolutional layer, and transmits the corresponding layer number to the second-level arbiter.

[0142] S406: The second-level arbiter enables the calculation unit to calculate the convolutional layer satisfying the preset condition according to a priority order.

[0143] Specifically, when there are multiple convolutional layers satisfying the preset condition, the second-level arbiter enables the calculation unit to calculate the convolutional layers satisfying the preset condition in time according to the preset priority order.

[0144] After introducing the scheme of the embodiment of the present application, the principles and power consumption generated when the scheme of the embodiment of the present application and the traditional scheduling scheme are respectively used are simulated and tested by taking the neural network 1 and the neural network 2 described above as examples. The specific test results are shown in Table 1.

[0145] Table 1

[0146] As can be seen from the above table, compared with the traditional scheme, when the scheme of the embodiment of the present application is used, the total power consumption of the neural network 1 is reduced by 52.07%, and the total power consumption of the neural network 2 is reduced by 52.46%. Therefore, the scheme of the embodiment of the present application effectively reduces the power consumption of the neural network system, thereby improving the energy efficiency of the neural network system.

[0147] In another embodiment of the present application, a neural network processing device is also provided, which includes a buffer, a scheduler and a plurality of calculation units, and is connected with an external memory. The scheduler is used to support the device to perform S200 and S201 in the method embodiment described above; and the plurality of calculation units are used to support the device to perform S202 in the method embodiment described above, which will not be described herein again.

[0148] In another embodiment of the present application, an electronic device is also provided, which includes a processor and a memory having instructions stored therein that, when executed by the processor, cause the electronic device to perform the steps of the above-described method embodiments. The processor can be the neural network processing apparatus in the above-described embodiments, and the memory can be the external memory in the above-described embodiments, which can be a DRAM.

[0149] It can be understood that all relevant content of the steps involved in the above-described method embodiments can be cited in the embodiments of the neural network processing apparatus and the electronic device, which will not be described herein again.

[0150] In several embodiments provided in the present application, it should be understood that the disclosed apparatus and method can be implemented by other means. For example, the above-described apparatus embodiments are only illustrative, and for example, the division of the modules or units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another apparatus, or some features can be ignored or not executed.

[0151] The units described as separate components can or can not be physically separate, and the components shown as units can be one physical unit or multiple physical units, that is, can be located in one place or can be distributed to multiple different places. Part or all of the units can be selected according to actual needs to achieve the purpose of the present embodiment.

[0152] If the integrated unit is implemented in the form of a software function unit and sold or used as an independent product, it can be stored in a readable storage medium, which can include a U disk, a mobile hard disk, a read-only memory, a random access memory, a magnetic disk or an optical disk, and various storage medium that can store program codes. Based on such understanding, the technical solutions of the embodiments of the present application can be embodied in the form of a software product, essentially or say the part of the prior art that contributes to the technical solutions or the whole or part of the technical solutions.

[0153] In another embodiment of the present application, a readable storage medium is also provided, which stores computer-executable instructions, and when the computer program or instructions are executed, an apparatus (which can be a single-chip microcomputer, a chip, etc.) or a processor performs the steps of the above-described method embodiments.

[0154] In yet another embodiment of the present application, a computer program product is also provided, which includes computer instructions stored in a readable storage medium; at least one processor of a device can read the computer instructions from the readable storage medium, and the at least one processor executes the computer instructions to make the device perform the steps in the above-mentioned method embodiments.

[0155] Finally, it should be noted that the above-mentioned is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any change or replacement within the technical scope disclosed in the present application should be covered in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A neural network processing apparatus, characterized by comprising: An apparatus for processing a neural network comprising a plurality of convolution layers, the apparatus comprising: a buffer, a scheduler and a plurality of computing units; each convolution layer of the plurality of convolution layers corresponds to at least one computing unit of the plurality of computing units in any of a plurality of computing cycles; the buffer comprises a plurality of buffer spaces corresponding to the plurality of convolution layers one-to-one, each buffer space being configured to buffer at least one sub-input feature map of a corresponding convolution layer, each sub-input feature map being a part of an input feature map; the scheduler is configured to enable the at least one computing unit corresponding to a first convolution layer of the plurality of convolution layers in a first cycle of the plurality of computing cycles when the first convolution layer satisfies a preset condition; wherein the first convolution layer satisfies the preset condition comprises that a first buffer space corresponding to the first convolution layer is in a readable state in the first cycle, a second buffer space corresponding to a next convolution layer of the first convolution layer is in a writable state in the first cycle, and the at least one computing unit corresponding to the first convolution layer is in an idle state in the first cycle; the at least one computing unit corresponding to the first convolution layer is configured to read the at least one sub-input feature map from the first buffer space, perform computation on the at least one sub-input feature map to obtain a sub-output feature map of the first convolution layer, and cache the sub-output feature map as a sub-input feature map of the next convolution layer of the first convolution layer into the second buffer space in the first cycle.

2. The apparatus of claim 1, wherein the scheduler is further configured to determine the first buffer space corresponding to the first convolution layer, the second buffer space corresponding to the next convolution layer of the first convolution layer, and the at least one computing unit corresponding to the first convolution layer in the first cycle according to resource mapping information; wherein the resource mapping information is configured to indicate a buffer space corresponding to each convolution layer of the plurality of convolution layers, and at least one computing unit corresponding to the each convolution layer in each cycle of the plurality of computing cycles.

3. The apparatus of claim 1 or 2, wherein, each convolution layer of the plurality of convolution layers corresponds to a read pointer and a write pointer, the read pointer being configured to read at least one sub-input feature map of the each convolution layer in a buffer space corresponding to the each convolution layer, and the write pointer being configured to write a sub-output feature map of the each convolution layer in a buffer space corresponding to a next convolution layer of the each convolution layer; the scheduler is further configured to update the read pointer corresponding to the first convolution layer when the at least one computing unit corresponding to the first convolution layer reads the at least one sub-input feature map from the first buffer space, and update the write pointer corresponding to the first convolution layer when the at least one computing unit corresponding to the first convolution layer caches the sub-output feature map of the first convolution layer into the second buffer space.

4. The device according to any of claims 1-3, characterized in that Each of the at least one computing unit corresponds to a computing flag bit, used to indicate a state of the each computing unit; the state of the each computing unit includes a working state or the idle state.

5. The device according to any of claims 1-4, characterized in that When there are at least two convolutional layers in the plurality of convolutional layers satisfying the preset condition, and the at least two convolutional layers correspond to a same computing unit in the plurality of computing units; The scheduler is further configured to select one convolutional layer from the at least two convolutional layers as the first convolutional layer according to a priority order of the at least two convolutional layers.

6. The apparatus of claim 5, wherein, In the at least two convolutional layers, a convolutional layer with a larger layer number has a higher priority.

7. A neural network processing method, comprising: The method is applied to a neural network processing device, the neural network includes a plurality of convolutional layers, and the device includes a buffer, a scheduler and a plurality of computing units; each of the plurality of convolutional layers corresponds to at least one computing unit in the plurality of computing units in any period of a plurality of computing periods; the buffer includes a plurality of buffer spaces corresponding to the plurality of convolutional layers one by one, each buffer space is used to buffer at least one sub-input feature map of a corresponding convolutional layer, and each sub-input feature map is a part of an input feature map; the method includes: When a first convolutional layer in the plurality of convolutional layers satisfies a preset condition, the scheduler enables at least one computing unit corresponding to the first convolutional layer in a first period of the plurality of computing periods; wherein the first convolutional layer satisfying the preset condition includes: a first buffer space corresponding to the first convolutional layer being in a readable state in the first period, a second buffer space corresponding to a next convolutional layer of the first convolutional layer being in a writable state in the first period, and the at least one computing unit corresponding to the first convolutional layer in the first period being in an idle state; In the first period, the at least one computing unit corresponding to the first convolutional layer reads the at least one sub-input feature map from the first buffer space, performs computation on the at least one sub-input feature map to obtain a sub-output feature map of the first convolutional layer, and caches the sub-output feature map as a sub-input feature map of a next convolutional layer of the first convolutional layer into the second buffer space.

8. The method of claim 7, wherein, The method further includes: The scheduler determines the first buffer space corresponding to the first convolutional layer, the second buffer space corresponding to the next convolutional layer of the first convolutional layer, and the at least one computing unit corresponding to the first convolutional layer in the first period according to resource mapping information; wherein the resource mapping information is used to indicate a buffer space corresponding to each convolutional layer in the plurality of convolutional layers, and at least one computing unit corresponding to each convolutional layer in each period of the plurality of computing periods.

9. The method according to claim 7 or 8, characterized in that, Each of the plurality of convolutional layers corresponds to a read pointer and a write pointer, the read pointer is used to read at least one sub-input feature map of the each convolutional layer in a buffer space corresponding to the each convolutional layer, and the write pointer is used to write a sub-output feature map of the each convolutional layer in a buffer space corresponding to a next convolutional layer of the each convolutional layer; The method further includes: The scheduler updates a read pointer corresponding to the first convolutional layer when the at least one calculation unit corresponding to the first convolutional layer reads the at least one sub-input feature map from the first cache space; The scheduler updates a write pointer corresponding to the first convolutional layer when the at least one calculation unit corresponding to the first convolutional layer caches the sub-output feature map of the first convolutional layer into the second cache space.

10. The method according to any one of claims 7 to 9, characterized in that, Each calculation unit of the at least one calculation unit corresponds to a calculation flag, used to indicate a state of the each calculation unit; the state of the each calculation unit includes a working state or the idle state.

11. The method according to any one of claims 7-10, characterized in that, When there are at least two convolutional layers in the plurality of convolutional layers satisfying the preset condition, and the at least two convolutional layers correspond to a same calculation unit in the plurality of calculation units, the method further comprises: The scheduler selects one convolutional layer as the first convolutional layer from the at least two convolutional layers according to a priority order of the at least two convolutional layers.

12. The method of claim 11, wherein, In the at least two convolutional layers, a convolutional layer with a larger layer number has a higher priority.

13. An electronic device, comprising: The electronic device includes a processor and a memory, and the memory stores instructions, when the processor executes the instructions, the electronic device executes the neural network processing method as claimed in any one of claims 7-12.

14. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, when the instructions run on a device, the device executes the neural network processing method as claimed in any one of claims 7-12.

15. A computer program product, characterised in that, The computer program product includes a computer program, when the computer program runs on a device, the device executes the neural network processing method as claimed in any one of claims 7-12.

Images