A data management method for storage medium of neural network accelerator

Abstract

The present invention provides a data management method for a storage medium of a neural network accelerator, wherein the storage medium includes a main memory, and the method includes: obtaining structural information and instruction information of a neural network-based model to be accelerated; based on the structural information and instruction parameters, dividing a dedicated storage area in the storage medium to store instructions and data corresponding to the model; wherein the dedicated storage area includes: an instruction area for storing instructions required for the accelerator to accelerate operations on the model; a data stacking area for storing intermediate result data output by a corresponding layer to be used in other layers in the interval, and performing spatial multiplexing within the area; and a data temporary storage area for storing intermediate result data output by a corresponding layer to be used only in the layer below the layer, and performing spatial multiplexing within the area.

Description

Technical Field

[0001] The present invention relates to the technical field of neural network accelerators, specifically to the technical field of data management on neural network accelerators, and more specifically to a data management method for a storage medium of a neural network accelerator. Background Art

[0002] In recent years, deep learning technology has rapidly developed, with extensive deployment and research in image processing. The processing of various visual tasks in autonomous driving and robotics requires substantial computing and storage resources. However, due to resource constraints on terminal devices and the limited parallel computing capabilities of general-purpose processors based on the von Yino architecture, neural network processing efficiency is inefficient. In particular, with the increasing number of layers in modern neural network models and the widespread application of deep neural networks, the demand for processing efficiency is increasing. To accelerate neural networks, neural network accelerators have emerged. All or part of the neural network tasks previously executed on the terminal device can be offloaded to neural network accelerators.

[0003] Existing neural network accelerator architectures offer good acceleration and energy efficiency for a single neural network-based vision acceleration task on a single terminal device. However, in some cases, multiple terminal devices may be connected to a single accelerator, requiring the neural network accelerator's storage medium to handle large amounts of data (input data, intermediate results, output data, etc.) from multiple tasks. Existing accelerators store intermediate data in a layered order. As the number of terminal device tasks increases (such as robotic depth perception, semantic segmentation, object recognition and detection, and optical flow detection) or as multiple models require acceleration, existing accelerators that manage intermediate data in a layered order consume significant main memory (also known as internal memory). Furthermore, due to inappropriate data management (for example, frequently used data may need to be constantly removed and loaded into main memory), the accelerator incurs significant storage overhead, potentially requiring an increase in main memory capacity, which in turn increases the accelerator's power consumption and hardware cost. Summary of the Invention

[0004] Therefore, the purpose of the present invention is to overcome the above-mentioned defects of the prior art and provide a data management method for a storage medium of a neural network accelerator.

[0005] The purpose of the present invention is achieved through the following technical solutions:

[0006] According to a first aspect of the present invention, there is provided a data management method for a storage medium of a neural network accelerator, wherein the storage medium includes a main memory, and the method includes: obtaining structural information and instruction information of a neural network-based model to be accelerated; according to the structural information and instruction parameters, dividing a dedicated storage area in the storage medium to store instructions and data corresponding to the model; wherein the dedicated storage area includes: an instruction area for storing instructions required for the accelerator to accelerate operations on the model; a data stacking area for storing intermediate result data output by a corresponding layer to be used in other layers in the interval, and performing spatial multiplexing within the area; and a data temporary storage area for storing intermediate result data output by a corresponding layer to be used only in the next layer of the layer, and performing spatial multiplexing within the area.

[0007] Preferably, the method includes: determining the size of the intermediate result data output by each layer and the data dependency relationship of each layer based on the structural information of the model; determining the maximum value of the space required for the set of intermediate result data that should be stored at any time during the acceleration of the model and that will be used in subsequent calculations based on the size of the intermediate result data output by each layer and the data dependency relationship of each layer, and using it as the stacking size; setting the space size of the data stacking area based on the stacking size.

[0008] Preferably, the method includes: allocating a stacking sub-area for the intermediate result data output by the corresponding layer in the data stacking area, wherein when the corresponding layer outputs the intermediate result data, it preferentially occupies an idle and available stacking sub-area, and when there is no idle and available stacking sub-area, another stacking sub-area is allocated, and the idle and available stacking sub-area refers to the intermediate result data currently stored therein which does not need to be used in subsequent calculations and whose space size is sufficient to store the intermediate result data output by the corresponding layer to be stored.

[0009] Preferably, the method includes: determining the maximum value of the space required for the output of the corresponding layer to be stored in the data temporary storage area based on the size of the intermediate result data output by each layer and the data dependency relationship of each layer, and doubling the value as the temporary storage size; setting the space size of the data temporary storage area according to the temporary storage size.

[0010] Preferably, the method includes: dividing the data temporary storage area into a first temporary storage sub-area and a second temporary storage sub-area of ​​the same spatial size; when performing accelerated operations on the model, alternately selecting an area in the first temporary storage sub-area and the second temporary sub-area to store the intermediate result data output by the corresponding layer.

[0011] Preferably, the method further includes: determining the size of the space required for the set of instructions required to accelerate the operation of the model based on the instruction information; and setting the size of the space of the instruction area based on the size of the space required for the set of instructions.

[0012] Preferably, the dedicated storage area also includes: a data input area for temporarily storing input data, wherein the space size of the data input area is the size of the input data; and / or a data output area for storing the accelerated results of the accelerated operation of the model, wherein the space size of the data output area is the size of the accelerated results.

[0013] Preferably, the dedicated storage area further includes: a trained parameter area for storing trained parameters including weight parameters and offset parameters.

[0014] According to a second aspect of the present invention, an electronic device is provided, comprising: one or more processors; and a memory, wherein the memory is used to store executable instructions; the one or more processors are configured to implement the steps of any one of the methods described in the first aspect by executing the executable instructions.

[0015] Compared with the prior art, the advantages of the present invention are:

[0016] The present invention obtains structural information and instruction information of a neural network-based model to be accelerated, and allocates a dedicated storage area in the storage medium to store the instructions and data corresponding to the model, thereby rationally and efficiently managing the storage process of various instructions and data required during acceleration. Specifically, a dedicated instruction area is set up to store the instructions required by the accelerator to accelerate the model, so that the relevant instructions will not be kicked out by other data during acceleration, and can be quickly accessed when needed, ensuring acceleration efficiency. An area is pre-allocated to store intermediate result data output by the corresponding layer that will be used in other layers in the interval, and spatially reused within the area, meeting the storage needs of the output of the interval layer with minimal space consumption and configuration cost. Another area is pre-allocated to store intermediate result data output by the corresponding layer that is only used in the next layer of the layer. Placing such frequently updated intermediate result data in one area facilitates the storage and update needs of the intermediate result data that is only used in the next layer of the corresponding layer with minimal space consumption and configuration cost. Thus, efficient reuse of storage space on the accelerator is achieved, reducing hardware requirements and configuration overhead for the storage medium. Under equivalent hardware conditions, the acceleration efficiency can be improved compared to existing technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The embodiments of the present invention are further described below with reference to the accompanying drawings, in which:

[0018] Figure 1 2. A flowchart of a method for managing data in a storage medium of a neural network accelerator according to an embodiment of the present invention;

[0019] Figure 2 is a schematic diagram of structural information of an exemplary neural network model according to an embodiment of the present invention;

[0020] Figure 3 A schematic diagram of an implementation of a dedicated storage area according to an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the principle of data management corresponding to an illustrative neural network model according to an embodiment of the present invention;

[0022] Figure 5 FIG. 4 is a schematic diagram of another implementation of a dedicated storage area according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below through specific embodiments in conjunction with the accompanying drawings. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0024] As mentioned in the background technology section, existing accelerators store intermediate result data in layered order. This results in high storage space consumption as the number of tasks increases, increasing the hardware requirements for the storage medium. Furthermore, irrational data storage methods lead to high space allocation overhead, affecting acceleration efficiency. To address this, the present invention obtains structural information and instruction information for the neural network-based model to be accelerated, and allocates a dedicated storage area in the storage medium to store the instructions and data corresponding to the model, thereby rationally and efficiently managing the storage process of the various instructions and data required during acceleration. Among them, a dedicated instruction area is set up to store the instructions required by the storage accelerator to accelerate the operation of the model, so that the relevant instructions will not be kicked out by other data during the acceleration period, so that they can be quickly accessed when needed, ensuring the acceleration efficiency; an area is pre-divided to store the intermediate result data output by the corresponding layer that will be used in other layers in the interval and spatially reused within the area, so as to meet the storage needs of the output of the interval layer with less space consumption and configuration cost; an area is pre-divided to store the intermediate result data output by the corresponding layer that is only used in the next layer of the layer. Placing such frequently replaced intermediate result data in one area is conducive to meeting the storage and update needs of the intermediate result data that is only used in the next layer of the corresponding layer with less space consumption and configuration cost. In this way, efficient reuse of the storage space on the accelerator is achieved, reducing the hardware requirements and configuration overhead of the storage medium; under the same hardware conditions, the acceleration efficiency can be improved compared with the existing technology.

[0025] According to one embodiment of the present invention, a data management method for a storage medium of a neural network accelerator is provided, wherein the storage medium is a main memory, see Figure 1The method includes: step S1 and step S2. In order to better understand the present invention, each step will be described in detail below in conjunction with specific embodiments.

[0026] Step S1: Obtain structural information and instruction information of a neural network-based model to be accelerated.

[0027] According to one embodiment of the present invention, the structural information includes the input data size of the model, the output size of each layer, the space required by the set of trained parameters, and data dependencies. Data dependencies refer to the source of input data of each layer and the flow of output data. The output size of each layer and the data dependencies of each layer can be used to subsequently determine the space required for the corresponding storage area. For example, see Figure 2 , assuming that a model includes multiple convolutional layers, namely: input layer, first layer, second layer, third layer, fourth layer, fifth layer, sixth layer, seventh layer, output layer, the output data of the convolutional layer corresponding to the output layer is the acceleration result; wherein, in step S1, which is equivalent to the parsing stage, it is necessary to determine the structural information, including: the size of the input data, the output size of the input layer, the output size of the first layer, the output size of the second layer, the output size of the third layer, the output size of the fourth layer, the output size of the fifth layer, the output size of the sixth layer, the output size of the seventh layer, and the output size of the output layer; the data dependency corresponds to Figure 2 Preferably, the instruction information includes the size of each instruction required by the accelerator to accelerate the model. The size of the space required for the instruction set is determined based on the size of each instruction, so as to subsequently determine the size of the instruction area. Alternatively, the instruction information includes the size of the instruction set required by the accelerator to accelerate the model.

[0028] S2. Based on the structure information and instruction parameters, a dedicated storage area is divided in the storage medium to store instructions and data corresponding to the model.

[0029] See also Figure 3 According to one embodiment of the present invention, the dedicated storage area includes: a trained parameter area 20, a trained parameter area 20, a data input area 30, a data stacking area 40, a data temporary storage area 50 and a data output area 60. Wherein:

[0030] Instruction area 10 is used to store the instructions needed by the accelerator to accelerate the model. The size of the instruction area is set to the required space for the set of instructions required by the accelerator to accelerate the model. This allows instructions required for acceleration to be permanently stored within instruction area 10, preventing them from being pushed out of the current storage medium by other data or instructions. This ensures faster instruction access and improves acceleration efficiency.

[0031] The trained parameter area 20 is used to store trained parameters including weight parameters and offset parameters; the space size of the trained parameter area 20 is equal to the space size required for the set of trained parameters of the model to be accelerated. The trained parameters refer to the parameters required for the operation of the model itself, the weight parameters are the parameters in the convolution kernel, and the offset parameters refer to the bias, which are known to those skilled in the art and will not be described in detail here. Since a large number of operations involve trained parameters during the acceleration period, the present invention pre-divides an area for storing the trained parameters of the weight parameters and the offset parameters, and establishes a dedicated storage area for the trained parameters, which can better ensure the efficiency of the accelerated operation process.

[0032] The data input area 30 is used to temporarily store the input data, wherein the space size of the data input area is the size of the input data. Figure 2 The input data is the input to the model. For example, if you are classifying an image or identifying an object, the input data is the input image. For another example, if you are identifying the state of an object based on sensor data, the input data is the input sensor data.

[0033] The data stacking area 40 is used to store the intermediate result data output by the corresponding layer and used in other layers in the interval, and to perform spatial multiplexing within the area; the size of the data stacking area 40 is set to the maximum value of the space required for the set of intermediate result data that should be stored at any time during the acceleration of the model and that will be used in subsequent calculations. According to one embodiment of the present invention, the method includes: determining the size of the intermediate result data output by each layer and the data dependency of each layer based on the structural information of the model; determining the maximum value of the space required for the set of intermediate result data that should be stored at any time during the acceleration of the model and that will be used in subsequent calculations based on the size of the intermediate result data output by each layer and the data dependency of each layer and using it as the stacking size; setting the spatial size of the data stacking area based on the stacking size. In other words, in the partitioning stage corresponding to step S2, what is related to the data stacking area 40 is: it is necessary to determine whether the output of a certain layer is data that needs to be stored in the data stacking area 40 based on the data dependency and to determine the size of the data stacking area based on the output size and data dependency of the corresponding layer. Figure 4Taking the model shown as an example, the intermediate result data output by the first layer is the intermediate result data to be used in the other layers (the sixth layer) of the interval; in addition, the intermediate result data output by the second layer is also the intermediate result data to be used in the other layers (the fifth layer) of the interval; from the overall perspective, during the period when the second layer outputs its intermediate result data and the sixth layer outputs its intermediate result data, the corresponding data stacking area 40 should store the maximum value of the space required for the set of intermediate result data that will be used in subsequent calculations, that is, the total space required for the intermediate result data output by the second layer and the intermediate result data output by the sixth layer.

[0034] In order to save storage resources, according to one embodiment of the present invention, spatial multiplexing is performed in the data stacking area 40 in the following manner: stacking sub-areas are allocated in the data stacking area for the intermediate result data output by the corresponding layer, wherein, when the corresponding layer outputs the intermediate result data, an idle available stacking sub-area is preferentially occupied, and when there is no idle available stacking sub-area, another stacking sub-area is allocated, wherein the idle available stacking sub-area refers to the intermediate result data currently stored therein which does not need to be used in subsequent calculations and whose space size is sufficient to store the intermediate result data output by the corresponding layer to be stored. For example, see Figure 5 As needed, the data stacking area 40 is divided into a first stacking sub-area 41, a second stacking sub-area (not shown), ..., an nth stacking sub-area 4n. Figure 4 Take the model structure of as an example, assuming that after the first layer outputs, its output will be used in the sixth layer, so stacking sub-region-1 (corresponding to the first stacking sub-region) is allocated to it; after the second layer outputs, its output will be used in the fifth layer, but the data in stacking sub-region-1 is still needed in subsequent calculations (the sixth layer), so stacking sub-region-1 is not an available stacking sub-region, so stacking sub-region-2 (corresponding to the second stacking sub-region) is allocated to it.

[0035] The data temporary storage area 50 is used to store the intermediate result data output by the corresponding layer, which is only used by the next layer of the layer, and to perform spatial multiplexing within the area; the size of the data temporary storage area 50 is set to twice the maximum value of the space required for the output of the corresponding layer to be stored in the data temporary storage area. According to one embodiment of the present invention, the method includes: according to the size of the intermediate result data output by each layer and the data dependency relationship of each layer, determining the maximum value of the space required for the output of the corresponding layer to be stored in the data temporary storage area 50 and doubling the value as the temporary storage size; setting the space size of the data temporary storage area according to the temporary size. For example, see again Figure 4According to the data dependency, it can be seen that the intermediate result data output by the input layer, the third layer, the fourth layer, the fifth layer, the sixth layer, and the seventh layer are intermediate result data that are only used by the next layer of the layer. Therefore, they need to be stored in the data temporary storage area 50. Assuming that the output of the fifth layer requires the largest space, the space occupied is K, and the size of the data temporary storage area is set to 2K.

[0036] In order to save storage resources, according to one embodiment of the present invention, refer again to Figure 5 , spatial multiplexing is performed in the data temporary storage area 50 in the following manner: the data temporary storage area is divided into a first temporary storage sub-area 51 and a second temporary storage sub-area 52 of the same spatial size; when the model is accelerated, an area is alternately selected from the first temporary storage sub-area 51 and the second temporary storage sub-area 52 to store the intermediate result data output by the corresponding layer. Figure 4 According to the data dependency, it can be seen that the intermediate result data output by the input layer, the third layer, the fourth layer, the fifth layer, the sixth layer, and the seventh layer are intermediate result data that are only used by the next layer of the layer. The corresponding alternating storage process is: the output of the input layer is stored in the temporary storage sub-area-A (corresponding to the first temporary storage sub-area 51), the output of the third layer is stored in the temporary storage sub-area-B (corresponding to the second temporary storage sub-area 52), the output of the fourth layer is stored in the temporary storage sub-area-A, the output of the fifth layer is stored in the temporary storage sub-area-B, the output of the sixth layer is stored in the temporary storage sub-area-A, and the output of the seventh layer is stored in the temporary storage sub-area-B.

[0037] Data output area 60 is used to store the acceleration results of the model acceleration operation. The size of the data output area is the size of the acceleration result. The acceleration result may be the output of a layer of the model or the final prediction result. The specific configuration is determined by the user and is not limited in this invention.

[0038] It should be understood that the above embodiment is only a preferred embodiment, and those skilled in the art can adaptively adjust some implementation details as needed to obtain other embodiments.

[0039] According to one embodiment of the present invention, unlike the aforementioned embodiment, a data input area is not provided in the dedicated storage area. Therefore, the structure information does not need to include the input data size of the model. The input data can be managed using existing methods.

[0040] According to one embodiment of the present invention, the difference from the above embodiment is that the trained parameter area is not set up in the dedicated storage area. Therefore, the structure information does not need to include the space size required by the set of trained parameters.

[0041] According to one embodiment of the present invention, unlike the aforementioned embodiment, a data output area is not established in the dedicated storage area. Therefore, the structure information does not need to include the data size of the output layer output (the size of the acceleration result). Data management of the output data can adopt existing methods.

[0042] According to one embodiment of the present invention, the difference from the above embodiment is that the storage medium is a cache.

[0043] According to one embodiment of the present invention, an electronic device is provided, comprising: one or more processors; and a storage medium for storing executable instructions; the one or more processors being configured to execute the executable instructions to implement the steps of the method described in the aforementioned embodiment. For example, the electronic device is a neural network accelerator, which is configured to manage data and instructions stored in the storage medium for a neural network-based model to be accelerated according to the method described in the aforementioned embodiment.

[0044] In general, the advantages of the present invention are: the storage method designed for data in the storage medium during neural network processing is partitioned and spatially multiplexed, which has greater space utilization than the traditional method of sequential storage of each network layer, reduces the space consumption of neural network application tasks, and reduces the hardware storage cost of the accelerator; it also reduces configuration overhead and better ensures acceleration efficiency.

[0045] It should be noted that although the above describes the various steps in a specific order, it does not mean that the steps must be performed in the above specific order. In fact, some of these steps can be executed concurrently or even in a different order as long as the required functions can be achieved.

[0046] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0047] Computer-readable storage media can be a tangible device that holds and stores the instructions used by an instruction execution device. Computer-readable storage media can, for example, include, but are not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, a punch card or a raised structure in a groove on which instructions are stored, for example, and any suitable combination thereof.

[0048] While various embodiments of the present invention have been described above, the above descriptions are intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A data management method for a storage medium of a neural network accelerator, wherein the storage medium includes a main memory, characterized in that: The method comprises: Obtaining structural information and instruction information of a neural network-based model to be accelerated; According to the structure information and instruction parameters, a dedicated storage area is divided in the storage medium to store instructions and data corresponding to the model; Wherein, the dedicated storage area includes: The instruction area is used to store the instructions required by the accelerator to accelerate the operation of the model; The data stacking area is used to store the intermediate result data output by the corresponding layer to be used in other layers in the interval, and perform spatial multiplexing within the area; The data temporary storage area is used to store the intermediate result data output by the corresponding layer that is only used by the next layer of the layer, and perform spatial multiplexing within the area.

2. The method according to claim 1, characterized in that The method comprises: According to the structural information of the model, determine the size of the intermediate result data output by each layer and the data dependency relationship of each layer; Based on the size of the intermediate result data output by each layer and the data dependencies of each layer, the maximum value of the space required for the set of intermediate result data that should be stored at any time during the model acceleration and will be used in subsequent calculations is determined and used as the stack size; The spatial size of the data stacking area is set according to the stacking size.

3. The method according to claim 2, characterized in that The method comprises: In the data stacking area, a stacking sub-area is allocated for the intermediate result data output by the corresponding layer, wherein when the corresponding layer outputs the intermediate result data, an idle and available stacking sub-area is occupied first, and when there is no idle and available stacking sub-area, another stacking sub-area is allocated. The idle and available stacking sub-area refers to the intermediate result data currently stored therein, which does not need to be used in subsequent calculations and whose space size is sufficient to store the intermediate result data output by the corresponding layer to be stored.

4. The method according to claim 2, characterized in that The method comprises: According to the size of the intermediate result data output by each layer and the data dependency relationship of each layer, the maximum value of the space required for the output of the corresponding layer to be stored in the data temporary storage area is determined and doubled as the temporary storage size; The space size of the data temporary storage area is set according to the temporary storage size.

5. The method according to claim 4, characterized in that The method comprises: Dividing the data temporary storage area into a first temporary storage sub-area and a second temporary storage sub-area of ​​the same size; When performing accelerated operations on the model, an area is alternately selected from the first temporary storage sub-area and the second temporary storage sub-area to store intermediate result data output by the corresponding layer.

6. The method according to claim 1, characterized in that The method further comprises: Determine the space required for the set of instructions required to accelerate the model based on the instruction information; Set the size of the instruction area according to the space required by the instruction set.

7. The method according to any one of claims 1 to 6, characterized in that The dedicated storage area also includes: A data input area for temporarily storing input data, wherein the size of the data input area is the size of the input data; and / or The data output area is used to store the acceleration results of the accelerated operation of the model, wherein the space size of the data output area is the size of the acceleration result.

8. The method according to any one of claims 1 to 6, characterized in that The dedicated storage area also includes: The trained parameter area is used to store trained parameters including weight parameters and offset parameters.

9. A computer-readable storage medium, characterized in that A computer program is stored thereon, and the computer program can be executed by a processor to implement the steps of the method according to any one of claims 1 to 8.

10. An electronic device, characterized in that: include: one or more processors; as well as a memory, wherein the memory is used to store executable instructions; The one or more processors are configured to implement the steps of the method of any one of claims 1 to 8 by executing the executable instructions.

Images