External data acquisition method and device based on CNN accelerator, and medium
By using an adaptive hybrid data stream approach, the data stream is segmented and accessed according to the storage space relationship between the input feature map and the weights. This solves the problems of poor model adaptability and high memory access power consumption in CNN accelerators, and achieves more efficient data stream management.
Patent Information
- Application Number
- CN202511191739.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-08-25
- Publication Date
- 2026-01-13
AI Technical Summary
The fixed data flow mode of existing CNN accelerators results in poor model adaptability and high memory access power consumption, making them unable to adapt to complex neural network models with inflated parameters.
An adaptive hybrid data stream approach is adopted, which divides the input feature map and weights according to the storage space relationship, and optimizes data access through the on-chip storage module to achieve one-time access to weights and input feature data, thereby reducing memory access power consumption.
It improves the adaptability of data flow, reduces memory access power consumption, and enhances the universality and computational efficiency of the model.
Smart Images
Figure CN121328629A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of AI and neural network accelerator technology, and in particular to an external data acquisition method, apparatus and storage medium based on a CNN accelerator. Background Technology
[0002] Existing CNN accelerators employ a fixed data flow approach for accessing and computing off-chip data. However, a fixed data flow inevitably requires cyclical access to off-chip input feature maps and weight data, resulting in multiple off-chip memory accesses and significant memory access power consumption. Furthermore, a fixed data flow necessitates cyclically cutting the input feature maps and weight data according to a fixed cutting method, which leads to poor model adaptability.
[0003] With the development of AI technology, neural network models are becoming more complex in structure, more inflated in parameters, and more computationally intensive. There are huge differences between different models. Even within a single network model, there is significant heterogeneity between different computational layers, and the proportion of input feature maps and weight data also varies.
[0004] Therefore, the single data flow mode of existing technologies has poor universality and its performance varies greatly on different general models. This situation is obviously contrary to the core design requirements of general accelerators in the industrial field. The original design purpose of accelerators is to achieve dynamic adaptive optimization and accurate adaptation mapping of complex model systems in all dimensions. However, existing technologies cannot achieve this effect in accelerators. Summary of the Invention
[0005] To address the problems of poor model adaptability and high memory access power consumption in existing technologies with fixed data streams, this application mainly provides an external data acquisition method, apparatus, storage medium, electronic device, and computer program product based on a CNN accelerator.
[0006] To achieve the above objectives, the first technical solution adopted in this application is an external data acquisition method based on a CNN accelerator, which includes: determining the size relationship between the first storage space size, the second storage space size, and the memory space size based on the first storage space size and the second storage space size occupied by the weights and the input feature map stored in the external memory outside the hardware accelerator, respectively, and the memory space size of the on-chip storage module of the hardware accelerator; dividing one of the weights or the input feature map according to the size relationship to obtain the division parameter, and taking the other undivided weight or input feature map as the undivided parameter; traversing and reading the undivided parameter from the external memory, and storing all the undivided parameters in the on-chip storage module, and reading the division parameter in multiple non-repeating steps, and storing the division parameter in the on-chip storage module; and calculating the division parameter using all the undivided parameters stored in the on-chip storage module when performing convolutional neural network calculations.
[0007] Optionally, based on the size relationship, one of the weights or the input feature map is segmented to obtain segmentation parameters, and the other of the weights or the input feature map that is not segmented is taken as an unsegmented parameter. This includes: under the condition that the size of the first storage space is greater than the size of the second storage space and the size of the first storage space is less than the size of the memory space, reading the weights from the external memory and storing all the weights in the on-chip storage module; and, based on the number of input channels of the computational unit of the convolutional neural network, segmenting the input feature map into multiple input feature map blocks, and reading and saving at least one different input feature map block to the on-chip storage module in stages.
[0008] Optionally, based on the size relationship, one of the weights or the input feature map is segmented to obtain a segmentation parameter, and the other of the weights or the input feature map that has not been segmented is taken as an unsegmented parameter. This includes: under the condition that the size of the second storage space is greater than the size of the first storage space and the size of the second storage space is less than the size of the memory space, traversing and reading the input feature map from the external memory and storing the entire input feature map in the on-chip storage module; and, based on the grouping of the weights relative to the convolution of the convolutional neural network, segmenting the weights into multiple weight groups, and reading and saving at least one different weight group to the on-chip storage module in stages.
[0009] Optionally, the unsegmented parameters are read from external memory and stored in full in the on-chip memory module, and the segmented parameters are read in multiple, non-repeating steps and stored in the on-chip memory module, including: reading unsegmented parameters and segmented parameters from external memory via the direct memory access channel of the hardware accelerator.
[0010] Optionally, when performing convolutional neural network calculations, the segmentation parameters are calculated using the full, unsegmented parameters stored in the on-chip storage module, including: storing the intermediate data generated during the calculation process in the computing unit of the hardware accelerator, and enabling the output reuse of the intermediate data.
[0011] Optionally, the unsegmented parameters are read from external memory and stored in full in the on-chip memory module, and the segmented parameters are read in multiple, non-repeating steps and stored in the on-chip memory module. This includes: dividing the on-chip memory module into an unsegmented parameter storage area and at least two segmented parameter storage areas for storing segmented parameters, based on their size; and controlling the two segmented parameter storage areas to work alternately through a ping-pong operation to achieve parallel operation of reading and calculating the segmented parameters.
[0012] The second technical solution adopted in this application is an external data acquisition device based on a CNN accelerator, which includes: a size relationship judgment unit, used to determine the size relationship between the first storage space size, the second storage space size, and the memory space size based on the first storage space size and the second storage space size occupied by the weights and the input feature map stored in the external memory outside the hardware accelerator, respectively, and the memory space size of the on-chip storage module of the hardware accelerator; a segmentation unit, used to segment one of the weights or the input feature map according to the size relationship to obtain segmentation parameters, and to use the other unsegmented weight or the input feature map as unsegmented parameters; a storage unit, used to traverse and read the unsegmented parameters from the external memory and store all the unsegmented parameters in the on-chip storage module, and to read the segmented parameters in multiple non-repeating steps and store the segmented parameters in the on-chip storage module; and a calculation unit, used to calculate the segmented parameters using all the unsegmented parameters stored in the on-chip storage module when performing convolutional neural network calculations.
[0013] The third technical solution adopted in this application is: a computer-readable storage medium storing a computer program / instruction, which is operated to execute the external data acquisition method based on a CNN accelerator in Solution 1.
[0014] The beneficial effects that the technical solution of this application can achieve are: the hybrid data stream formed by this application will automatically adapt to the corresponding output multiplexed data stream form according to the ratio between the input feature map and weight of the current convolutional layer, thereby improving the adaptability of the data stream. At the same time, the adaptive hybrid data stream of this application does not require cyclic access to off-chip storage data, that is, the weight and input feature data only need to be accessed once, thereby reducing memory access power consumption. Attached Figure Description
[0015] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0016] Figure 1 This is a schematic diagram of a specific implementation of an external data acquisition method for a CNN accelerator according to this application;
[0017] Figure 2 This is a schematic diagram of the input feature map multiplexing data stream in existing technologies;
[0018] Figure 3 This is a schematic diagram of the right reuse data flow in existing technology;
[0019] Figure 4 This is a schematic diagram of parallel computing using existing hardware accelerators;
[0020] Figure 5 This is a schematic diagram of the hardware accelerator structure of this application;
[0021] Figure 6 This is a mapping of the mixed data stream of this application onto a pulsating array;
[0022] Figure 7 This is a schematic diagram of a specific embodiment of an external data acquisition device for a CNN accelerator according to this application.
[0023] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation
[0024] The preferred embodiments of this application will now be described in detail with reference to the accompanying drawings, so that the advantages and features of this application can be more easily understood by those skilled in the art, thereby providing a clearer and more definite definition of the scope of protection of this application.
[0025] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising..." does not exclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element.
[0026] Neural networks are computationally intensive algorithms, with data transfer power consumption far exceeding computational power consumption, leading to the memory wall problem. Current hardware implementations of neural networks mitigate the memory wall problem through three fixed data flow methods: input data reuse, output reuse, and weight reuse. The core objective is to maximize data reuse to reduce off-chip memory access, thereby improving energy efficiency and performance.
[0027] like Figure 2 As shown, input data reuse refers to reusing input feature data. This means using the same feature data with different weights in multiple convolution calculations. It maintains the input feature map used in the calculation, iterates through all relevant weights to perform convolution calculations, and outputs the multiplication and accumulation results block by block along the output channel direction. After the calculation is complete, a new input feature map is acquired, and then all relevant weights are iterated through again for convolution calculations. Input reuse reduces the number of times the input data is accessed and the bandwidth requirement is lower, but it still requires multiple accesses to the weight data.
[0028] like Figure 3 As shown, weighted reusable data flow refers to a process where the weight parameters involved in the computation remain unchanged. Convolution calculations are performed along all relevant input feature maps through the input channel, and intermediate data is stored. After completing the output of one set of input feature maps, the weight data is updated, and convolution calculations are performed again along all relevant input feature maps. Weighted reusability reduces access to weight data and increases computational bandwidth, but it still requires multiple accesses to the input feature map data.
[0029] In summary, a single, fixed data flow pattern results in poor model adaptability and high memory access power consumption. In contrast, a hybrid computational data flow can match the needs of different models, requiring only one off-chip memory access for all weights and input feature data, and reducing memory access power consumption and time through on-chip loop scheduling. Therefore, this application achieves lower memory access power consumption and improved model versatility by employing an adaptive hybrid data flow approach on the embedded side.
[0030] The technical solution of this application and how it solves the above-mentioned technical problems will be described in detail below with specific embodiments. The specific embodiments described below can be combined with each other to form new embodiments. The same or similar ideas or processes described in one embodiment may not be repeated in other embodiments.
[0031] Figure 1 A schematic diagram of an external data acquisition method for a CNN accelerator according to this application is shown.
[0032] Figure 1 The external data acquisition method for the CNN accelerator shown includes: Step S101, determining the size relationship between the first storage space size, the second storage space size, and the memory space size based on the first storage space size and the second storage space size occupied by the weights and the input feature map stored in the external memory outside the hardware accelerator, respectively, and the memory space size of the on-chip storage module of the hardware accelerator; Step S102, dividing one of the weights or the input feature map according to the size relationship to obtain the division parameter, and taking the other undivided weight or input feature map as the undivided parameter; Step S103, reading the undivided parameter from the external memory and storing all the undivided parameters in the on-chip storage module, and reading the division parameter in multiple non-repeating steps and storing the division parameter in the on-chip storage module; Step S104, calculating the division parameter using all the undivided parameters stored in the on-chip storage module when performing convolutional neural network calculation.
[0033] Specifically, the output multiplexed data stream refers to continuously inputting feature maps and weight data during the computation process until the current output feature map data is calculated. Output multiplexing directly fixes the intermediate results generated during the calculation of the output feature map data in the computation unit, eliminating the need for additional data storage units. This design helps reduce memory access frequency and output bandwidth, and it also eliminates the need for additional cache units to store intermediate calculation results, effectively reducing memory access power consumption. Based on the output multiplexed data stream, this application dynamically adjusts the data stream format of the output multiplexed data stream to improve the adaptability of the data stream.
[0034] like Figure 4 As shown, in existing accelerators, output feature points are obtained by performing convolution operations on weights and input feature maps, and different convolution kernels output their corresponding output feature points through different output channels. The computational parallelism of CNN hardware accelerators is mainly reflected in the parallel convolution calculations of multiple output channels. That is, the hardware accelerator uses different convolution kernels computed in parallel to perform convolution calculations on different weights and different input feature maps to obtain different output results.
[0035] Based on output multiplexing and CNN hardware accelerator, this application can form a hybrid data stream according to the ratio between the input feature map and weights of the current convolutional layer, thereby automatically adapting to the corresponding output multiplexing data stream form and improving the adaptability of the data stream. Secondly, the adaptive hybrid data stream of this application does not require cyclic access to off-chip storage data, that is, the weights and input feature data only need to be accessed once, which reduces memory access power consumption.
[0036] Figure 1 The external data acquisition method for the CNN accelerator shown includes step S1O1, which determines the size relationship between the first storage space size, the second storage space size, and the memory space size based on the first storage space size and the second storage space size occupied by the weights and the input feature map stored in the external memory outside the hardware accelerator, respectively, and the memory space size of the on-chip storage module of the hardware accelerator.
[0037] Specifically, the weight information and input feature maps related to the CNN algorithm are stored on external memory outside the hardware accelerator. Before the hardware accelerator obtains the weight information and input feature maps, the control unit needs to first obtain the size of the storage space occupied by the weight data and the size of the storage space occupied by the input feature maps. At the same time, it needs to obtain the memory size of the on-chip storage module in the hardware accelerator. This third memory is used to determine which parameter data to split during subsequent data splitting and to determine which parameter data to transfer to the hardware accelerator for storage all at once.
[0038] Figure 1 The external data acquisition method for the CNN accelerator shown includes step S1O2, which involves splitting either the weights or the input feature map according to their size relationship to obtain splitting parameters, and using the other weight or input feature map that has not been split as the unsplit parameter. This step improves the adaptability of the data flow and reduces memory access power consumption.
[0039] In a specific example of this application, step S102 includes: under the condition that the size of the first storage space is greater than the size of the second storage space and the size of the first storage space is less than the size of the memory space, reading weights from the external memory and storing all weights in the on-chip storage module; and, according to the number of input channels of the computing unit of the convolutional neural network, dividing the input feature map into multiple input feature map blocks, and reading and saving at least one different input feature map block to the on-chip storage module in stages.
[0040] In a specific example of this application, step S102 includes: under the condition that the size of the second storage space is greater than the size of the first storage space and the size of the second storage space is less than the size of the memory space, reading the input feature map from the external memory and storing the full set of input feature maps in the on-chip storage module; and, according to the grouping of the weights relative to the convolution of the convolutional neural network, dividing the weights into multiple weight groups, and reading and saving at least one different weight group to the on-chip storage module in stages.
[0041] Specifically, when the input feature map is large and the weights are small, and the memory occupied by the input feature map and the memory occupied by the weights of the segmented parts are less than the total memory of the on-chip SRAM, the calculation formula can be expressed as:
[0042] ceil(C O ×C I ×K×K)+2×ceil(K×C I ×W I )≤N-2
[0043] Where ceil() is the floor function, C_I represents the number of input channels of the computation unit, W_I represents the width of the input feature map, K represents the size of the weights, C0 represents the number of output channels, N is the total memory of SRAM, and 2 is the memory unit allocated to the weight reassembly for data storage.
[0044] At this point, all weights are loaded into the accelerator's internal SRAM at once. The input feature map is then segmented according to the input channels of the computation module to obtain input feature map blocks. All data from these blocks are then input into the computation unit through all its input channels, along with the weights, to complete the calculations related to all weights in the input feature map blocks. Within the computation unit, calculations are performed along the output channels. Upon completion of each calculation, all output channels of the computation unit simultaneously output features, forming a single line of feature output.
[0045] Among them, the preferred solution for data acquisition is the alternating operation of ping-pong.
[0046] When all the input feature data can fit into the on-chip SRAM, and the weight data can fit into two sets of convolutional kernels, meaning the SRAM can hold at least two weight sets, the formula can be expressed as:
[0047] ceil(C_I×H_I×W_I)+2*ceil(SA×C_I×K×K)≤SRAM_SIZE
[0048] Where ceil() is the floor function, C_I represents the number of input channels of the computation unit, SA represents the number of columns of the computation unit, H_I represents the height of the input feature map, W_I represents the width of the input feature map, K represents the weight size, and SRAM_SIZE represents the total size of the on-chip SRAM.
[0049] At this point, input features Figure 1 All weights are initially stored in SRAM. Based on the grouping of weights relative to the convolution of the convolutional neural network, the weights are divided into multiple weight sets. These weight sets are then accessed in batches without repetition to store them in SRAM. The computation unit then retrieves the input feature map and weight sets from SRAM for computation, completing the computation of multiple complete output feature maps in each iteration. Simultaneously, a ping-pong operation is used to introduce the weight set data into SRAM, ensuring that both the weight sets used for computation and those yet to be computed exist simultaneously in SRAM. This ping-pong operation masks the computation time of the data, improving computational efficiency.
[0050] Among them, the preferred solution for data acquisition is the alternating operation of ping-pong. Figure 1 The external data acquisition method of the CNN accelerator shown includes step S1O3, which involves reading unsegmented parameters from external memory and storing all unsegmented parameters in the on-chip storage module, and reading segmented parameters in multiple non-repeating steps and storing the segmented parameters in the on-chip storage module.
[0051] In one specific embodiment of this application, step S103 includes: dividing the on-chip storage module into an unsegmented parameter storage area for storing unsegmented parameters and at least two segmented parameter storage areas for storing segmented parameters according to their size relationship; and controlling the two segmented parameter storage areas to work alternately through a ping-pong operation to achieve parallel operation of reading and calculating the segmented parameters.
[0052] To ensure computational reliability and efficiency, the internal storage module is divided into an unsegmented parameter storage area for storing unsegmented parameters and a segmented parameter storage area for storing segmented parameters. At least two segmented parameter storage areas are required. In other words, unsegmented parameters are introduced into the unsegmented parameter storage area through a ping-pong operation. When calculating unsegmented parameters, the ping-pong operation reduces access time and also enables efficient utilization of the unsegmented region's storage space.
[0053] In one specific embodiment of this application, step S103 includes: reading unsplit parameters and split parameters from external memory via the direct memory access channel of a hardware accelerator.
[0054] Specifically, such as Figure 5 As shown, the hardware structure of the accelerator in this application includes a top-level control module (TOPCONTROL), a direct memory access channel (DMA), a double data rate memory control module (ddr_ctrl), different types of data memory, and a computing array. The external memory outside the accelerator is DDR, wherein... Figure 4 The hardware results described herein are for illustrative purposes only, and the specific hardware structure can be adjusted according to the actual situation.
[0055] For example, such as Figure 5 As shown in the basic hardware structure of the accelerator, the input feature map, weights, and bias data are obtained by accessing the external DDR memory via DMA and stored in the on-chip SRAM of the hardware accelerator. The ddr_ctrl module is an important unit for implementing the mixed data stream of this application. It determines whether to perform cyclic cutting of the input feature map or weights, as well as cyclic cutting of the bias and output rearrangement. It also controls the DMA to complete the scheduling and transportation of the global data stream. Finally, the data after cutting and other processing is multiplied and accumulated through the PE computing array to output the convolution calculation result. The PE computing array here can also be replaced by other computing methods.
[0056] Figure 1 The external data acquisition method of the CNN accelerator shown includes step S1O4, which calculates the segmentation parameters using the full unsegmented parameters stored in the on-chip storage module during the calculation of the convolutional neural network.
[0057] In one specific embodiment of this application, when performing calculations on a convolutional neural network, the segmentation parameters are calculated using the full, unsegmented parameters stored in the on-chip storage module, including: storing the intermediate data generated during the calculation process in the computing unit of the hardware accelerator, and reusing the output of the intermediate data.
[0058] Specifically, the input feature map blocks and all weight data in the on-chip storage module are used for convolution processing to obtain all convolution results corresponding to the input feature map blocks. These convolution results are then stored in the hardware accelerator to achieve output reuse. Similarly, the weighted reassemblies in the on-chip storage module and all input feature map data are used for convolution processing to obtain all convolution results corresponding to the weighted reassemblies. These convolution results are also stored in the hardware accelerator to achieve output reuse.
[0059] In the accelerator of this application, the controller obtains the weights stored in the external memory located outside the hardware accelerator and the storage space size corresponding to the input feature maps, respectively. Simultaneously, the controller also needs to obtain the memory size of the on-chip storage module within the hardware accelerator. The obtained on-chip storage space size, where the external memory corresponds to… Figure 4The DDR in the chip corresponds to the on-chip memory module. Figure 4 The SARM in the controller then determines the relationship between the storage space occupied by the weights, the storage space occupied by the input feature map, and the on-chip storage space. In other words, it determines the relationship between the weights corresponding to the neural network and the weights occupied by the input feature map, as well as the relationship between the two and the SARM memory.
[0060] Then, under the condition that the storage space occupied by the weights is greater than the storage space occupied by the input feature map, and the sum of the storage space occupied by the weights and the storage space occupied by the segmented input feature map blocks is less than the on-chip storage space, all weight data and part of the input feature map blocks are directly stored in SARM via DMA; under the condition that the storage space occupied by the weights is less than the storage space occupied by the input feature map, and the sum of the memory space occupied by the segmented weighted reassemblies and the input feature map is less than the on-chip storage space, all input feature maps and part of the weighted reassemblies are directly stored in SARM via DMA. Then, using all weights or all input feature maps stored in SARM, the corresponding input feature map blocks and weighted reassemblies are processed through different convolutional kernel groups in multiple channels to obtain the output data, and the output data is stored in the computing array for reuse. This application reduces access to off-chip memory by completing all convolutional calculations corresponding to the input feature map blocks and weighted reassemblies at once, and then loading new input feature map blocks and weighted reassemblies after completion, thus achieving only one access to a set of data, thereby reducing the accelerator's memory access power consumption.
[0061] The mapping of data streams on the computing array is as follows Figure 6 As shown, the rows of the array correspond to the convolution calculation window, and the columns correspond to the convolution output channels. The data stored in SARM is transmitted to the computing unit for convolution calculation to obtain intermediate calculation results. The intermediate calculation results are then locked inside the computing unit to achieve output multiplexing.
[0062] Through the above processing, the adaptive data flow of this application has optimized the handling of feature and weight data, reducing the number of off-chip data accesses.
[0063] Figure 7 This paper illustrates a specific embodiment of an external data acquisition device based on a CNN accelerator according to this application.
[0064] exist Figure 7In the specific implementation shown, the external data acquisition device based on the CNN accelerator mainly includes: a size relationship judgment unit 701, which is used to determine the size relationship between the first storage space size, the second storage space size and the memory space size based on the first storage space size and the second storage space size occupied by the weights and the input feature map stored in the external memory outside the hardware accelerator, respectively, and the memory space size of the on-chip storage module of the hardware accelerator.
[0065] The segmentation unit 702 is used to segment either the weight or the input feature map according to their size relationship to obtain segmentation parameters, and to take the other weight or the input feature map that has not been segmented as the unsegmented parameter;
[0066] Storage unit 703 is used to traverse and read unsegmented parameters from external memory and store all unsegmented parameters in the on-chip storage module, and to read segmented parameters in multiple non-repeating steps and store the segmented parameters in the on-chip storage module;
[0067] The computing unit 704 is used to calculate the segmentation parameters using the full set of unsegmented parameters stored in the on-chip storage module when performing computations on the convolutional neural network.
[0068] The external data acquisition device based on a CNN accelerator provided in this application can be used to execute the external data acquisition method of the CNN accelerator described in any of the above embodiments. Its implementation principle and technical effect are similar, and will not be repeated here.
[0069] In another specific embodiment of this application, a computer-readable storage medium is provided, which stores a computer program / instructions that are operated to perform the external data acquisition method for the CNN accelerator described in the above embodiments.
[0070] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.
[0071] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0072] The above description is merely an embodiment of this application and does not limit the patent scope of this application. Any equivalent structural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. A method for acquiring external data based on a CNN accelerator, characterized in that, include: Based on the first storage space size and the second storage space size occupied by the weights and the input feature map stored in the external memory outside the hardware accelerator, respectively, and the memory space size of the on-chip storage module of the hardware accelerator, the size relationship between the first storage space size, the second storage space size and the memory space size is determined. Based on the size relationship, one of the weights or the input feature map is segmented to obtain segmentation parameters, and the other of the weights or the input feature map that is not segmented is taken as the unsegmented parameter; The unsegmented parameters are read from the external memory and stored in full in the on-chip memory module, and the segmented parameters are read in multiple, non-repeating steps and stored in the on-chip memory module. When performing calculations for a convolutional neural network, the segmentation parameters are calculated using all the unsegmented parameters stored in the on-chip storage module.
2. The external data acquisition method based on a CNN accelerator according to claim 1, characterized in that, The step of segmenting either the weight or the input feature map according to the size relationship to obtain segmentation parameters, and using the other unsegmented weight or the input feature map as an unsegmented parameter, includes: Given that the size of the first storage space is greater than the size of the second storage space, and the size of the first storage space is less than the size of the memory space, the weights are read from the external memory and all the weights are stored in the on-chip storage module; and, Based on the number of input channels of the computational unit of the convolutional neural network, the input feature map is divided into multiple input feature map blocks, and at least one different input feature map block is read and saved to the on-chip storage module in stages.
3. The external data acquisition method based on a CNN accelerator according to claim 1, characterized in that, The step of segmenting either the weight or the input feature map according to the size relationship to obtain segmentation parameters, and using the other unsegmented weight or the input feature map as an unsegmented parameter, includes: Under the condition that the size of the second storage space is greater than the size of the first storage space and the size of the second storage space is less than the size of the memory space, the input feature map is read from the external memory and the entire input feature map is stored in the on-chip storage module; and, Based on the grouping of the weights relative to the convolution of the convolutional neural network, the weights are divided into multiple weight groups, and at least one different weight group is read and saved to the on-chip storage module in stages.
4. The external data acquisition method based on a CNN accelerator according to claim 1, characterized in that, The steps of reading the unsegmented parameters from the external memory and storing all the unsegmented parameters in the on-chip memory module, and reading the segmented parameters in multiple, non-repeating steps and storing the segmented parameters in the on-chip memory module, include: The unsegmented parameters and the segmented parameters are read from the external memory via the direct memory access channel of the hardware accelerator.
5. The external data acquisition method based on a CNN accelerator according to claim 1, characterized in that, When performing the calculation of the convolutional neural network, the segmentation parameters are calculated using all the unsegmented parameters stored in the on-chip storage module, including: Intermediate data generated during the computation process is stored in the computing unit of the hardware accelerator, and the output of the intermediate data is reused.
6. The external data acquisition method based on a CNN accelerator according to claim 1, characterized in that, The steps of reading the unsegmented parameters from the external memory and storing all the unsegmented parameters in the on-chip memory module, and reading the segmented parameters in multiple, non-repeating steps and storing the segmented parameters in the on-chip memory module, include: Based on the size relationship, the on-chip storage module is divided into an unsegmented parameter storage area for storing the unsegmented parameters and at least two segmented parameter storage areas for storing the segmented parameters; and The two storage areas for the segmentation parameters are controlled to work alternately by a ping-pong operation, so as to achieve parallel operation of reading and calculating the segmentation parameters.
7. An external data acquisition device based on a CNN accelerator, characterized in that, include: The size relationship determination unit is used to determine the size relationship between the first storage space size, the second storage space size and the memory space size based on the first storage space size and the second storage space size occupied by the weights and the input feature map stored in the external memory outside the hardware accelerator, respectively, and the memory space size of the on-chip storage module of the hardware accelerator. The segmentation unit is used to segment either the weight or the input feature map according to the size relationship to obtain segmentation parameters, and to take the other weight or the input feature map that has not been segmented as an unsegmented parameter; A storage unit is configured to traverse and read the unsegmented parameters from the external memory and store all the unsegmented parameters in the on-chip storage module, and to read the segmented parameters in multiple, non-repeating steps and store the segmented parameters in the on-chip storage module. The computing unit is used to calculate the segmented parameters using all the unsegmented parameters stored in the on-chip storage module when performing calculations on the convolutional neural network.
8. The external data acquisition device based on a CNN accelerator according to claim 7, characterized in that, The segmentation unit includes: The system is configured to: read the weights from the external memory and store all the weights in the on-chip storage module when the size of the first storage space is greater than the size of the second storage space and the size of the first storage space is less than the size of the memory space; and divide the input feature map into multiple input feature map blocks according to the number of input channels of the computation unit of the convolutional neural network, and read and save at least one different input feature map block to a sub-unit in the on-chip storage module in stages.
9. The external data acquisition device based on a CNN accelerator according to claim 7, characterized in that, The segmentation unit includes: The system is configured to: traverse and read the input feature map from the external memory when the second storage space size is greater than the first storage space size and the second storage space size is less than the memory space size; and store the entire input feature map in the on-chip storage module; and divide the weights into multiple weight groups according to the grouping of the weights relative to the convolution of the convolutional neural network, and read and save at least one different weight group to a sub-unit in the on-chip storage module in stages.
10. A computer-readable storage medium storing a computer program / instructions, characterized in that, The computer program / instructions are operated to perform the external data acquisition method based on a CNN accelerator as described in any one of claims 1-6.