A feedback neural network model computing flow pipeline arrangement method and AI compiler

Through the feedback neural network model, the pipeline arrangement method is dynamically adjusted according to the DDR bandwidth occupancy, which solves the problem of performance degradation of AI compilers and overlapping convolutional calculations during DDR bandwidth occupancy, achieving more efficient computing performance .

CN117669668BActive Publication Date: 2024-11-26BEIJING HUIXI TECHNOLOGY CO LTD
View PDF 10 Cites 0 Cited by

Patent Information

Application Number
CN202310486544.2
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2023-05-04
Publication Date
2024-11-26
Estimated Expiration
2043-05-04

AI Technical Summary

Technical Problem

When existing AI compilers occupy DDR bandwidth, the idle time of the computing unit increases, resulting in performance degradation, and the overlap of convolutional calculations leads to an increase in the amount of computation.

Method used

The flow pipeline arrangement method is adopted for the feedback neural network model. The flow pipeline arrangement is dynamically switched according to the DDR bandwidth occupation. By adding or decreasing blocks, the number of blocks is calculated and the block loading/storage time is adjusted, and the calculation flow pipeline arrangement instructions for different DDR bandwidth occupations are generated. and switch the corresponding pipeline arrangement when DDR bandwidth preemption.

Benefits of technology

Effectively prevent performance degradation, dynamically determine the appropriate number of calculation layers, avoid the increase in calculation amount caused by convolutional layer fusion, and improve system performance.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN117669668B_ABST
    Figure CN117669668B_ABST
Patent Text Reader

Abstract

The present invention relates to a feedback neural network model calculation flow pipeline arrangement method and an AI compiler, the method comprising the following steps: S1, according to the size of block loading or block storage time, by increasing or decreasing the number of block calculations, several calculation flow pipeline arrangements with different block calculation time sizes are made, and the calculation flow pipeline arrangement instruction stream under different DDR bandwidth occupancy is compiled and generated; S2, when DDR bandwidth preemption occurs, according to the ratio of the number of block calculation cycles of the computing unit operation and the number of block loading and block storage cycles, the calculation flow pipeline arrangement under different DDR bandwidth occupancy is dynamically compiled, and the calculation flow pipeline arrangement instruction stream with equal or similar block calculation time size and block loading or block storage time size is switched. The beneficial effect is that the relatively suitable neural network model calculation flow pipeline arrangement is dynamically switched according to the DDR bandwidth occupancy.
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present invention relates to the field of computer technology, and in particular to a feedback neural network model calculation flow pipeline arrangement method and an AI compiler. [Background technology]

[0002] A compiler is a program that translates one language (usually a high-level language) into another language (usually a low-level language).

[0003] IR (Intermediate Representation) is the intermediary for translating between source code and target code during program compilation. The design of IR is very critical to the compiler. A good IR should consider the completeness of compilation from source code to target code, the ease of use and performance of compilation optimization.

[0004] Data type - Tensor: In the field of AI, calculations are abstracted into tensor calculations, which means that the main data type processed in the AI ​​compiler is also tensor, which is a very important premise.

[0005] Compared with traditional compilers, the AI ​​compiler is a domain-specific compiler with four obvious features: 1. Dynamic interpreter language front-end based on Python, 2. Multi-layer IR design, 3. Specific optimization for neural networks, and 4. Support for DSA chip architecture.

[0006] AI compilers are generally designed for some hardware structures. Figure 1 This is a common AI computing unit hardware structure diagram in the industry. Figure 1 As shown in the figure, each time the computing unit calculates, it first performs a loading process, which requires loading the input data in DDR into SRAM; then performs a calculation process, loading the data in SRAM into the vector register, and the computing unit takes data from the vector register for calculation; after calculation, the result is output to the vector register, and then the data in the vector register is stored back to SRAM; finally, a storage process is performed, storing the data in SRAM back to DDR. The speed of reading and writing data from DDR is lower than the speed of reading and writing data from SRAM.

[0007] Figure 2 This is an example diagram of the computational flow of a chain-like neural network model. Figure 2As shown in the figure, in the calculation of the AI ​​compiler, the neural network model (computational graph IR) is a directed acyclic graph composed of several computing nodes. The input and output of each computing node is a tensor. Since the SRAM in the processor is relatively small and cannot hold the entire tensor, the tensor (stored in DDR) needs to be cut into blocks. Each time a block is loaded into the SRAM for calculation, the block size is generally the same.

[0008] Problems with conventional AI compiler pipeline arrangement: Conventional AI compiler pipeline arrangement is that each block pipeline includes loading, fixed number calculations, and storage processes. Figure 3 This is a schematic diagram of the arrangement structure of the neural network model calculation flow pipeline under normal circumstances. Figure 3 As shown in the figure, the purpose of establishing a pipeline is to use the calculation time to cover up the load and store time, thereby improving system performance. For example, block 1 is in the calculation process, the calculation result of block 0 is stored from SRAM back to DDR, and block 2 is loaded from DDR to SRAM. In order to achieve a balance between the calculation time and the loading data time, each block needs to go through a fixed number of calculations after loading, so that the calculation time of the computing unit and the load / store time are not too different, thereby reducing the idle time of the computing unit. The gray part is the idle time of the computing unit. Although this fixed pipeline seems to achieve the optimal pipeline arrangement of the system, when the user opens a module at any time, if the module occupies the DDR bandwidth more obviously, the block load and block store time in the current pipeline will become longer, which increases the idle time of the computing unit, resulting in performance degradation. At this time, the pipeline arrangement efficiency is relatively poor.

[0009] In order to solve this problem, some methods will add more block compute to increase the time of each calculation. Since there are many convolution calculations in the neural network model, there will be obvious overlap when too many convolution calculations are connected together, that is, the input data block is large and the output data block is small, which increases the computational complexity of the entire model.

[0010] The Chinese invention patent with publication number CN111090464A and title "A Data Stream Processing Method and Related Equipment" discloses a data stream processing method and related equipment, including obtaining the dependency relationship and execution sequence of multiple processing units operating on data streams, generating synchronization logic according to the dependency relationship and the execution sequence; inserting the synchronization logic into the operation pipeline of each processing unit in the multiple processing units to generate executable code; improving compilation performance and data processing efficiency. However, the pipeline arrangement generated by this patent is determined to be unchanged after compilation. Regardless of the system DDR bandwidth, the pipeline arrangement is fixed, and the pipeline arrangement cannot be switched according to the DDR bandwidth occupancy.

[0011] Aiming at the technical problem that DDR bandwidth occupation in pipeline arrangement of neural network model calculation flow leads to increased idle time of computing units and overlapping convolution calculations, the present invention makes technical improvements to the pipeline arrangement method of neural network model calculation flow and the AI ​​compiler. [Summary of the invention]

[0012] The purpose of the present invention is to provide a method for dynamically switching the pipeline arrangement of the calculation flow of the relatively suitable neural network model according to the DDR bandwidth occupancy.

[0013] To achieve the above-mentioned purpose, the technical solution adopted by the present invention is a feedback neural network model calculation flow pipeline arrangement method, the neural network model includes a plurality of calculation nodes, the calculation nodes include a calculation unit, SRAM and DDR; the intermediate representation of the neural network model when compiled is a directed acyclic graph composed of a plurality of calculation nodes, the input and output of the calculation node is a tensor stored in the DDR, and the tensor is cut into a plurality of blocks; the calculation unit is a calculation flow that sequentially executes a plurality of block tasks when calculating, the block task first performs block loading, each time a block in the DDR is loaded into the SRAM, then performs block calculation, reads the block from the SRAM, outputs the result to the SRAM after calculation, and finally performs block storage, and stores the result in the SRAM back to the DDR; the calculation flow is arranged according to the pipeline, and the block calculation time of the current block task is used to cover up the block loading time of the block task executed later and the block storage time of the block task executed earlier, and the DDR bandwidth refers to the time that the block loading or block storage occupies the pipeline; the steps are included:

[0014] S1. According to the size of the block loading or block storage time, the number of block calculations is increased or decreased to make several calculation flow pipeline arrangements with different block calculation time sizes, and the calculation flow pipeline arrangement instruction flow under different DDR bandwidth occupancy is compiled and generated;

[0015] S2. When DDR bandwidth preemption occurs, dynamically switch the computing flow pipeline arrangement under different DDR bandwidth occupancy according to the ratio of the number of block computing cycles and the number of block loading and block storage cycles of the computing unit, and switch to the computing flow pipeline arrangement instruction flow with equal or similar block computing time and block loading or block storage time.

[0016] Preferably, step S1: the computing flow pipeline arrangement instruction flow includes the computing flow pipeline arrangement instruction flow when the DDR bandwidth is not preempted, the computing flow pipeline arrangement instruction flow when the DDR bandwidth is slightly preempted, and the computing flow pipeline arrangement instruction flow when the DDR bandwidth is severely preempted.

[0017] Preferably, step S2: according to condition a: the number of block calculation cycles divided by the number of block loading and block storage cycles < the minimum threshold, condition b: the number of block calculation cycles divided by the number of block loading and block storage cycles >= the maximum threshold, switch the calculation flow pipeline to arrange the instruction flow, wherein the maximum threshold and the minimum threshold are set manually.

[0018] Preferably, step S2: when DDR bandwidth preemption occurs, if the computing flow is in a computing flow pipeline working mode when the DDR bandwidth is not preempted, when condition a is met, it is switched to a computing flow pipeline arranging instruction flow when the DDR bandwidth is slightly preempted, and when condition b is met, the working mode is not switched.

[0019] Preferably, step S2: when DDR bandwidth preemption occurs, if the computing flow is in a computing flow pipeline working mode when the DDR bandwidth is slightly preempted, when condition a is met, it is switched to a computing flow pipeline arranging instruction flow when the DDR bandwidth is severely preempted; when condition b is met, it is switched to a computing flow pipeline arranging instruction flow when the DDR bandwidth is not preempted.

[0020] Preferably, step S2: when DDR bandwidth preemption occurs, if the computing flow is in a computing flow pipeline working mode when the DDR bandwidth is severely preempted, when condition a is met, the working mode is not switched, and when condition b is met, it is switched to a computing flow pipeline arrangement instruction flow when the DDR bandwidth is slightly preempted.

[0021] Preferably, before each switching of the computing flow pipeline working mode, the computing flow pipeline is required to meet condition a or condition b within a time slice, and ensure that the computing flow pipeline working mode is switched after the computing unit completes one inference calculation.

[0022] Preferably, each of the computing flow pipeline arrangement instruction streams is stored in a continuous memory. When the computing flow pipeline arrangement is switched, only the starting address of the computing flow pipeline arrangement instruction stream needs to be modified, and the memory addresses corresponding to the input, output and intermediate caches reuse the addresses corresponding to the computing flow pipeline arrangement instruction stream before switching.

[0023] Another object of the present invention is to provide an AI compiler that dynamically switches the relatively appropriate neural network model calculation flow pipeline arrangement according to DDR bandwidth occupancy.

[0024] To achieve the above-mentioned another purpose, the technical solution adopted by the present invention is a feedback neural network model calculation flow pipeline arrangement AI compiler, which is used to execute the above-mentioned feedback neural network model calculation flow pipeline arrangement method.

[0025] The beneficial effects of the neural network model computing flow pipeline arrangement method and AI compiler of the present invention are as follows: 1. It can effectively prevent the performance from being significantly degraded when other modules occupy the DDR bandwidth; 2. It can dynamically determine a more appropriate number of computing layers, thereby avoiding the technical problem of too much overlap caused by the fusion of too many convolutional layers, resulting in a large input and a small output, and a significant increase in the amount of calculation.

Brief Description of the Drawings

[0026] Figure 1 This is a common hardware structure diagram of an AI computing unit in the industry.

[0027] Figure 2 This is an example diagram of the computational flow of a chain-like neural network model.

[0028] Figure 3 This is a schematic diagram of the pipeline arrangement structure of the neural network model calculation flow under normal circumstances.

[0029] Figure 4 This is a schematic diagram of the pipeline layout structure of the neural network model computing flow when the DDR bandwidth is not preempted.

[0030] Figure 5 This is a schematic diagram of the pipeline layout structure of the neural network model calculation flow when the DDR bandwidth is slightly occupied.

[0031] Figure 6 This is a schematic diagram of the pipeline layout structure of the neural network model computing flow when the DDR bandwidth is severely occupied. [Specific implementation method]

[0032] The features and exemplary embodiments of various aspects of the present invention will be described in detail below. In the detailed description below, many specific details are proposed to provide a comprehensive understanding of the present invention. But it is obvious to those skilled in the art that the present invention can be implemented without the need for some of these specific details. The following description of the embodiments is only to provide a better understanding of the present invention by illustrating examples of the present invention. The present invention is by no means limited to any specific configuration and algorithm proposed below, but covers any modification, replacement and improvement of elements, parts and algorithms without departing from the concept of the present invention. In the accompanying drawings and the following description, known structures and technologies are not shown to avoid unnecessary ambiguity to the present invention.

[0033] Example

[0034] This embodiment implements a feedback neural network model calculation flow pipeline arrangement method.

[0035] This embodiment adopts a feedback-based compilation method through an AI compiler, and dynamically switches a more appropriate neural network model to calculate the flow arrangement mode according to the DDR bandwidth occupancy.

[0036] By the attached Figure 3 It can be seen from the calculation process of the neural network model calculation flow pipeline arrangement that when the system is running, the user can open another module at any time, which will cause the efficiency of the pipeline arrangement of the existing module to be low. Since the module can count the number of load / store cycles, the number of cycles of the computing unit running, and the number of cycles of the computing unit idle (the number of cycles is equivalent to time), this embodiment makes several corresponding pipeline arrangements according to the size of the load / store delay. First, compile and generate pipeline arrangement instruction streams under different bandwidths, and then dynamically switch the pipeline arrangement according to the ratio of the number of cycles of the computing unit running and the number of cycles of load plus store. That is, after compiling and generating the instruction stream, it will be dynamically switched during operation. Specifically: before running, when compiling, since the compiler can estimate the calculation time of a block, but the compiler cannot determine the preemption of the bandwidth at run time, it can assume several preemption situations with blocks, and compile several corresponding pipeline instruction streams accordingly; then load the instruction stream into DDR; during run time (bandwidth preemption), switch between these instruction streams according to the above switching method.

[0037] Figure 4 This is a schematic diagram of the pipeline layout structure of the neural network model computing flow when the DDR bandwidth is not preempted.

[0038] Figure 5 This is a schematic diagram of the pipeline layout structure of the neural network model calculation flow when the DDR bandwidth is slightly occupied.

[0039] Figure 6 This is a schematic diagram of the neural network model computing pipeline layout structure when the DDR bandwidth is severely occupied. Figure 4 , Figure 5 , Figure 6 As shown, this embodiment uses three different pipeline settings as an example to illustrate the implementation process of the method of this embodiment. Figure 4 , Figure 5 , Figure 6 Three pipeline arrangements with different DDR bandwidths are set up. Figure 4 This is the pipeline arrangement when the DDR bandwidth is not preempted. Figure 5 To arrange the pipeline when the DDR bandwidth is slightly occupied, the number of calculations is increased without increasing the calculation time. Figure 6 This is the pipeline arrangement when the DDR bandwidth is severely occupied.

[0040] The number of cycles run by the computing unit / (the number of cycles of load and store) < the minimum threshold (1)

[0041] The number of cycles of the computing unit running / (the number of cycles of load and store) > the maximum threshold (2)

[0042] Formulas (1) and (2) are the conditions for dynamically switching the computational flow pipeline of the neural network model, where the maximum and minimum thresholds are set artificially.

[0043] When bandwidth preemption occurs, if the system is currently in Figure 4 In the pipeline operation mode shown in the figure, the number of cycles of load and store increases, while the number of cycles of the computing unit remains unchanged. When formula (1) is satisfied, switch to Figure 5 The pipeline working mode shown in FIG. 1 is not satisfied. If the formula (1) is not satisfied and the formula (2) is satisfied, the pipeline mode will not switch. Figure 5 The pipeline mode shown satisfies formula (1) and switches to Figure 6 The pipeline working mode shown in the figure, if formula (2) is satisfied, switches to Figure 4 The pipeline operation mode shown. Figure 6 The pipeline mode shown in the figure satisfies formula (1) and does not switch. If it satisfies formula (2), it switches to Figure 5 The pipeline working mode shown.

[0044] To avoid frequent switching, before each switch, conditions (1) or (2) must be met within a time slice, and the switch can only be made after an inference is completed.

[0045] In addition, since the compiled instructions and data are stored separately, the instructions of this embodiment will be divided into several parts according to the different pipeline arrangements (this embodiment has three different pipelines, corresponding to three different instruction streams). The instruction stream corresponding to each pipeline is stored in a continuous memory. When switching, you only need to modify the starting address corresponding to the instruction stream starting address register. The memory addresses corresponding to the input, output and intermediate cache can reuse the addresses corresponding to the previous pipeline.

[0046] This embodiment can effectively prevent a significant performance degradation when other modules seize the DDR bandwidth.

[0047] This embodiment can dynamically determine a more appropriate number of computing layers, thereby avoiding the problem of too much overlap caused by the fusion of too many convolutional layers, resulting in a large input and a small output, and a significant increase in the amount of calculation.

[0048] Those skilled in the art will appreciate that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, wherein the storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0049] The above is only a preferred embodiment of the present invention. It should be pointed out that ordinary technicians in this technical field can make several improvements and supplements without departing from the principle of the present invention. These improvements and supplements should also be regarded as the scope of protection of the present invention.

Claims

1. A method for arranging a computational flow pipeline of a feedback neural network model, wherein the neural network model comprises a plurality of computational nodes, wherein the computational nodes comprise a computational unit, an SRAM and a DDR; the intermediate representation of the neural network model during compilation is a directed acyclic graph composed of a plurality of computational nodes, wherein the input and output of the computational node are a tensor stored in the DDR, and the tensor is cut into a plurality of blocks; the computational unit performs computational flow of sequentially executing a plurality of block tasks during computation, wherein the block task first performs block loading, wherein a block in the DDR is loaded into the SRAM each time, and then performs block computation, wherein the block is read from the SRAM, and the result is output to the SRAM after computation, and finally performs block storage, wherein the result in the SRAM is stored back into the DDR; the computational flow is arranged according to a pipeline, wherein the block computation time of the current block task covers up the block loading time of the block task executed later and the block storage time of the block task executed earlier, and the DDR bandwidth refers to the time that the block loading or block storage occupies the pipeline; and it is characterized in that The following steps are involved: S1. According to the size of the block loading or block storage time, the number of block calculations is increased or decreased to make several calculation flow pipeline arrangements with different block calculation time sizes, and the calculation flow pipeline arrangement instruction flow under different DDR bandwidth occupancy is compiled and generated; S2. When DDR bandwidth preemption occurs, dynamically switch the computing flow pipeline arrangement under different DDR bandwidth occupations according to the ratio of the number of block computing cycles and the number of block loading and block storage cycles of the computing unit, and switch to the computing flow pipeline arrangement instruction flow with equal or similar block computing time and block loading or block storage time; Specifically, Step S1: the computing flow pipeline arrangement instruction flow includes the computing flow pipeline arrangement instruction flow when the DDR bandwidth is not preempted, the computing flow pipeline arrangement instruction flow when the DDR bandwidth is slightly preempted, and the computing flow pipeline arrangement instruction flow when the DDR bandwidth is severely preempted; Step S2: according to condition a: the number of block calculation cycles divided by the number of block loading and block storage cycles < the minimum threshold, condition b: the number of block calculation cycles divided by the number of block loading and block storage cycles >= the maximum threshold, switch the calculation flow pipeline to arrange the instruction flow, wherein the maximum threshold and the minimum threshold are set manually; When DDR bandwidth preemption occurs, if the computing flow is in the computing flow pipeline working mode when the DDR bandwidth is not preempted, when condition a is met, it is switched to the computing flow pipeline arrangement instruction flow when the DDR bandwidth is slightly preempted, and when condition b is met, the working mode is not switched; When DDR bandwidth preemption occurs, if the computing flow is in the computing flow pipeline working mode when the DDR bandwidth is slightly preempted, when condition a is met, it is switched to the computing flow pipeline arrangement instruction flow when the DDR bandwidth is severely preempted, and when condition b is met, it is switched to the computing flow pipeline arrangement instruction flow when the DDR bandwidth is not preempted; When DDR bandwidth preemption occurs, if the computing flow is in the computing flow pipeline working mode when the DDR bandwidth is severely preempted, when condition a is met, the working mode is not switched, and when condition b is met, it is switched to the computing flow pipeline arrangement instruction flow when the DDR bandwidth is slightly preempted.

2. The method for arranging a feedback neural network model computing flow pipeline according to claim 1, characterized in that: Before each switching of the working mode of the computing flow pipeline, the computing flow pipeline is required to meet condition a or condition b within a time slice, and ensure that the computing flow pipeline working mode is switched after the computing unit completes one inference calculation.

3. The method for arranging a feedback neural network model computing flow pipeline according to claim 1, characterized in that: Each of the computing stream pipeline arrangement instruction streams is stored in a continuous memory. When the computing stream pipeline arrangement is switched, only the starting address of the computing stream pipeline arrangement instruction stream needs to be modified, and the memory addresses corresponding to the input, output and intermediate caches reuse the addresses corresponding to the computing stream pipeline arrangement instruction streams before switching.

4. A feedback neural network model computing flow pipeline arrangement AI compiler, characterized by: A feedback neural network model calculation flow pipeline arrangement method used to execute any one of claims 1 to 3.

Citation Information

Patent Citations

  • Data stream processing method and related equipment

    CN111090464A

  • Computing device and method

    CN107609642A

  • Neural network compiler architecture and compiling method

    CN110766147A

  • Data processing method based on pipeline technology and related product

    CN111338695A

  • Deep convolutional neural network acceleration method and system suitable for NPU

    CN113469350A