Edge hardware high-energy-efficiency hybrid floating point neural network reasoning acceleration method and device

Abstract

The invention discloses an edge hardware high-energy-efficiency hybrid floating point neural network reasoning acceleration method and device, and the method comprises the steps: generating a DAG matrix for calculating and transmitting a load through analyzing the structure of a to-be-processed DNN model; a configurable hardware accelerator circuit is generated by configuring a target edge platform mixed precision pipeline floating point arithmetic unit template, meanwhile, hardware feature parameters are extracted, and hardware parameters for hardware deployment modeling are provided; joint optimization of hardware perception is carried out based on a deep reinforcement learning strategy gradient algorithm; and generating a multi-target deployment scheme oriented to the energy efficiency delay product, the operation speed or the power consumption according to the optimized operation unit configuration parameters, the cache strategy and the data flow scheduling scheme. According to the method, the technical problems of low hardware resource utilization efficiency, single optimization target and high manual tuning dependency degree in hybrid precision DNN deployment in the prior art are solved, and the technical effect of automatically generating a high-energy-efficiency complex DNN deployment scheme on a resource limited edge platform is achieved.

Description

Technical Field

[0001] The present invention relates to the field of integrated circuit and artificial intelligence technologies, and in particular to an acceleration method and device for high-energy-efficiency hybrid floating-point neural network inference on edge hardware. Background Art

[0002] In recent years, deep neural networks (DNNs) have been widely used in fields such as image recognition, object detection, and natural language processing. With the rapid development of deep learning technology, the complexity of DNN models has continued to increase, manifested in an increase in the number of parameters and a more complex network connection structure. To efficiently deploy these models on edge computing devices, current products focus on either individual software-level optimizations (such as pruning and quantization) or support for general hardware development and design. However, limited edge hardware resources make it difficult to meet the deployment requirements of high-precision hybrid floating-point quantized DNN models, which has become a major challenge facing the industry.

[0003] Currently, mature hardware with fixed-bitwidth floating-point arithmetic units (FPUs) and software neural network runtime compilers exist for deploying hybrid floating-point quantized DNN models on edge hardware. However, the software cannot be efficiently optimized for the characteristics of edge hardware, and the hardware cannot be specifically reconfigured in conjunction with the algorithm. Therefore, the greatest difficulty and challenge lies in breaking through the barriers that have long hindered the integration and optimization of software and hardware, and establishing a highly energy-efficient hybrid floating-point neural network inference acceleration method on edge hardware. Summary of the Invention

[0004] In order to solve the technical problems of inefficient hardware resource utilization, single optimization target, and high dependence on manual tuning in the deployment of mixed-precision DNN in the existing technology, the present invention provides an acceleration method and device for high-energy-efficiency hybrid floating-point neural network inference on edge hardware, which achieves the technical effect of automatically generating high-energy-efficiency complex DNN deployment solutions on resource-constrained edge platforms. Compared with the existing optimal solution, it can improve the energy efficiency index by 29.4%, and at the same time break through the technical bottleneck that traditional methods cannot deploy complex networks on small-scale hardware.

[0005] In a first aspect, the present invention provides an acceleration method capable of running hybrid floating-point neural network inference on edge hardware with high energy efficiency, the method comprising: By analyzing the structure of the DNN model to be processed, a directed acyclic graph (DAG) matrix of computation and transmission load is generated, providing software parameters for neural network software transmission and load modeling. By configuring the target edge platform's mixed-precision pipeline floating-point arithmetic unit template, a configurable hardware accelerator circuit is generated. Hardware characteristic parameters are extracted, providing hardware parameters for hardware deployment modeling. Through the deep reinforcement learning policy gradient algorithm, combined with the software parameters and hardware parameters, multi-objective optimization for energy per frame (EPF), energy efficiency or speed is achieved to generate the optimal deployment plan.

[0006] In conjunction with the first aspect, in one embodiment, a directed acyclic graph (DAG) matrix of computation and transmission loads is generated based on the structure of the DNN model to be processed, including: List the different layer types in the DNN model in order, where the layer types include convolutional layers and fully connected layers; Determine, based on the parameters of the convolutional layer, a computational load, a transmission load, and a CTR of the convolutional layer, wherein the computational load is the total number of floating-point multiplication / addition operations performed by the layer, the transmission load is the total number of parameters required for the layer to be calculated and the total number of parameters generated after the calculations are completed, and the CTR is the ratio of the computational load of the layer to the transmission load of the layer; Determining, according to the parameters of the fully connected layer, a computational load, a transmission load, and a CTR of the fully connected layer, wherein the computational load is the total number of floating-point multiplication / addition operations performed by the layer, the transmission load is the total number of parameters required for the layer to be calculated and the total number of parameters generated after the calculations are completed, and the CTR is the ratio of the computational load of the layer to the transmission load of the layer; According to the layer structure listed in the order of the DNN model, combined with the connection relationship, the node values and edge values are filled in the DAG matrix for generating the calculation and transmission load in sequence, wherein the connection relationship is the flow order of the input and output data of the DNN model, wherein the node value of the DAG matrix is a weighted node, and its value is the CTR of the layer corresponding to the DNN model, wherein the edge value of the DAG matrix is a weighted edge, and its value is the transmission amount between all the layers corresponding to the DNN model.

[0007] In conjunction with the first aspect, in one embodiment, a configurable hardware accelerator circuit is generated by configuring a mixed-precision pipeline floating-point arithmetic unit template of a target edge platform, and hardware characteristic parameters are extracted, including: Configuring a mixed-precision pipelined floating-point unit template for a target edge platform according to the input, wherein the input configuration includes the type of edge hardware, the total amount of computing resources and storage resources included, the bandwidth of the supported bus, the internal computing level, and the maximum supported operating frequency; Extracting hardware feature parameters according to the configurable hardware accelerator circuit, wherein the configurable hardware accelerator circuit includes a hard-core fixed circuit and a field programmable gate array (FPGA), wherein the fixed circuit off-chip comprises a large-capacity storage medium DRAM (Dynamic Random Access Memory), the fixed circuit off-chip comprises a bus circuit, the FPGA on-chip comprises a global buffer, and the FPGA on-chip comprises a computing unit; According to the type of FPGA, the edge hardware type and the total amount of computing resources and storage resources included are extracted; according to the Global Buffer composed of the FPGA in the chip, the bus bandwidth is extracted; according to the computing unit composed of the FPGA in the chip, the internal computing level and the maximum supported operating frequency are extracted.

[0008] In one embodiment, extracting the internal computing level and the maximum supported operating frequency based on the computing unit composed of the FPGA in the chip includes: Extracting three calculation levels based on the calculation unit composed of the FPGA, wherein the first calculation level is the basic unit for vector multiplication / addition operations in the DNN, which is composed of a variable-bit-width floating-point multiplier and a variable-bit-width floating-point adder of the hybrid floating-point hardware accelerator circuit; the second calculation level is the inter-vector parallelism of the convolutional layer or the fully connected layer, which is composed of the hybrid floating-point hardware accelerator circuit in parallel; the third calculation level is the parallel calculation of multiple convolution kernels of the convolutional layer of the DNN or multiple neurons of the fully connected layer, which is obtained by the parallelization of the inter-vector parallelism; The maximum supported operating frequency is obtained by burning EDA (Electronic Design Automation) software of the FPGA according to the RTL (Register-Transfer Level) level expression code of the hardware circuit composed of the FPGA.

[0009] In one embodiment, a hybrid floating-point hardware accelerator parallel circuit is formed by a variable bit-width floating-point multiplier and a variable bit-width floating-point adder of the hybrid floating-point hardware accelerator circuit, including: The 5-stage pipeline of the variable bit width floating-point multiplier is formed according to an exponent separation stage, a mantissa multiplication stage, an exponent alignment stage, an exponent addition stage and an output recovery stage, wherein the exponent separation stage separates the sign bit, the exponent bit and the mantissa bit of the input floating-point value, the mantissa multiplication stage completes the addition of the mantissas of the input floating-point values, the exponent alignment stage completes the recovery of the hidden bits of the mantissas, the exponent addition stage completes the exponent addition according to the result of the exponent alignment stage, and the output recovery stage integrates the floating-point value in accordance with the IEEE 754 format according to the results of the exponent separation stage, the mantissa multiplication stage, the exponent alignment stage and the exponent addition stage; The 4-stage pipeline of the variable bit width floating-point adder is formed according to an exponent separation stage, a mantissa shift stage, a mantissa addition stage, and an exponent recovery stage, wherein the exponent separation stage separates the sign bit, exponent bit, and mantissa bit of the input floating-point value, the mantissa shift stage determines the direction and number of bits of mantissa shift according to the sign bit and exponent bit obtained by the exponent separation stage, the mantissa addition stage performs mantissa addition and subtraction according to the shifted mantissa obtained by the mantissa shift stage, and the exponent recovery stage integrates the results of the exponent separation stage, the mantissa shift stage, and the mantissa addition stage to produce a floating-point value conforming to the IEEE 754 format; The floating-point hardware accelerator parallel circuit forms a floating-point multiplication-addition tree according to the variable-bit-width floating-point multiplier and the variable-bit-width floating-point adder, which is used to calculate the mixed floating-point vector operation.

[0010] In conjunction with the first aspect, in one embodiment, based on the DAG generated by the graph construction module and the hardware feature parameters extracted by the hardware design module, a deep reinforcement learning policy gradient algorithm is used to achieve multi-objective optimization for EPF, energy efficiency, or speed, and generate an optimal deployment plan, including: According to the above, the deep reinforcement learning policy gradient algorithm is used to achieve multi-objective optimization for EPF, energy efficiency or speed, and generate the optimal deployment plan, including: According to the structure of the DNN model to be processed, a DAG of computing and transmission loads is generated and divided into a set of subgraphs; Using the deep reinforcement learning agent to generate a node sequence, representing the grouping of different computing nodes in the DAG, and determining whether basic hardware constraints are met based on the grouping, including constraints on the FPGA hardware storage and computing resources, and the bus bandwidth constraint; According to the subgraph set, in combination with the mixed-precision pipeline floating-point unit of the target edge platform and the extracted hardware characteristic parameters, the maximum supported operating frequency is obtained according to the FPGA burning EDA (Electronic Design Automation) software, the operating time and power consumption are calculated, and the EPF is calculated; The reward function of the agent is calculated based on the EPF and the running time calculated according to the grouping of different computing nodes in the generated DAG, and the generated deep neural network of the agent is reversely updated to gradually approach the optimal result.

[0011] In a second aspect, the present invention provides an acceleration device capable of running hybrid floating-point neural network inference on edge hardware with high energy efficiency, characterized in that the device includes: A load modeling module, configured to generate a DAG matrix of computing and transmission loads by analyzing the structure of the DNN model to be processed in combination with the first aspect; A hardware generation module, configured to generate a configurable hardware accelerator circuit by configuring a mixed-precision pipeline floating-point arithmetic unit template of a target edge platform in combination with the first aspect, and to extract hardware characteristic parameters; The mapping deployment module is used to combine the deep reinforcement learning policy gradient algorithm described in the first aspect to achieve multi-objective optimization for EPF, energy efficiency or speed and generate the optimal deployment plan.

[0012] Compared with the prior art, the present invention has the following beneficial effects: The present invention generates a DAG representation of the computation and transmission load of the DNN model through a graph construction module, designs a mixed-precision pipeline floating-point operation unit and a cache access strategy through a hardware design module, and implements multi-objective optimization deployment through a deep reinforcement learning mapping module. This solves the technical problems in the prior art of high energy consumption, large amount of intermediate data access, and difficulty in balancing performance and cost parameters when mixed floating-point precision DNN models are deployed on edge hardware, thereby achieving the goal of efficiently deploying complex mixed-precision DNN models on edge hardware and improving deployment accuracy and efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, a brief introduction will be given below to the drawings used in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0014] Figure 1 A flowchart of a method for accelerating energy-efficient hybrid floating-point neural network inference on edge hardware provided by an embodiment of the present invention.

[0015] Figure 2 Schematic diagram of the algorithm for updating the agent RNN using the deep reinforcement learning policy gradient algorithm provided in an embodiment of the present invention.

[0016] Figure 3On the FPGA development board provided by the embodiment of the present invention, the optimization scheme data diagram of GoogLenet is deployed using objective function schemes A, B, and C respectively.

[0017] Figure 4 Schematic diagram of the internal structure of the acceleration device for high-efficiency hybrid floating-point neural network inference on edge hardware provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention. In addition, the technical features in the various embodiments or single embodiments provided by the present invention are arbitrarily combined with each other to form a new technical solution. This combination is not restricted by the sequence of steps and / or structural composition mode, but must be based on the ability of ordinary technicians in this field to implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that this combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0019] In a first aspect, an embodiment of the present invention provides an acceleration method for energy-efficient hybrid floating-point neural network inference on edge hardware.

[0020] In one embodiment, referring to Figure 1 , Figure 1 This is a flow chart of the acceleration method for high-efficiency hybrid floating-point neural network inference on edge hardware of the present invention. Figure 1 The acceleration method for energy-efficient hybrid floating-point neural network inference on edge hardware includes: S10: Generate a DAG matrix of computation and transmission load by analyzing the structure of the DNN model to be processed; For example, in this embodiment, the DNN model to be processed is GoogLeNet, a classification model commonly used in the industry with a multi-layer branching structure. By analyzing the structure of the DNN model to be processed, a DAG matrix of computing and transmission load is generated. The DNN structure to be analyzed includes convolutional layers and fully connected layers, wherein the convolutional layers include ordinary 3*3 convolutional layers, 1*1 point convolutional layers, 1*5 / 5*1 / 1*3 / 3*1 non-square convolutional layers, and the number of ordinary 3*3 convolutional layers, 1*1 point convolutional layers, 1*5 / 5*1 / 1*3 / 3*1 non-square convolutional layers and / or fully connected layers in the DNN model to be analyzed is at least one. For example, the DNN to be processed includes 10 ordinary 3*3 convolutional layers, 9 ordinary 5*5 convolutional layers, 1 ordinary 7*7 convolutional layer, 36 1*1 point convolutional layers, and 2 fully connected layers.

[0021] The exemplary classification model GoogLeNet has a total of 58 computing layers. For each computing layer, all its input parameters, output parameters, computational cost, and CTR are calculated and assigned to a DAG represented by a matrix. The filling method is expressed as formula (1):

[0022] Where DAG(i,j) in (1) represents the value of the element in the i-th row and j-th column of the DAG represented by the matrix, CTR i represents the CTR value of the i-th layer, and Data_trans(i,j) represents the amount of data transferred from the i-th layer to the j-th layer.

[0023] S20: Generate a configurable hardware accelerator circuit by configuring the mixed-precision pipeline floating-point operation unit template of the target edge platform, and extract hardware feature parameters at the same time.

[0024] Exemplarily, the computing resources, storage resources, and bus bandwidth of the edge FPGA platform are obtained, and the hardware characteristic parameters are extracted in combination with the hybrid floating-point multiplication and addition operation template. In this embodiment, the edge FPGA platform to be deployed has a 220DSP (Digital Signal Processor) and a 5040Kb BRAM (Block Random Access Memory). Combined with the three-layer configurable hybrid floating-point multiplication and addition operation template hardware accelerator circuit, the (V, C, S) pairs are extracted, where V is the internal floating-point multiplication and addition, using the parallelism of V; C refers to the floating-point multiplication and addition vectors, using the parallelism of C; S is the parallelism between convolution kernels or fully connected channels. Based on the (V, C, S) pair, the RTL-level description code of the hardware circuit is generated, and the maximum supported operating frequency is obtained through the FPGA burning EDA software.

[0025] S30: Through the deep reinforcement learning policy gradient algorithm, multi-objective optimization for EPF, energy efficiency or speed is achieved to generate the optimal deployment plan.

[0026] For example, the agent uses the recurrent neural network RNN as the main decision center to generate the grouping matrix nowmatrix, which represents the calculation group corresponding to each neural network node. First, the constraint judgment algorithm is used to determine whether the generated calculation group meets the maximum limit of the hardware's on-chip storage resources and computing resources. If it passes, the delay time t is calculated based on the extracted hardware feature parameters and the parameters of the neural network, and the power consumption p is calculated based on the hardware perception parameters. .

[0027] In one example, the objective function is set as formula (2):

[0028] in represents the objective function, represents the normalized power consumption p, represents the normalized delay time t, scaled by , can control the proportion of power consumption p and delay time t in the final objective function, which is suitable for different application scenarios. When , formula (2) degenerates into , which is equivalent to considering only the delay time t, and is suitable for scenarios that require high performance; when When , formula (2) degenerates into , which is equivalent to considering only power consumption p, and is suitable for scenarios requiring low power consumption; when When , formula (2) degenerates into , namely EPF, is suitable for balanced situations where equal power consumption and delay are required.

[0029] In one example, the algorithm definition of the deep reinforcement learning policy gradient algorithm to update the agent RNN is as shown in the attached Figure 2 shown.

[0030] In one example, on an FPGA development board, GoogLenet's optimization schemes were deployed using target function scheme A (90% Power - 10% Delay), target function scheme B (50% Power - 50% Delay), and target function scheme C (10% Power - 90% Delay). The final parameters obtained are shown in the attached figure. Figure 3 shown.

[0031] As can be seen, in this embodiment, compared to the unoptimized control group, Schemes A, B, and C perform 4.54%, 60.35%, and 64.09% faster per inference, saving EPF by 4.54%, 23.53%, and 10.35%, respectively. A horizontal comparison of the schemes shows that Scheme A, which is more power-efficient, reduces power consumption by 48.16% and 58.53% compared to Schemes B and C, respectively. Scheme B, which is more balanced, reduces total energy by 19.89% and 14.70% compared to Schemes A and C, respectively. Scheme C, which is more focused on reducing latency and improving FPS (Frame per Second) performance, reduces total latency by 61.06% and 6.23% compared to Schemes A and B, respectively. It can be seen that this embodiment solves the technical problems of inefficient hardware resource utilization, single optimization target, and high dependence on manual tuning in the existing technology of mixed-precision DNN deployment, and achieves the technical effect of automatically generating high-energy-efficiency complex DNN deployment solutions on resource-constrained edge platforms. Compared with the existing optimal solution, it can improve the energy efficiency index by 29.4%, and at the same time break through the technical bottleneck of traditional methods that cannot deploy complex networks on small-scale hardware.

[0032] In a second aspect, the present invention also provides a computing device for high-energy-efficiency hybrid floating-point neural network inference on edge hardware.

[0033] In one embodiment, referring to Figure 4 , Figure 4 This is a schematic diagram of the internal structure of the acceleration device for high-efficiency hybrid floating-point neural network inference on edge hardware of the present invention. Figure 4 As shown, the energy-efficient hybrid floating-point neural network inference acceleration device on edge hardware includes: The load modeling module S41 is used to generate a DAG matrix of computing and transmission loads by analyzing the structure of the DNN model to be processed in combination with the first aspect; A hardware generation module S42 is configured to generate a configurable hardware accelerator circuit by configuring a mixed-precision pipeline floating-point operation unit template of a target edge platform in combination with the first aspect, and to extract hardware characteristic parameters; The mapping deployment module S43 is used to combine the deep reinforcement learning policy gradient algorithm described in the first aspect to achieve multi-objective optimization for EPF, energy efficiency or speed and generate an optimal deployment plan.

[0034] Furthermore, in one embodiment, the load modeling module S41 is used to: Analyze the structure of the DNN model to be processed; Generate a DAG matrix of computation and transmission loads.

[0035] Furthermore, in one embodiment, the hardware generation module S42 is used to: Obtain computing resources, storage resources, and bus bandwidth of the edge FPGA platform, and extract hardware feature parameters based on the mixed floating-point multiplication-addition operation template; Generate RTL level representation code for hardware circuit; The maximum supported operating frequency is obtained through the FPGA burning EDA software.

[0036] Furthermore, in one embodiment, the mapping deployment module S43 is used to: Through deep reinforcement learning policy gradient algorithm, multi-objective optimization for EPF, energy efficiency or speed is achieved; Generate the optimal deployment plan.

[0037] Among them, the functions of each module of the computing device for high-energy-efficiency hybrid floating-point neural network inference on edge hardware correspond to the steps in the embodiment of the acceleration method for high-energy-efficiency hybrid floating-point neural network inference on edge hardware, and their functions and implementation processes will not be repeated here one by one.

[0038] It should be noted that the above-mentioned order of the embodiments of the present invention is only for description and does not represent the advantages or disadvantages of the embodiments.

[0039] The terms "including" and "having" and any variations thereof in the specification and claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units that are not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices. The terms "first", "second" and "third" are used to distinguish different objects, etc., and do not represent a sequence, nor do they limit the "first", "second" and "third" to different types.

[0040] In the description of the embodiments of the present invention, the words "exemplary," "for example," or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary," "for example," or "for example" in the embodiments of the present invention should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary," "for example," or "for example" is intended to present the relevant concepts in a concrete manner.

[0041] In the description of the embodiments of the present invention, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present invention, “multiple” refers to two or more than two.

[0042] In some processes described in the embodiments of the present invention, multiple operations or steps are included in a specific order. However, it should be understood that these operations or steps may not be executed in the order in which they appear in the embodiments of the present invention, or may be executed in parallel. The sequence numbers of the operations are only used to distinguish different operations and do not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations or steps may be executed in sequence or in parallel, and these operations or steps may be combined.

[0043] Through the above description of the embodiments, those skilled in the art will clearly understand that the methods of the above embodiments can be implemented using software plus the necessary general-purpose hardware platform. Of course, hardware can also be used, but in many cases the former is the more preferred implementation method. Based on this understanding, the technical solution of the present invention, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, or optical disk) as described above and includes a number of instructions for enabling a terminal device to execute the methods described in various embodiments of the present invention.

[0044] In summary, the present invention relates to a method and device for accelerating energy-efficient hybrid floating-point neural network inference on edge hardware. The method comprises: generating a DAG matrix of computational and transmission loads by analyzing the structure of a DNN model to be processed; generating a configurable hardware accelerator circuit by configuring a mixed-precision pipeline floating-point arithmetic unit template on the target edge platform, while extracting hardware characteristic parameters and providing hardware parameters for hardware deployment modeling; performing joint optimization of hardware perception based on a deep reinforcement learning policy gradient algorithm; and generating a multi-objective deployment scheme targeting energy efficiency delay product, computing speed, or power consumption based on the optimized arithmetic unit configuration parameters, cache strategy, and data flow scheduling scheme. The present invention solves the technical problems of low hardware resource utilization, single optimization target, and high reliance on manual tuning in the deployment of mixed-precision DNNs in the prior art, and achieves the technical effect of automatically generating energy-efficient complex DNN deployment schemes on resource-constrained edge platforms. Compared with the existing optimal solution, it can improve energy efficiency indicators by 29.4%, while breaking through the technical bottleneck that traditional methods cannot deploy complex networks on small-scale hardware.

[0045] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the technical solutions of the embodiments of the present invention.

Claims

1. An energy-efficient hybrid floating-point neural network inference acceleration method for edge hardware, characterized by: include: Analyze the structure of the DNN model to be processed, generate a directed acyclic graph matrix of computing and transmission loads, and provide software parameters for neural network software transmission and load modeling; Configure the target edge platform's mixed-precision pipeline floating-point unit template to generate a configurable hardware accelerator circuit. It also extracts hardware characteristic parameters and provides hardware parameters for hardware deployment modeling. Through the deep reinforcement learning policy gradient algorithm, combined with the software parameters and hardware parameters, multi-objective optimization for per-frame inference energy, energy efficiency or speed is achieved to generate the optimal deployment plan.

2. The method for accelerating edge hardware high-efficiency hybrid floating-point neural network inference according to claim 1 is characterized in that: Generates a directed acyclic graph matrix of computation and transmission loads, including: List the different layer types in the DNN model in order, where the layer types include convolutional layers and fully connected layers; Determine the computational cost, transmission cost, and CTR of the convolutional layer calculation according to the parameters of the convolutional layer, wherein the computational cost is the total number of floating-point multiplication / addition operations completed by the layer, the transmission cost is the total number of parameters required for the layer to be calculated and the total number of parameters generated after the calculation is completed, and the CTR is the ratio of the computational cost of the layer to the transmission cost of the layer; Determine the computational load, transmission load, and CTR of the fully connected layer according to the parameters of the fully connected layer; According to the layer structure listed in the order of the DNN model and the connection relationship, the node values and edge values are filled in the DAG matrix of the calculation and transmission load to be generated.

3. The method for accelerating edge hardware high-efficiency hybrid floating-point neural network inference according to claim 1 is characterized in that: Configure the target edge platform's mixed-precision pipeline floating-point unit template to generate a configurable hardware accelerator circuit and extract hardware characteristic parameters, including: Configure a mixed-precision pipelined floating-point unit template for the target edge platform based on inputs, including edge hardware type, total compute and storage resources, bus bandwidth, internal compute hierarchy, and maximum supported operating frequency. Hardware characteristic parameters are extracted according to the configurable hardware accelerator circuit, wherein the configurable hardware accelerator circuit includes a hard-core fixed circuit and a field programmable logic gate array.

4. The acceleration method for high-efficiency hybrid floating-point neural network inference on edge hardware according to claim 1 or 3, characterized in that: The configurable hardware accelerator circuit includes: DRAM, a large-capacity storage medium composed of fixed circuits outside the chip; A bus circuit composed of off-chip fixed circuits; Global Buffer composed of on-chip FPGA; The computing unit composed of on-chip FPGA.

5. The acceleration method for high-efficiency hybrid floating-point neural network inference on edge hardware according to claim 1 or 3, characterized in that: The extracting of hardware characteristic parameters includes: Extract the edge hardware type and the total amount of computing and storage resources included based on the FPGA type; Extract the bus bandwidth based on the Global Buffer composed of the on-chip FPGA; The internal computing level and the maximum supported operating frequency are extracted based on the computing units composed of the on-chip FPGA.

6. The method for accelerating edge hardware high-efficiency hybrid floating-point neural network inference according to claim 5, characterized in that: The internal calculation levels are extracted based on the calculation units composed of the on-chip FPGA, including: Three computational levels are extracted, where the first computational level is the basic unit for vector multiplication / addition operations in DNN, consisting of a variable-bit-width floating-point multiplier and a variable-bit-width floating-point adder of a hybrid floating-point hardware accelerator circuit; The second computational level is the inter-vector parallelism of the convolutional layer or the fully connected layer; The third computing level is the parallel computing of multiple convolution kernels in the DNN convolution layer or multiple neurons in the fully connected layer.

7. The method for accelerating edge hardware high-efficiency hybrid floating-point neural network inference according to claim 6, characterized in that: The variable bit width floating point multiplier comprises: The exponent separation stage is used to separate the sign bit, exponent bit and mantissa bit of the input floating-point value; The mantissa multiplication stage is used to complete the addition of the mantissas of the input floating-point values; The order of the exponent is used to recover the hidden bits of the mantissa. Exponential addition level, used to complete exponential addition of the results of its level according to the exponent code; Output recovery stage, used to integrate floating-point values compliant with the IEEE 754 format.

8. The method for accelerating edge hardware high-efficiency hybrid floating-point neural network inference according to claim 6, characterized in that: The variable bit width floating point adder comprises: The exponent separation stage is used to separate the sign bit, exponent bit and mantissa bit of the input floating-point value; The mantissa shift stage is used to determine the direction and number of mantissa shifts according to the sign bit and exponent bit obtained in the exponent separation stage; a mantissa addition stage for performing mantissa addition and subtraction according to the shifted mantissa obtained by the mantissa shift stage; The exponent recovery stage is used to integrate floating-point values that conform to the IEEE 754 format.

9. The method for accelerating edge hardware high-efficiency hybrid floating-point neural network inference according to claim 1, characterized in that: Multi-objective optimization is achieved through deep reinforcement learning policy gradient algorithm, including: Divide the DAG into a set of subgraphs; Use a deep reinforcement learning agent to generate a node sequence that represents the grouping of different computational nodes in the DAG. Determine whether the basic hardware constraints are met based on the grouping situation, including the constraints on FPGA hardware storage resources and computing resources, and bus bandwidth constraints; Calculate the running time and power consumption, and calculate the EPF; The agent’s reward function is calculated based on the EPF and running time, and the agent’s generative deep neural network is updated in reverse.

10. An energy-efficient hybrid floating-point neural network inference acceleration device for edge hardware, characterized by: The device comprises: The load modeling module is used to generate a directed acyclic graph matrix of computing and transmission loads by analyzing the structure of the DNN model to be processed; The hardware generation module is used to generate a configurable hardware accelerator circuit by configuring the mixed-precision pipeline floating-point unit template of the target edge platform and extract hardware characteristic parameters; The mapping deployment module is used to achieve multi-objective optimization for per-frame inference energy, energy efficiency, or speed through a deep reinforcement learning policy gradient algorithm to generate the optimal deployment plan.

Images