A method for accelerating joint debugging of software and hardware for neural network deployment

Abstract

The application provides a kind of to accelerate neural network deployment software and hardware joint debugging method, it is related to integrated circuit debugging technical field, its main steps are: the neural network that meets the requirement is sent into model converter and is converted, obtains the model after conversion;Quantification is carried out to the model after conversion and is sent into software quantifier, and quantification model is obtained;Quantification model is converted into the model under target platform by model converter again, and inference, evaluation, DEBUG are carried out under target platform using hardware simulator;The DEBUG analysis result of hardware simulator is output, and the weight configuration file related to hardware is output, and the weight is written into storage device.The application uses software to simulate the running process of neural network on hardware platform, provides the hardware simulator of target platform, knows the execution result of hardware in advance, and flexibly adjusts the execution process of hardware, conveniently inserts debugging point, saves the workload of debugging.

Description

Technical Field

[0001] This invention relates to the field of integrated circuit debugging technology, and in particular to a method for accelerating the joint debugging of hardware and software for neural network deployment. Background Technology

[0002] Modern neural networks are large-scale, and their training must be performed on GPU clusters. The trained network is then used on client devices such as embedded CPUs, inference NPUs, and edge FPGAs. Due to the differences between the training and inference environments, the weight parameters obtained from neural network training cannot be directly deployed to the inference environment. Even after quantizing and pruning the weights in the training environment before deployment to the inference environment, the software's data performance will differ due to the different hardware infrastructure. To address this difference, a hardware-software co-debugging method is needed to accelerate neural network deployment, ensure the accuracy of neural network deployment, and reduce debugging time.

[0003] On the other hand, the sheer size and parameter redundancy of modern neural networks make deployment at the inference end, where computing resources are limited, quite difficult. To enable deployment at the inference end, neural networks need to undergo quantization pruning. The quantized and pruned model operates differently from the original model and behaves differently on different hardware platforms. This necessitates designing operators for different platforms and verifying their correctness.

[0004] like Figure 1 As shown in the "Hardware Simulation Process" section, when the trained floating-point model is deployed on hardware with floating-point computing capabilities, if training is done on a GPU and inference is performed on a CPU, the model is converted to a float32 model supported by CPU operators; if inference is performed on a GPU, it is converted to a bfloat16 or float32 model supported by GPU operators. The converted model can run natively on either CPU or GPU. However, on FPGA / DSP / NPU, floating-point computing capabilities are relatively limited, but integer computing resources are abundant and consume relatively few resources. Therefore, the model should be quantized into an INT8 model for inference on this platform. The entire deployment process involves training, structure optimization, quantization, rewriting the PyTorch model into an HDL model, and deploying it to the FPGA via Verilog. Hardware debugging can only be performed using simulation. The simulation process can ultimately determine the correctness of the deployment process, but it is difficult to pinpoint the specific problem points. Summary of the Invention

[0005] Based on the above problems, the purpose of this invention is to use software to simulate the operation of neural networks on a hardware platform, provide a hardware simulator for the target platform, know the execution results of the hardware in advance, flexibly adjust the execution process of the hardware, conveniently insert debug points, and save debugging workload.

[0006] The technical solution adopted by this invention to achieve its objective is a method for accelerating the joint debugging of hardware and software in neural network deployment, comprising the following steps:

[0007] S1. Evaluate the neural networks that meet the requirements and send them to the model converter for conversion to obtain the converted model. The model converter can parse subgraphs and generate graph models in different formats according to different application needs. When using the PyTorch framework, the PyTorch pth format subgraph is used. It can also convert the subgraphs to the general framework ONNX format to increase software support.

[0008] S2. The converted model is sent to a software quantizer for quantization to obtain a quantized model;

[0009] S3. Convert the obtained quantization model into a model for the target platform using a model converter, and then use a hardware simulator on the target platform for inference, evaluation, and debugging.

[0010] S4. Insert DEBUG points into the required convolutional layers, use the probe function, save the results of the probe function to a file, and thus output the DEBUG analysis results of the hardware simulator.

[0011] The quantization model contains multiple layers of convolution operators. The entire model outputs inference results, and on the evaluation set, an evaluator is used to obtain some metrics of the neural network, such as mAP (mean accuracy, accuracy evaluation) and fps (images processed per second, speed indicator), which can demonstrate the performance of the model in actual deployment. During simulation or actual hardware deployment, if there is a deviation between the result of a certain layer and the result of the hardware simulator, a probe function is inserted into the tile where the deviation point is located. The probe function compares all relevant operations from the input feature map of that layer to the tile that caused the deviation with the simulation or hardware execution results, quickly and accurately locating the problem and providing a correction approach for the hardware implementation.

[0012] S5. The hardware simulator provides FPGA initialization files or binary files used by the SOC, outputs hardware-related weight configuration files, writes the weights to the storage device according to the weight configuration files, and implements the entire hardware system according to hardware instructions.

[0013] Furthermore, in step S1, the process of obtaining the qualified neural network includes the following steps:

[0014] First, using the COCO training set under the YOLOv6 neural network model, a PyTorch pth model is trained in the PyTorch framework to obtain the PyTorch pth model.

[0015] Then, the trained pth model is structurally optimized by replacing the SiLU operator with the ReLU operator and removing the backend computations that do not participate in quantization.

[0016] Finally, the optimized pth model is fed into the evaluator to assess the performance of the neural network.

[0017] Furthermore, the converted model described in step S1 is one of the following formats: Tensorflow db, Pytorch pth, PPQ, and ONNX.

[0018] Furthermore, the software quantizer mentioned in step S2 corresponds to one of the Tensorflow quantizer, PyTorch quantizer, PPQ quantizer, and ONNX quantizer.

[0019] Furthermore, in step S3, the target platform is the PyTorch framework, but it can also be the TensorFlow or ONNX framework.

[0020] The models under these frameworks are all computation graph-based models, which can be parsed into a unified computation graph data structure within the model parser, and then converted into models under the corresponding framework. This allows for development and debugging using the toolchain within the framework.

[0021] Furthermore, in step S3, the hardware simulator includes an input feature map transformation module, a convolution kernel transformation module, an EWMM operation simulation module, an output feature map inverse transformation module, a fixed-point quantization parameter generation module, an output feature map scaling module, and an output feature map shifting module.

[0022] The specific process of using a hardware simulator for reasoning, evaluation, and debugging is as follows:

[0023] First, the input feature map transformation module simulates the input conversion process of the hardware: within the input feature map transformation module, the transpose of the input conversion matrix is ​​multiplied with the input feature map, and then multiplied with the input conversion matrix to obtain the converted result. The input feature map transformation module checks for data overflow and saves the conversion result as a file that can be compared with the hardware simulation result.

[0024] Secondly, the convolution kernel transformation module simulates the input transformation process of the hardware for the convolution kernel: within the convolution kernel transformation module, the transpose of the input transformation matrix is ​​multiplied by the input convolution kernel, and then multiplied by the input transformation matrix to obtain the input transformation result. The convolution kernel transformation module checks for data overflow and saves the transformation result as a file that can be compared with the hardware simulation result.

[0025] Then, the transformed input feature map and convolution kernel are fed into the EWMM operation simulation module for cross-correlation operation to obtain the operation results in the new space;

[0026] Then, the results in the new space are inversely transformed back to the original space through the output feature map inverse transformation module: first, the output transformation matrix is ​​transposed and multiplied by the EWMM output result, and then multiplied by the output transformation matrix to obtain the final hardware EWMM operator simulation result.

[0027] Then, the fixed-point quantization parameter generation module generates the quantization bias, the converted fixed-point scaling factor and the shift factor, and statistically analyzes the data distribution to evaluate the quantization error.

[0028] Then, the output feature map scaling module performs calculations to reduce the bit width of the output features and statistically analyzes the relevant data distribution to provide a reference for hardware deployment;

[0029] Finally, the final calculation result is obtained by shifting and rounding the output feature map according to the shift factor and round_half_up rounding strategy, and saved as a coe file and a bin file, which can be compared with the simulation result or the result in the actual hardware buffer.

[0030] The core algorithm of the hardware simulator transforms the input feature map and convolution kernel into a space that can be computed in parallel. After parallel computation in this space, it is inversely transformed back to the original space. This invention uses the Winograd Minimal Filtering Algorithm (WMFA), which utilizes the overlapping computation between adjacent windows to reduce the number of multiplications required for convolution, replacing expensive multiplication operations with inexpensive addition operations. The WINOGRAD algorithm also needs to transform the calculation process of the float32 scale into an integer multiplication and shift operation, thus making full use of the hardware platform's performance.

[0031] The beneficial effects of this invention are as follows:

[0032] This invention uses software to simulate the operation of neural networks on a hardware platform. This method significantly reduces debugging time compared to traditional simulation and greatly improves the difficulty in locating problems. Using a hardware simulator allows quantization, inference, and debugging to be implemented on the software platform, eliminating the need to wait for physical hardware verification to eliminate errors. Furthermore, the hardware simulator's flexibility supports the conversion between different models and inference verification on different platforms. It provides highly visualized results, allowing users to gain a deep understanding of the model's operation at the hardware level, aiding in debugging and hardware design. In the early stages of hardware design, testing and verification through simulators can avoid the cost and time of manufacturing actual hardware prototypes. Hardware simulators are also highly scalable; based on software design, they can be relatively easily modified and upgraded to adapt to different hardware architectures and neural network models, evolving continuously with technological advancements. Debugging analysis through a hardware simulator allows for analysis of hardware performance and optimization of task execution bottlenecks. Attached Figure Description

[0033] Figure 1 This is a diagram illustrating the joint hardware and software debugging process.

[0034] Figure 2 Convert different model diagrams for the model converter;

[0035] Figure 3 A schematic diagram illustrating the quantization model of the target platform obtained by the software quantizer;

[0036] Figure 4 This is a schematic diagram of the core algorithm of the hardware simulator;

[0037] Figure 5 This example illustrates the block-based computation process of a hardware simulator. Detailed Implementation

[0038] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] Figures 1-5 This invention discloses a method for accelerating the joint debugging of hardware and software in neural network deployment, comprising the following steps:

[0040] 1. Using the YOLOv6 neural network model and the COCO training set, a PyTorch pth model is trained in the PyTorch framework. In this embodiment, all parameters in the pth model are float32 data. This model has more than 50 convolutional layers, and the network has approximately 430,000,000 floating-point parameters, which consumes considerable resources.

[0041] 2. Optimize the structure of the trained pth model by replacing the SiLU operator with the ReLU operator and removing backend calculations that do not participate in quantization; the adjusted structure can be fine-tuned to improve network accuracy.

[0042] 3. The optimized PTH model is fed into the evaluator to evaluate the performance of the neural network and ensure that the performance meets the requirements. The evaluation results of this embodiment show that it has an average prediction accuracy of 35% under the COCO evaluation tool, which is among the top performance of the currently released models.

[0043] 4. Evaluate the neural network that meets the requirements and send it to the model converter for conversion to obtain the converted ONNX model. The ONNX model is a general neural network exchange model. Software vendors and hardware vendors give priority to adapting to the ONNX model. In this embodiment, the ONNX model is used as the input for PPQ quantization to obtain INT8 quantization parameters.

[0044] 5. The converted ONNX model is fed into a software quantizer for quantization to obtain a quantized model. The quantizer can be a PyTorch quantizer, a PPQ quantizer, etc. In this embodiment, the PPQ quantizer with higher degrees of freedom is selected. The rounding strategy and quantization strategy are selected to match the FPGA hardware used in this embodiment.

[0045] 6. The obtained quantized model is converted into a PyTorch model on the FPGA chip platform independently developed by Chengdu Huawi using a model converter. Inference, evaluation, and debugging are then performed using a hardware simulator in PyTorch. The hardware simulator has the same interface as ordinary convolution operators, ensuring that the inference process is not affected by structural changes. The hardware simulator cannot read all input feature maps at once, nor can it generate all output feature maps at once; therefore, it is necessary to perform block-based computation on the input and output feature maps, generating them sequentially. The Tiling in convolution computation is as follows: Figure 5 As shown, the height, width, and number of output channels of OFM are H_out, W_out, and C_out, respectively; the height, width, and number of input channels of IFM are H_in, W_in, and C_in, respectively; the size of the convolutional kernel is K_h×K_w; and the stride of the convolutional kernel is S.

[0046] 7. Insert DEBUG points into the required convolutional layers, use various probe functions, save the results of the probe functions to a file, and thus output the DEBUG analysis results of the hardware simulator to provide software algorithm developers and hardware circuit designers with DEBUG.

[0047] 8. The hardware simulator improves the FPGA initialization file or the binary file used by the SOC, and outputs the hardware-related weight configuration file. The hardware circuit designer can write the weights to storage devices such as BRAM and DDR3 according to the weight configuration file, and implement the entire hardware system according to the hardware instructions.

[0048] In this embodiment, the hardware simulator in step 6 includes an input feature map transformation module, a convolution kernel transformation module, an EWMM operation simulation module, an output feature map inverse transformation module, a fixed-point quantization parameter generation module, an output feature map scaling module, and an output feature map shifting module.

[0049] The specific process of using a hardware simulator for reasoning, evaluation, and debugging is as follows:

[0050] First, the input feature map transformation module simulates the input conversion process of the hardware: within the input feature map transformation module, the transpose of the input conversion matrix is ​​multiplied with the input feature map, and then multiplied with the input conversion matrix to obtain the converted result. The input feature map transformation module checks for data overflow and saves the conversion result as a file that can be compared with the hardware simulation result.

[0051] Secondly, the convolution kernel transformation module simulates the input transformation process of the hardware for the convolution kernel: within the convolution kernel transformation module, the transpose of the input transformation matrix is ​​multiplied by the input convolution kernel, and then multiplied by the input transformation matrix to obtain the input transformation result. The convolution kernel transformation module checks for data overflow and saves the transformation result as a file that can be compared with the hardware simulation result.

[0052] Then, the transformed input feature map and convolution kernel are fed into the EWMM operation simulation module for cross-correlation operation to obtain the operation results in the new space;

[0053] Then, the results in the new space are inversely transformed back to the original space through the output feature map inverse transformation module: first, the output transformation matrix is ​​transposed and multiplied by the EWMM output result, and then multiplied by the output transformation matrix to obtain the final hardware EWMM operator simulation result.

[0054] Then, the fixed-point quantization parameter generation module generates the quantization bias, the converted fixed-point scaling factor and the shift factor, and statistically analyzes the data distribution to evaluate the quantization error.

[0055] Then, the output feature map scaling module performs calculations to reduce the bit width of the output features and statistically analyzes the relevant data distribution to provide a reference for hardware deployment;

[0056] Finally, the final calculation result is obtained by shifting and rounding the output feature map according to the shift factor and round_half_up rounding strategy, and saved as a coe file and a bin file, which can be compared with the simulation result or the result in the actual hardware buffer.

[0057] The above embodiments of the present invention are merely illustrative examples and are not intended to limit the implementation of the invention. Those skilled in the art can make other variations and modifications based on the above description. It is impossible to exhaustively list all possible implementations here. All obvious variations or modifications derived from the technical solutions of the present invention are still within the protection scope of the present invention.

Claims

1. A method for accelerating the joint debugging of hardware and software in neural network deployment, characterized in that, Includes the following steps: S1. The qualified neural network is sent to the model converter for conversion to obtain the converted model; S2. The converted model is sent to a software quantizer for quantization to obtain a quantized model; S3. The obtained quantization model is converted into a model under the target platform using a model converter, and inference, evaluation, and debugging are performed using a hardware simulator on the target platform. The hardware simulator includes an input feature map transformation module, a convolution kernel transformation module, an EWMM operation simulation module, an output feature map inverse transformation module, a fixed-point quantization parameter generation module, an output feature map scaling module, and an output feature map shifting module. The specific process of using a hardware simulator for reasoning, evaluation, and debugging is as follows: First, the input feature map transformation module simulates the input conversion process of the hardware: within the input feature map transformation module, the transpose of the input conversion matrix is ​​multiplied with the input feature map, and then multiplied with the input conversion matrix to obtain the converted result. The input feature map transformation module checks for data overflow and saves the conversion result as a file that can be compared with the hardware simulation result. Secondly, the convolution kernel transformation module simulates the input transformation process of the hardware for the convolution kernel: within the convolution kernel transformation module, the transpose of the input transformation matrix is ​​multiplied by the input convolution kernel, and then multiplied by the input transformation matrix to obtain the input transformation result. The convolution kernel transformation module checks for data overflow and saves the transformation result as a file that can be compared with the hardware simulation result. Then, the transformed input feature map and convolution kernel are fed into the EWMM operation simulation module for cross-correlation operation to obtain the operation results in the new space; Then, the results in the new space are inversely transformed back to the original space through the output feature map inverse transformation module: first, the output transformation matrix is ​​transposed and multiplied by the EWMM output result, and then multiplied by the output transformation matrix to obtain the final hardware EWMM operator simulation result. Then, the fixed-point quantization parameter generation module generates the quantization bias, the converted fixed-point scaling factor and the shift factor, and statistically analyzes the data distribution to evaluate the quantization error. Then, the output feature map scaling module performs calculations to reduce the bit width of the output features and statistically analyzes the relevant data distribution to provide a reference for hardware deployment; Finally, the output feature map is shifted and rounded according to the shift factor and round_half_up rounding strategy to obtain the final calculation result, which is saved as a coe file and a bin file, and can be compared with the simulation results or the results in the actual hardware buffer. S4. Insert DEBUG points in the required convolutional layers, use probe functions at the DEBUG points, save the results of the probe functions to a file, and thus output the DEBUG analysis results of the hardware simulator. S5. The hardware simulator provides FPGA initialization files or binary files used by the SOC, outputs hardware-related weight configuration files, writes the weights to the storage device according to the weight configuration files, and implements the entire hardware system according to hardware instructions.

2. The method for accelerating the joint debugging of hardware and software in neural network deployment according to claim 1, characterized in that, In step S1, the process of obtaining the qualified neural network includes the following steps: First, using the COCO training set under the YOLOv6 neural network model, a PyTorch pth model is trained in the PyTorch framework to obtain the PyTorch pth model. Then, the trained pth model is structurally optimized by replacing the SiLU operator with the ReLU operator and removing the backend computations that do not participate in quantization. Finally, the optimized pth model is fed into the evaluator to assess the performance of the neural network.

3. The method for accelerating the joint debugging of hardware and software in neural network deployment according to claim 1, characterized in that: The converted model described in step S1 is one of the following formats: Tensorflow db, Pytorch pth, PPQ, and ONNX.

4. The method for accelerating the joint debugging of hardware and software in neural network deployment according to claim 3, characterized in that: The software quantizer mentioned in step S2 corresponds to one of the Tensorflow quantizer, PyTorch quantizer, PPQ quantizer, and ONNX quantizer.

5. The method for accelerating the joint debugging of hardware and software in neural network deployment according to claim 1, characterized in that: In step S3, the target platform is the PyTorch framework, TensorFlow, or ONNX framework.

Images