Intelligent traffic target detection method based on multi-core DSP

Abstract

A traffic target detection method based on multi-core DSP is proposed. A general convolutional neural network transplantation method is proposed for multi-core DSP platform. According to the characteristics of convolutional network structure and the architecture and instruction characteristics of multi-core DSP itself, convolutional network optimization, loop optimization, inline instruction optimization, cache optimization and other optimization methods are used to realize efficient combination optimization network performance scheme. At the same time, a multi-core processing model based on data segmentation is proposed to improve the efficiency of multi-core operation. In order to further improve the execution speed, the convolutional network is quantized, and the inference implementation of the quantized model is realized on the DSP. Finally, MobileNetV2-YOLOV3 network is successfully transplanted and implemented, and the model detection performance is tested on vehicle dataset. The final result verifies the feasibility of the application of DSP in intelligent traffic monitoring.

Description

Technical Field

[0001] This invention relates to intelligent transportation, deep learning, and object detection, specifically a traffic object detection method based on a multi-core DSP. Background Technology

[0002] Video surveillance technology is widely used in intelligent transportation systems, such as tracking vehicles and pedestrians from video feeds. Traditional techniques are hampered by complex background interference, drastic dynamic changes, and varying lighting and weather conditions, hindering their effectiveness. In recent years, the development of deep learning technology has significantly improved automatic monitoring techniques. However, due to the large number of parameters and computational demands of convolutional neural networks, and the fact that the devices used in intelligent transportation are typically edge devices with relatively limited computing power and storage, direct porting and implementation often fails to achieve the desired performance.

[0003] DSP, as a common edge computing device, is widely used in industry, such as image processing, voice processing, and industrial control. It has the characteristics of small size, low power consumption, and high performance. However, to our knowledge, there is currently no successful deployment of a model capable of performing complex traffic monitoring tasks on a DSP. The main reasons are: (1) The on-chip storage is small, and it is difficult to save network parameters and output results to the on-chip storage. There is a speed difference between off-chip storage and on-chip storage, so the kernel reads and writes data from off-chip storage at a slow rate, resulting in poor overall performance. (2) In order to improve the network execution speed, multi-core is usually used to accelerate the overall operation. Therefore, how multi-core communicates, how to divide network layer computation into tasks, and how to improve the parallel efficiency of multi-core have become the difficulties in multi-core development and implementation. In addition, performance optimization of DSP involves a variety of optimization methods, and a single method is difficult to achieve good performance. Summary of the Invention

[0004] To overcome the difficulties in applying convolutional neural networks to DSPs and their poor performance, this invention proposes an intelligent traffic target detection method based on multi-core DSPs. It combines various optimization methods and general convolutional neural network implementation methods on DSPs, and implements it on DSPs based on the lightweight network MobileNetV2-YOLOv3. Combined with intelligent traffic, it realizes an applicable pedestrian and vehicle target detection scheme.

[0005] This invention is achieved through the following technical solution:

[0006] A method for intelligent traffic target detection based on multi-core DSP includes the following steps:

[0007] 1) Single-core implementation and optimization

[0008] This paper implements a simplified version of the Darknet framework to port the object detection network to DSP. By analyzing the time consumption of each part of the network and the Darknet framework, the convolutional network is improved. Then, the specific execution process of the framework on the DSP is optimized, including loop optimization, inline instruction optimization, compiler optimization and mathematical operation optimization.

[0009] 2) Multi-core parallel acceleration

[0010] Design a master-slave model based on data partitioning to perform parallel computing. In this model, the master core first preprocesses the data. After the preprocessing is completed, the data is divided into multiple data blocks according to the number of cores. The data block addresses are transmitted to other cores through multi-core communication. After the other cores receive the addresses, they read the data and perform the corresponding operations. After the calculation is completed, the master core merges the results.

[0011] 3) Network Quantization Acceleration

[0012] By incorporating quantization techniques from convolutional networks, the original floating-point representation of weights is quantized into a low-bit fixed-point representation, resulting in a significant reduction in the number of parameters and computational cost. Furthermore, low-bit network inference operations are implemented in hardware, accelerating the execution process.

[0013] 4) MobileNetV2-YOLOV3 optimization

[0014] We designed and optimized the lightweight MobileNetV2-YOLOV3 network to verify the feasibility of the proposed optimization method. We also evaluated the model performance on the vehicle datasets KITTI and UA-DETRAC.

[0015] Furthermore, in step 1), the convolutional network structure is analyzed, and various optimization methods are adopted in combination with the characteristics of the DSP architecture to improve the execution speed of the convolutional network on the DSP.

[0016] Furthermore, in step 2), a multi-core model based on data segmentation is designed to achieve parallel computing of the network layer through multi-core synchronization and communication.

[0017] The main benefits of this invention are: it proposes a general scheme for implementing convolutional networks on DSPs and adopts various optimization methods to optimize the performance of DSP convolutional neural network porting, which greatly improves the execution speed of the network on DSPs. Attached Figure Description

[0018] Figure 1 This is the overall system flowchart.

[0019] Figure 2 This is a comparison of the time consumption of convolution operations when different numbers of kernels are involved in the operation.

[0020] Figure 3 This is a comparison of the runtime of MobileNetV2-YOLOV3 on different platforms and different datasets. Detailed Implementation

[0021] The present invention will now be further described with reference to the accompanying drawings.

[0022] Reference Figures 1-3 A method for intelligent traffic target detection based on multi-core DSP includes the following steps:

[0023] 1) Single-core implementation and optimization

[0024] This invention implements a simplified version of the Darknet framework for porting an object detection network to a DSP. Since DSPs only support C / C++ development, while the Darknet framework is implemented in C, supports both CPU and GPU computation, and is powerful enough for DSP platforms, it is necessary to remove certain code during DSP implementation. This invention improves the convolutional network by analyzing the time consumption of various parts of the network and the Darknet framework itself. Furthermore, it optimizes the framework's execution process on the DSP, including loop optimization, inline instruction optimization, compiler optimization, and mathematical operation optimization. The specific optimization methods are detailed below:

[0025] 1.1) Optimization of Convolutional Networks

[0026] Considering the higher matrix operation efficiency of DSPs, the im2col algorithm is used to transform convolution operations into matrix operations. Compared with direct convolution using a simulated sliding window, this method improves memory access efficiency and reduces computational complexity. Furthermore, the TIDSPLIB library provides the DSPF_sp_mat_mul function to accelerate matrix multiplication operations. Meanwhile, BatchNorm is widely used in convolutional network models to stabilize the output of each layer during training, but it increases computation during inference. To further reduce this computation, this invention fuses the convolutional layer and the BatchNorm layer, designing a new fusion operator to reduce the time required to compute BatchNorm separately. The fusion formula is shown below.

[0027] Convolution calculation formula:

[0028] (1);

[0029] BatchNorm calculation formula:

[0030] (2);

[0031] The formula for fusing convolution and BatchNorm is as follows:

[0032] (3);

[0033] final, w and b Assigned a new value w and b The new calculation formula is:

[0034] (4);

[0035] (5);

[0036] in and The value can be calculated before the network runs, so it will not increase the computation time of the convolutional layer;

[0037] 1.2) CACHE Optimization

[0038] To improve the efficiency of accessing off-chip data, a cache is used for optimization. The DSP includes a three-level memory: the first-level memory consists of L1D and L1P, used to store data and programs respectively; the second-level memory is L2; ​​and the third-level memory consists of the multi-core shared memory MSM and off-chip DDR space. L1D and L1P are configured as cache space by default upon power-on. In order for the data in DDR to be cached, the corresponding memory addresses need to be enabled for caching. At the same time, this invention allocates a portion of space in L2 for use as a cache. This allows more data to be prefetched into the cache, and some calculation results can also be stored in the cache. Thus, when performing the next operation, this part of the data can be read from the cache without interacting with the off-chip space, thereby improving the overall read and write performance.

[0039] 1.3) Compiler optimization

[0040] The CCS software provides four levels of compiler optimization options. As the optimization level increases, the degree of optimization also increases. Convolutional network inference involves a large number of matrix operations, and in the actual implementation, it contains a large number of loop operations. Enabling -On(n=2,3) can optimize the software pipeline and loops, reducing redundant loops.

[0041] 1.4) Optimization of Mathematical Operations

[0042] Network inference involves numerous multiplication and addition operations. When directly porting C language to DSP, the compiler defaults to using shift operations to implement multiplication and division, which consumes a lot of cycles. TI provides low-level operation libraries DSPLIB, MATHLIB, and IMGLIB. DSPLIB provides functions for matrix and vector multiplication and addition, MATHLIB provides some mathematical operation functions, and IMGLIB includes some filtering operations. For example, division operations can be replaced by the Divsp function in MATHLIB, which is more efficient than the division operation implemented in C language.

[0043] 1.5) Inline Function Optimization

[0044] Some time-consuming parts, when implemented in C, often fail to achieve the performance we expect. Some C code requires calling multiple instructions to perform calculations at the low level of the DSP, which is inefficient. TI provides inline functions to accelerate the calculation. These inline instructions correspond to assembly instructions, but compared to assembly, which requires manipulating registers for calculations, inline functions can directly manipulate constants defined in C / C++, reducing development difficulty. At the same time, inline instructions have higher execution efficiency.

[0045] 1.6) DMA acceleration

[0046] The computation process involves data copying, and DMA can be used to accelerate this process. DMA (Direct Memory Access) uses a DMA controller to copy data from one address to another. Once the kernel has configured DMA, the DMA controller performs the data copying process without kernel intervention.

[0047] 2) Multi-core parallel acceleration

[0048] To fully leverage the multi-core advantages of the DSP platform, a master-slave model based on data partitioning is designed for parallel computing. In this model, the master core first preprocesses the data, dividing it into multiple data blocks according to the number of cores. The addresses of these data blocks are then transmitted to other cores via multi-core communication. Each core receives its address, reads the data, and performs the corresponding calculations. After the calculations are complete, the master core merges the results. During convolution operations, `im2col` converts the operation into a matrix operation, allowing for partitioning by matrix rows. This data partitioning approach eliminates data dependencies between the data blocks processed by the multiple cores, thus enabling parallel computing.

[0049] Implementing a parallel model on a DSP mainly requires consideration of inter-core synchronization and communication, as well as a conflict protection mechanism for memory space. In this invention, inter-core synchronization is achieved by assigning a variable to each core to store its state. During initialization, this variable is set to 0, and after each core completes its calculation, it is set to 1. This variable allows the determination of the execution status of each core. Multi-core communication can be implemented using shared memory. The two cores that need to communicate first agree on the data storage address and space size. During execution, the sending core saves the data address to the agreed address space and notifies the receiving core to read the address. After obtaining the data address from the agreed address space, the receiving core reads the data from the corresponding address, thereby realizing data interaction between multiple cores. Meanwhile, conflict protection for memory space can be implemented using hardware semaphores. When a core accesses a corresponding resource, it acquires a semaphore, while other cores wait for the core to release the semaphore. When the core finishes its operation, it releases the semaphore, thus avoiding conflicts.

[0050] 3) Network Quantization Acceleration

[0051] Since DSP computation is more efficient at fixed-point operations than floating-point operations, this invention integrates quantization techniques for convolutional networks to further improve network speed. Quantization includes training-time quantization and post-training quantization. Compared to training-time quantization, post-training quantization does not require retraining the network and is simpler to implement. Therefore, this invention uses post-training quantization to quantize the original network. The quantization method used is the post-training quantization method in NVIDIA TensorRT. After quantization, the inference operation of the quantized model is implemented on the DSP.

[0052] 4) MobileNetV2-YOLOv3 optimization

[0053] Since most of the time consumption in the MobileNetV2-YOLOv3 network is concentrated in the depthwise separable convolutional structure, the optimization in the DSP implementation mainly focuses on this part. The depthwise separable convolutional structure includes two types: depthwise (DW) and pointwise (PW). These two structures can be implemented by converting them into matrix operations using im2col, and multi-core processing is used to accelerate their computation speed. When dividing the computation among the multiple cores, DW, due to its characteristics, has the same number of input feature map channels as the number of convolutional kernels, and each channel corresponds to a separate convolutional kernel. Therefore, the multi-core division can be based on the number of input channels. The master kernel passes the number of channels to be computed by each kernel and the corresponding channel addresses to the other kernels, and the other kernels then read the data for computation. PW, on the other hand, has the same computation method as standard convolution. Since multiple convolutional kernels share the same input feature map, the main kernel can be responsible for transforming the input feature map matrix. After transformation, the data can be divided according to the rows of the weight matrix, and the addresses of the corresponding data blocks, the addresses of the transformed input feature map matrix, and the number of rows to be calculated can be passed to other kernels for matrix operations. Secondly, the shortcut and concat structures in the network can be optimized. The shortcut structure fuses the outputs of two network layers in the pixel dimension, which can be implemented in hardware by matrix addition. This calculation can be optimized by the DSPF_sp_vecadd function in the TIDSPLLIB library. The concat structure fuses the outputs of two network layers in the channel dimension, which can be implemented by data copying. To improve the efficiency of data copying, it can be implemented by EDMA3.

[0054] The embodiments described in this specification are merely examples of implementations of the inventive concept and are for illustrative purposes only. The scope of protection of this invention should not be considered limited to the specific forms described in these embodiments; rather, it extends to equivalent technical means conceived by those skilled in the art based on the inventive concept.

Claims

1. A method for intelligent traffic target detection based on multi-core DSP, characterized in that, The method includes the following steps: 1) Single-core implementation and optimization This paper implements a simplified version of the Darknet framework to port the object detection network to DSP. By analyzing the time consumption of each part of the network and the Darknet framework, the convolutional network is improved. Then, the specific execution process of the framework on the DSP is optimized, including cache optimization, inline instruction optimization, compiler optimization and mathematical operation optimization. 2) Multi-core parallel acceleration Parallel computing is performed based on a master-slave model with data partitioning. In this model, the master core first preprocesses the data. After preprocessing, the data is divided into multiple data blocks according to the number of cores. The addresses of the data blocks are transmitted to other cores through multi-core communication. After receiving the addresses, the other cores read the data and perform the corresponding calculations. After the calculations are completed, the master core merges the results. 3) Network Quantization Acceleration By incorporating quantization techniques from convolutional networks, the original floating-point representation of weights is quantized into a low-bit fixed-point representation, resulting in a significant reduction in the number of parameters and computational cost. Furthermore, low-bit network inference operations are implemented in hardware, accelerating the execution process. 4) MobileNetV2-YOLOV3 optimization For the lightweight MobileNetV2-YOLOV3 network, design and optimization were carried out. The depthwise separable convolutional structure includes two structures: depthwise (DW) and pointwise (PW). These two structures are implemented by converting into matrix operations using im2col and employing multiple kernels to accelerate computation. When partitioning the multi-kernel operation, DW, due to its characteristics, has the same number of input feature map channels as the number of convolutional kernels, and each channel corresponds to a separate convolutional kernel. Therefore, the multi-kernel partitioning is based on the number of input channels. The master kernel passes the number of channels to be computed by each kernel and the corresponding channel addresses to the other kernels, and the other kernels then read the data for computation. The PW calculation method is consistent with standard convolution. Each convolutional kernel shares the same input feature map, so the main kernel is responsible for transforming the input feature map matrix. After transformation, the data is divided according to the rows of the weight matrix, and the corresponding data block address, the address of the transformed input feature map matrix, and the number of rows to be calculated are passed to other kernels for matrix operations. The shortcut and concat structures in the network are optimized. The shortcut structure fuses the outputs of two network layers in the pixel dimension, which is implemented in hardware by matrix addition. This calculation is optimized by the DSPF_sp_vecadd function in the TIDSPLLIB library. The concat structure fuses the outputs of two network layers in the channel dimension, so it is implemented by data copying, which is implemented by EDMA3. The feasibility of the proposed optimization method was verified, and the model performance was evaluated on the vehicle datasets KITTI and UA-DETRAC.

2. The intelligent traffic target detection method based on multi-core DSP as described in claim 1, characterized in that, In step 1), the convolutional network structure is analyzed, and various optimization methods are adopted in combination with the characteristics of the DSP architecture to improve the execution speed of the convolutional network on the DSP.

3. The intelligent traffic target detection method based on multi-core DSP as described in claim 1 or 2, characterized in that, In step 2), based on the multi-core model of data segmentation, parallel computing of the network layer is achieved through multi-core synchronization and communication.

Images