Low-power-consumption low-resource-occupancy cross-network-layer data flow method and hardware structure of mobile terminal equipment target detection system

By integrating the YOLO network layer on mobile devices and adopting multi-level data multiplexing and buffering design, the high-precision object detection problem under low hardware resources and low power consumption is solved, and the goals of low power consumption and low memory consumption are achieved. The detection system improves the detection performance of mobile devices.

CN119067184APending Publication Date: 2024-12-03PEKING UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202310666999.2
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-31
Publication Date
2024-12-03

AI Technical Summary

Technical Problem

It is difficult for mobile devices to efficiently deploy high-precision YOLOv2 target detection network under low hardware resources and low power consumption, resulting in excessive power consumption and memory resource usage, affecting target detection performance.

Method used

The method of cross-network layer data flow is adopted by low power consumption and low resource occupancy, and the convolution layer, activation layer and maximum pooling layer of the YOLO network model are integrated into linear convolutional layer and scale scaling layer, and multi-level data multiplexing and dual sliding are used. Window design, reducing data overload and memory footprint through parallel computing and multi-level buffering.

Benefits of technology

It significantly reduces the power consumption and memory resource usage of YOLOv2 convolutional neural network, improves the data multiplexing efficiency and performance of the target detection system, and is suitable for mobile devices with economic costs and limited battery power.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119067184A_ABST
    Figure CN119067184A_ABST
Patent Text Reader

Abstract

The invention discloses a low-power-consumption low-resource-occupancy cross-network-layer data flow method for a target detection system of mobile terminal equipment, and hardware structure design of the corresponding target detection system is carried out based on the data flow method. In the reasoning process of the YOLOv2 target detection network designed by the invention, after quantitative reconstruction, calculation of a main convolutional layer, a scale scaling layer, a maximum pooling layer and a reordering layer is carried out. The process of executing YOLOv2 network reasoning by the mobile terminal equipment target detection system comprises the following steps of: inputting an input feature map into a linear convolutional layer circuit to execute parallel MAC (Media Access Control) calculation to obtain a convolutional output feature map; inputting the convolution output feature map into a scale scaling layer circuit for scale scaling operation; if the subsequent layer is the maximum pooling layer, inputting the convolution output feature map into the maximum pooling operation of a maximum pooling layer circuit; if the subsequent layer is a reordering layer, inputting the convolution output feature map into a reordering layer circuit for reordering operation; and according to the quantized and reconstructed YOLO network structure, repeating the on-chip cross-network-layer execution with the convolutional layer as the center until calculation of all network layers in the network structure is completed. By adopting a multi-level data multiplexing strategy and a multi-level buffer design, data multiplexing of the input feature map and the multi-channel convolution kernel weight in the chip is maximized, a large amount of extra off-chip data heavy load generating power consumption and delay is reduced, the power consumption of a target detection system and occupation of on-chip memory resources are reduced to the minimum, and the detection efficiency of the target detection system is improved. Compared with the work of other related literatures, the method is more suitable for mobile terminal equipment with extremely limited hardware resources and power consumption.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention discloses a low-power, low-resource-occupancy cross-network layer data flow method and hardware structure for a mobile terminal device target detection system, and maps a high-precision YOLOv2 network model to a mobile terminal device with low power consumption and low resource conditions. The data flow strategy is based on the characteristics of the YOLOv2 network structure, and is targeted to improve the reasoning process of the YOLOv2 target detection network after quantization and compression. After the pure integer operation reasoning quantization strategy is used to quantize the compression model, the batch normalization layer subsequent to the convolution layer becomes a process of linearly compressing the feature map scale after MAC calculation and adding β-γμ / σ bias. The present invention extracts the MAC calculation, which is the most critical and suitable for hardware optimization in the convolution calculation process, from the convolution layer, optimizes it separately as a linear convolution layer, and integrates the calculation process of the remaining other steps of the convolution layer, that is, putting the bias addition, shift operation and Leaky Relu processing into a scale scaling layer for processing, so as to optimize the structure of the entire YOLO network convolution layer. For the operations between two linear convolutions of the YOLOv2 network after the quantization compression and reconstruction convolution layer, including the scale scaling layer, the maximum pooling layer, and the reordering layer, they are actually the output feature map of the previous linear convolution layer. Therefore, the present invention integrates the linear convolution layer with these network layers into a cross-network layer slice execution of the data flow strategy, and the data loading operations of each network layer and the input feature map and weight are all implemented in parallel computing. From a holistic perspective, by splitting the calculation of the entire YOLO target detection convolutional neural network into multiple cross-network layer slice executions, the entire network reasoning process of calculating the input feature map of fc channels and then outputting the output activation of fn channels can be executed in a pipeline manner, fully improving parallelism. Background Art

[0002] Edge AI systems are systems that include a large number of sensors and convolutional neural network algorithms. The data collected by sensors needs to be analyzed through the reasoning of convolutional neural network algorithms. In order to be applied in a wide range of complex environments including complex urban traffic sections and logistics sorting centers, mobile device platforms are usually used to deploy convolutional neural network algorithm models. In addition, current edge AI solutions usually use FPGAs with flexible and adjustable features to implement and execute convolutional neural network algorithms. With the rise of edge AI, a large number of edge-side mobile devices with image sensors, such as smart city transportation systems, smart home devices, drones, smart cameras, and smart glasses, need to deploy and execute target detection algorithm networks. However, due to the limitations of economic costs and battery power, such edge-side mobile devices can often only use embedded FPGAs with low hardware resources and low power consumption.

[0003] The data flow strategy is a data partitioning and data organization scheduling strategy used to solve the data reuse and data temporary storage problems when the convolutional neural network accelerator executes the convolutional neural network. It will directly affect and determine the performance and energy efficiency of the accelerator. Therefore, in order to meet the low hardware resource and low power consumption restrictions of mobile devices and the requirements of high-precision target detection, the hardware implementation of the target detection system should adopt a data flow strategy that reduces power consumption and hardware resource usage for the high-precision YOLOv2 network model after quantization and compression, realize multi-level data reuse and multi-level buffer design, save the complete input feature map and multi-channel convolution kernel weights in the off-chip DDR, and only transfer part of the input activation and weights required for network calculation to the chip each time, fully reuse them in the on-chip calculation, reduce the additional delay and power consumption caused by a large amount of off-chip data overload, and finally complete the deployment of the target detection algorithm on the mobile device.

[0004] The present invention fully exploits the characteristics of the high-precision YOLOv2 target detection network, innovatively proposes a low-power, low-resource occupancy cross-network layer data flow method for a mobile device target detection system, and completes its hardware structure design. Summary of the invention

[0005] The technical objectives of the present invention are:

[0006] In order to enable the high-precision YOLOv2 target detection convolutional neural network model to be deployed on a mobile device FPGA platform with low hardware resources and low power consumption, a targeted low-power and low-resource cross-network layer data flow strategy for the mobile device target detection system is adopted, and the hardware implementation is completed based on this strategy to reduce the power consumption and on-chip memory resource occupation of the target detection system.

[0007] The technical solution implemented by the present invention is as follows:

[0008] The low-power, low-resource-occupancy, cross-network layer data flow method and hardware structure optimization of the mobile terminal device target detection system designed by the present invention mainly include the following four aspects:

[0009] First, as Figure 1 and Figure 2 As shown, the data flow strategy of the present invention reconstructs the convolution layer, activation layer and maximum pooling layer of the YOLO network model into a linear convolution layer, a scale scaling layer and a maximum pooling layer, and integrates the reordering layer into a cross-network layer on-chip execution (COE) to achieve it. High-bandwidth FIFO connections are used between the network layers on the chip to reduce a large amount of delay and power consumption caused by unnecessary off-chip data access, and parallel computing across network layers is performed centered on the output data stream of each layer.

[0010] Second, as Figure 3 and Figure 4As shown, the present invention adopts a multi-level data reuse strategy and a dual sliding window design in the linear convolution layer, expands and reconstructs the convolution loop, adjusts fn multi-channel convolution kernels to the innermost loop, reuses the convolution kernels with a dedicated PE array, and performs parallel MAC calculations of the receptive field and PE_num convolution kernel weights on each input channel. The generated convolution intermediate results are accumulated in the output cache to obtain the convolution output activation, avoiding data overloading of the input feature map, and sliding through the input feature map to avoid data overloading of the overlapping part of the receptive field when calculating adjacent output activations. The PE array output path adopts a high-bandwidth design, expanding the bandwidth to PE_num×8bit, and simultaneously transmitting PE_num output activations output by each round of parallel MAC calculation, reducing the delay in transmitting the linear convolution results to subsequent network layers.

[0011] Thirdly, Figure 5 As shown, the present invention provides memory for data flow strategy through multi-level buffering. The on-chip buffering enables the PE array to obtain input activations and weights in a timely manner. The DDR off-chip buffering saves the complete weights and input feature maps and calculates intermediate results. Only the part required for on-chip calculation is passed to the on-chip buffering, which effectively reduces the on-chip memory resource usage.

[0012] Fourthly, for other network layers, the present invention takes PE_num output by the linear convolutional layer as a group of transmitted output feature map data streams that first traverse fn output channels of a single pixel, then traverse each column, and then traverse each row. The feature map is not temporarily stored as a whole for data reconstruction, and scale scaling, maximum pooling, reordering and other operations are directly performed on the data stream without delay. Only the corresponding on-chip buffer is designed for the intermediate results that need to be temporarily stored in the operation, so as to minimize the on-chip memory resource usage.

[0013] Compared with the traditional data flow strategy and its hardware circuit, the present invention has the following advantages:

[0014] 1. Reduced on-chip memory usage of YOLOv2 convolutional neural network hardware implementation. In the hardware structure implementation of the target detection system, a multi-level buffer design is used to provide memory for the data flow strategy, including on-chip input feature map buffer, on-chip weight buffer and DDR off-chip buffer. The on-chip input feature map buffer is used to slide through the fc channels of the input feature map from top to bottom three rows at a time, and cooperate with the PE array local buffer to slide through the input activations from front to back three columns at a time, providing a 3×3 receptive field for the MAC calculation of the PE array in time, improving its efficiency in obtaining input activations. The DDR off-chip buffer adopts a partition design to store the complete weight parameters and input feature maps, the intermediate results of convolution calculations, and the convolution output feature maps required by the routing layer. Only the input feature maps and multi-channel convolution kernel weights required for the on-chip convolution calculations are passed to the on-chip buffer, effectively reducing the on-chip memory resource usage.

[0015] 2. Reduced the power consumption of YOLOv2 convolutional neural network inference. In the hardware-implemented linear convolutional layer logic circuit of the target detection system, a multi-level data reuse strategy and a dual sliding window design are used to reconstruct the expansion method of the convolution loop, and the fn multi-channel convolution kernels are adjusted to the innermost loop, so that the input activation on each channel of the input feature map only needs to be read once to be multiplied by the weights of the corresponding positions on the fn multi-channel convolution kernels at the same time, avoiding fn-1 data overloading of all pixels in each channel of the input feature map. Two three-dimensional sliding windows are introduced to traverse the input feature map to obtain a 3×3 receptive field. The sliding traversal method avoids the data overloading of 3×2×fc input activations caused by the overlapping part of the receptive field when calculating adjacent output activations, and the input activations of the two non-edge rows and two columns on the input feature map can be reused 3×3=9 times. The data flow strategy improves the data reuse of the target detection system, reduces the data overloading that causes a lot of power consumption and delay, and reduces the system power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 Low-power and low-resource cross-network layer data flow strategy for mobile device target detection system

[0017] Figure 2 The overall architecture of the hardware implementation of the target detection system

[0018] Figure 3 Data flow using multi-level buffering and double sliding window post-convolution calculation

[0019] Figure 4 The PE array performs parallel MAC calculations for 3×3 convolutions

[0020] Figure 5 Multi-level cushioning design

[0021] Figure 7 Linear convolutional layer output data flow

[0022] Figure 6 Scaling layer design

[0023] Figure 8 Max pooling layer design

[0024] Fig. 9 Reordering layer design DETAILED DESCRIPTION

[0025] The following describes in detail the low-power, low-resource-occupancy cross-network layer data flow method and hardware structure of the mobile terminal device target detection system provided by the present invention in conjunction with the accompanying drawings, but does not constitute a limitation to the present invention.

[0026] The present invention provides a low-power, low-resource cross-network layer data flow method and hardware structure for a mobile device target detection system. The calculation of each network layer in the YOLOv2 network model inference process follows Figure 1 The process shown in the figure is based on the hardware implementation of the target detection system based on the data flow strategy. Figure 2 The steps of the data flow strategy proposed in the present invention to perform YOLOv2 network inference are as follows:

[0027] Step 1: Parallel MAC calculation of linear convolution layer. The first sliding window slides from top to bottom three rows at a time to traverse the fc channels of the input feature map. The second sliding window slides from front to back three columns at a time to traverse the three rows of input activations in the on-chip input feature map buffer to capture the receptive field. The PE array performs parallel MAC calculations of the receptive field and PE_num convolution kernel weights on fc channels, accumulates the output activations in the PE array output buffer, and then loops to perform parallel MAC calculations of the PE array to traverse fn multi-channel convolution kernels with PE_num. After the second sliding window traverses each column of the input feature map, a row of output feature maps with fn output channels can be output.

[0028] Step 2: Scaling operation of the scaling layer. Taking advantage of the fact that no off-chip data transfer is required between network layers in the same cross-network layer intra-chip execution, and the output results of the linear convolution layer are transmitted in parallel as a group of PE_num, the internal minimum functional unit PE is designed to be PE_num, and the operations of adding bias, bitwise right shift, and Leaky Relu are performed completely in parallel. The input and output data paths of the scaling layer are designed with high bandwidth, with a bandwidth of PE_num×8bit, to maximize data read and write efficiency and improve the I / O performance of the scaling layer.

[0029] Step 3: Maximum pooling operation of the maximum pooling layer. The size of the first sliding window is 1×Ofm_row×fn, and the content is a row of convolution output feature maps. Since each row of the convolution output feature map is directly passed from the previous network layer, it does not need to be temporarily stored. Then use the pooling window composed of 1×2×fn buffer as the second sliding window to traverse this row of feature maps with a span of 2, and perform maximum pooling on the two activations of each channel. The pooling result is saved in the on-chip maximum pooling buffer of Ofm_row / 2×fn. After processing the first row of the feature map, repeat the maximum pooling on the next row, and then compare it with the saved maximum pooling result of the first row of feature maps, leaving the larger activation, and process the second row in this cycle to achieve maximum pooling of the first two rows of the feature map. Continue to traverse all rows of the feature map in this way to obtain the result of the maximum pooling of the feature map.

[0030] Step 4: Reordering operation of the reordering layer. Use an on-chip reordering buffer of 1×Ofm_row / 2×4fn to store each row of the reordered feature map, which is reconstructed from two rows of convolution output feature maps. Use a 2×fn sliding window with a span of 2 to traverse the first row of convolution output feature maps, and fill them into the first half of the on-chip reordering buffer containing 1×Ofm_row / 2×2fn. Then, in the same way, fill the pixels of the second row of convolution output feature maps into the second half of the on-chip reordering buffer. Whenever a column of 4fn positions in the on-chip buffer is filled with activations, the reordered data stream can be output backwards. The output path bandwidth still adopts a high-bandwidth design to transmit PE_num activations in parallel.

[0031] Based on the above steps, the present invention completes the hardware structure design of the YOLOv2 target detection system using the low-power and low-resource cross-network layer data flow method of the mobile terminal device target detection system. By performing multi-level data reuse and using multi-level buffer design, the data reuse of the input feature map and multi-channel convolution kernel weights within the chip is maximized, and a large amount of extra off-chip data overloading that generates power consumption and delay is reduced, and the power consumption and on-chip memory resource occupation of the target detection system are reduced to a minimum. Compared with the work of other related literature, it is more suitable for mobile devices with extremely limited hardware resources and power consumption.

[0032] The above embodiments are only used to illustrate the method of the present invention rather than to limit it. A person skilled in the art may modify or make equivalent substitutions to the method of the present invention without departing from the spirit and scope of the present invention. The scope of protection of the present invention shall be subject to the claims.

Claims

1. A low-power, low-resource-occupancy cross-network layer data flow method and hardware structure for a mobile terminal device target detection system, including parallel MAC calculation of a linear convolution layer, a scaling operation of a scaling layer, a maximum pooling operation of a maximum pooling layer, and a reordering operation of a reordering layer, characterized in that: The convolutional layers, batch normalization layers, activation layers, and maximum pooling layers of the YOLOv2 network model are quantized and reconstructed into linear convolutional layers, scale scaling layers, and maximum pooling layers, and then integrated with the reordering layers into a cross-network layer on-chip execution. High-bandwidth FIFO connections are used between the network layers on the chip to reduce the large amount of delay and power consumption caused by unnecessary off-chip data access. Parallel calculations across network layers are performed with the data flow of each layer as the center. A multi-level buffer design is adopted. The DDR off-chip buffer stores the complete weights and input feature maps, and calculates the intermediate results. Only the part required for on-chip calculations is passed to the on-chip buffer, effectively reducing the on-chip memory resource usage.

2. The parallel MAC calculation method of a linear convolutional layer according to claim 1, characterized in that: The first sliding window slides from top to bottom three rows at a time to traverse the fc channels of the input feature map, and the second sliding window slides from front to back three columns at a time to traverse the three rows of input activations in the on-chip input feature map buffer to capture the receptive field. The PE array performs parallel MAC calculations of the receptive field and PE_num convolution kernel weights on fc channels, accumulates the output activations in the PE array output buffer, and then loops to perform parallel MAC calculations of the PE array to traverse fn multi-channel convolution kernels with PE_num. After the second sliding window traverses each column of the input feature map, a row of output feature maps with fn output channels can be output.

3. The scaling operation method of the scaling layer according to claim 1, characterized in that Taking advantage of the fact that no off-chip data transfer is required between network layers in the same cross-network layer on-chip execution, and the output results of the linear convolution layer are transmitted in parallel in groups of PE_num, the internal scale scaling processing PE is designed to be PE_num, and the operations of adding bias, bitwise right shift, and Leaky Relu are performed completely in parallel. The input and output data paths of the scale scaling layer are designed with high bandwidth, with a bandwidth of PE_num×8bit, which improves the I / O performance of the scale scaling layer.

4. The maximum pooling operation method of the maximum pooling layer according to claim 1, characterized in that: The size of the first sliding window is 1×Ofm_row×fn, and its content is a row of convolution output feature maps. Since each row of the convolution output feature map is directly passed from the previous network layer, it does not need to be temporarily stored. Then use the 1×2×fn pooling window as the second sliding window to traverse this row of feature maps with a stride of 2, and perform maximum pooling on the two activations of each channel. The pooling result is saved in the on-chip buffer of Ofm_row / 2×fn. After processing the first row of the feature map, repeat the maximum pooling on the next row, and then compare it with the saved maximum pooling result of the first row of the feature map, leaving the larger activation, and process the second row in this cycle to achieve the maximum pooling of the first two rows of the feature map, and continue to traverse all rows of the feature map in this way.

5. The reordering operation method of the reordering layer according to claim 1, characterized in that: A 1×Ofm_row / 2×4fn on-chip buffer is used to store each row of data of the reordered feature map, which is reconstructed from two rows of convolution output feature maps; a 2×fn sliding window is used to traverse the first row of convolution output feature maps with a span of 2, filling the first half of the on-chip buffer 1×Ofm_row / 2×2fn; then, in the same way, the second row of convolution output feature map pixels are filled into the second half of the on-chip buffer; whenever a column of 4fn positions in the on-chip buffer is filled with activations, the reordered data stream can be output backwards, and the output path adopts a high-bandwidth design to transmit PE_num activations in parallel.