Face detection acceleration design method based on ZYNQ

Abstract

The invention discloses a ZYNQ-based face detection acceleration design method, which is used for processing a face detection task and is based on an upper computer, an image acquisition module and a ZYNQ board card, and is characterized in that the image acquisition module acquires a face image by using a CMOS (Complementary Metal Oxide Semiconductor) camera and outputs original video stream data in a color restoration RGB565 format; the ZYNQ board card comprises a PS end and a PL end, the PL end comprises an input cache module, a filling module, a convolution module, a pooling module, an up-sampling module and an output cache module, and all the modules are connected in sequence; the PS end is used for overall task scheduling, reading model weight parameters in the SD card, opening a DMA channel to complete data interaction of the PL end, and storing the read data in a DDR3 for display of an upper computer. According to the method, operations such as image preprocessing, command and data sending scheduling are carried out at a PS end, and intensive calculation tasks related to an algorithm are processed at a PL end. High-speed interaction of data of a PL end and a PS end is achieved through a DMA module, data are sent in an AXI-Stream format, and high-speed operation of the system is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of target detection, and in particular to a ZYNQ-based face detection acceleration design method. Background Art

[0002] Face detection is a hot topic in the field of computer vision. It is widely used in monitoring systems, public security, attendance and access control, robot navigation and other scenarios, bringing great convenience to people's daily lives. At present, face detection algorithms based on neural networks are mostly deployed on GPUs and CPUs. However, due to the limitations of traditional CPUs in parallel pipeline computing, its control and serial computing modes are difficult to meet the requirements of neural network models for large-scale, energy-efficient parallel computing.

[0003] The ZYNQ chip combines ARM and FPGA architectures to achieve more efficient face detection algorithm deployment. By using the FPGA side to accelerate algorithm processing in ZYNQ and the ARM core to perform target detection reasoning, real-time face detection can be effectively achieved. However, existing FPGA accelerator design methods are limited by the resources on the FPGA chip, especially the RAM resources, which affects the storage capacity of the computational workload. Existing neural network acceleration methods mostly use a row cache mechanism, but for convolutional neural network feature maps that require zero padding and frequently change in size, the row cache mechanism easily causes a waste of storage resources.

[0004] In summary, the existing problems are:

[0005] 1. Zynq’s computing and storage resources are limited, making it difficult to meet high-precision and high-energy efficiency requirements. In particular, insufficient RAM resources on FPGAs limit computing and data storage capabilities.

[0006] 2. The CPU serial computing mode cannot effectively meet the neural network model's demand for large-scale parallel computing, resulting in performance bottlenecks. Summary of the invention

[0007] The present invention proposes a ZYNQ-based face detection acceleration design method, which can at least solve one of the technical problems in the background technology.

[0008] To achieve the above object, the present invention adopts the following technical solutions:

[0009] A ZYNQ-based face detection acceleration design method is used to process face detection tasks, based on a host computer, an image acquisition module and a ZYNQ board.

[0010] The image acquisition module uses a CMOS camera to capture facial images and outputs original video stream data in color-restored RGB565 format;

[0011] The ZYNQ board includes a PS end and a PL end, wherein the PL end includes an input buffer module, a padding module, a convolution module, a pooling module, an upsampling module, and an output buffer module, and each module is connected in sequence;

[0012] The PS side is used for overall task scheduling, reading the model weight parameters in the SD card, opening the DMA channel to complete the data interaction on the PL side, and storing the read data in DDR3 for display on the host computer.

[0013] Further, the following steps are included,

[0014] Step 1: The PS end obtains raw image data through an external camera or other data source. The image data is processed by the image preprocessing module and adjusted to the 416×416 pixel resolution required by the YOLOv3-tiny network model. Image preprocessing includes image scaling, grayscale conversion, normalization and other operations to ensure that the input image format matches the requirements of the network model and optimize processing efficiency.

[0015] Step 2: The PS reads the pre-trained convolutional neural network parameter file in the SD card, including the convolution kernel weights, biases, and scaling factors, and stores these parameters in the specified area of ​​DDR3 memory through the AXI-Stream interface. In this step, the network parameter storage uses on-chip cache optimization to reduce data access latency and increase reading speed;

[0016] Step 3: Use INT8 quantization to convert the parameters in the model from 32-bit floating point numbers to 8-bit integers. The amount of calculation becomes 1 / 4 of the original, requiring only one byte of space, saving memory consumption on the PS side. The quantization calculation formula is: Where r represents the original floating point parameter value, z is the zero point, s is the scaling factor, and the s conversion factor calculation formula is: where r max and r min Indicates the maximum and minimum values ​​of floating-point numbers, q max and q min Indicates the maximum and minimum values ​​represented by the fixed-point number. The calculation formula for converting a fixed-point number to zero point z is as follows:

[0017] Step 4: Input cache module: temporarily stores input data transmitted from the PS end through the FIFO cache, including feature maps, convolution kernel parameters, etc., to ensure the continuous data flow of convolution, pooling and other computing modules. The control logic manages data transmission through the AXI-Stream interface and uses registers to buffer data to avoid transmission delays. Output cache module: used to cache the calculated output data and temporarily store it through the FIFO, and then transmit it to the PS end or DDR memory through the AXI-Stream interface. The multiplexer controls the data output of multiple modules to ensure that the calculation results do not conflict;

[0018] Step 5: The padding module first generates a control signal through the row and column counters to determine the edge position, fills the edge area with zero values, and transmits actual data in the non-edge area. The data selector outputs the padded feature map according to the control signal and writes it into the FIFO buffer to ensure that the padding is completed before entering the convolution calculation. The final padded data is stored in DDR3 in sequence to provide formatted input for subsequent operations.

[0019] Step 6: The parallel convolution calculation module performs convolution operation on the input feature map and processes it using the sliding window method. The convolution kernel slides on the input image and performs product and accumulation operations with the pixels in the corresponding area of ​​the image at each position to generate an output feature map. After each operation, the sliding window moves one step to the right. When a row of convolution operations is completed, the window moves one step down to continue processing the next row of data. To improve the operation efficiency, the convolution operation adopts an 8-channel parallel method, and the parallel convolution module processes the data of multiple input and output channels at the same time. The operation results are transmitted back to the PS end through the AXI-Stream interface to achieve efficient data transmission and resource optimization, and maximize the use of FPGA resources;

[0020] Step 7: The maximum pooling layer is used to reduce the dimension of the feature map output by the convolution. The pooling operation compares adjacent pixels and takes the maximum value. The step size of the pooling layer is 2. The pipeline structure is used to accelerate the pooling process, reduce redundant information, and retain the significant features of the input image.

[0021] Step 8: Upsample the pooled feature map by copying each pixel data to expand it into a 2×2 matrix, and use RAM cache and FIFO to complete the upsampling. The upsampling operation increases the resolution of the feature map to ensure accurate feature extraction and processing at subsequent levels.

[0022] Furthermore, in step 1, the PS side obtains raw image data through an external camera or other data source. The image data is processed by the image preprocessing module and adjusted to the 416×416 pixel resolution required by the YOLOv3-tiny network model. The image preprocessing includes image scaling, grayscale transformation, normalization and other operations to ensure that the input image format matches the requirements of the network model while optimizing processing efficiency.

[0023] Furthermore, in step 2, the PS side reads the network parameter (including weights, biases, scaling factors, etc.) file (bin file) obtained by Pytorch training from the SD card (storage device), including the weights and biases of the convolution kernel, and writes it to another part of DDR3 through VDMA.

[0024] Furthermore, in step 3, INT8 quantization is used to convert the parameters in the model from 32-bit floating point numbers to 8-bit integers, and the amount of calculation becomes 1 / 4 of the original, requiring only one byte of space, saving memory consumption on the PS side. The quantization calculation formula is: Where r represents the original floating point parameter value, z is the zero point, s is the scaling factor, and the s conversion factor calculation formula is: where r max and r min Indicates the maximum and minimum values ​​of floating-point numbers, q max and q min Indicates the maximum and minimum values ​​represented by the fixed-point number. The calculation formula for converting a fixed-point number to zero point z is as follows:

[0025]

[0026] Furthermore, in step 4, the input cache module: temporarily stores the input data transmitted from the PS end through the FIFO cache, including feature maps, convolution kernel parameters, etc., to ensure the continuous data flow of convolution, pooling and other computing modules. The control logic manages data transmission through the AXI-Stream interface and uses registers to buffer data to avoid transmission delays. Output cache module: used to cache the calculated output data, and temporarily store it through the FIFO, and then transmit it to the PS end or DDR memory through the AXI-Stream interface. The multiplexer controls the data output of multiple modules to ensure that the calculation results do not conflict.

[0027] Furthermore, in step 5, in order to ensure that the size of the feature map remains unchanged or even increases, the input data is usually padded before the convolution layer of the YOLO network. The input data is first cached in the FIFO. Because the depth of the FIFO is 4096, it is impossible to cache the entire image data at one time. Through batch processing, 9 rows are cached each time. Specifically, 4096 divided by 416 is approximately equal to 9, so 9 rows of data are cached each time, and then the data is sent to the PL side through the PS side. The PS side controls the filling process through registers and passes the following information to the PL side: Site_type: Indicates the location information of the batch data currently transmitted. The 9-10 bit encoding indicates different location types. 0 indicates that the first row is included, 1 indicates the middle position, and 2 indicates that the last row is included. Col_cnt: Indicates the number of columns of the current batch data. There are 6 cases in total (416, 208, 104, 52, 11, 13). Row_cnt: Indicates the number of rows sent in the current batch. 0 indicates 1 row and 63 indicates 64 rows. Filling logic: If the first row of data is included, a row of 0s must be added before the first row, and a column of 0s must be added on both sides. If the data in the middle is being processed, just add 0s on both sides of each row. If the last row of data is included, another row of 0s must be added after the last row, and 0s must be added on both sides. The first and last rows will not be included at the same time. Position determination and data transfer: The data type is determined based on the Site_type in the register. 0 means the first row is included, 1 means the middle position, and 2 means the last row is included. At the same time, Col_cnt and Row_cnt are used to determine the number of columns and rows in the current cache to achieve batch transmission and correct filling.

[0028] Furthermore, in step 6, the core of the convolution operation is to use the convolution kernel as a sliding window, slide it on the input image, and perform product and accumulation operations with the pixels in the corresponding area of ​​the image at each position to generate a pixel in the output feature map. After each operation, the sliding window moves one step to the right. When the convolution operation of a whole row is completed, the window moves one step down and starts convolution on the next row until all rows of the image are traversed. The specific operation process is shown in the formula: where x i is the input data, w iis the convolution kernel parameter, b is the bias term, and n is the number of channels. To improve the utilization of hardware resources, the row and column dimensions of the convolution kernel are expanded, and 8 channel data are generated on the input channel and output channel to achieve parallel calculation of 8 inputs and 8 outputs. The feature map data is stored in the form of a 3×3 matrix after caching, and the convolution operation is performed simultaneously with the multi-channel data of the convolution kernel. The output is directly added to avoid caching of intermediate results. At the same time, the output channel of the feature map is expanded so that different convolution kernels can be calculated in parallel with the data of the same input channel, thereby obtaining data from different output channels of the feature map and accelerating the convolution reasoning process. For convolution layers with more input and output channels, all outputs cannot be completed at once, so they need to be processed in batches: the input or output channels are divided into multiple batches, and each batch only calculates the convolution results of some channels. This not only saves computing resources, but also improves computing efficiency. The specific operation steps are as follows: the first batch writes the input data into the FIFO buffer; the middle batch adds the data in the FIFO to the new input data and writes it into the FIFO again; the last batch adds the data in the FIFO to the current input data and directly outputs the convolution result of the current output channel.

[0029] Furthermore, in step 7, the role of the maximum pooling layer is to reduce the dimension of the input feature map to speed up the operation, reduce the number of parameters, prevent overfitting, and improve the scale invariance and rotation invariance of the model. First, the input data is delayed by one beat and assigned to a new register, so that the maximum value of two adjacent data in the same row can be obtained in the same clock. A10 is the maximum value of the first column and the second column in the first row. Similarly, B10 is the maximum value of the first column and the second column in the second row. These values ​​are written into the RAM in sequence, and the value of the RAM is read out at the same time. After a row of cache, the maximum value between every two columns in the previous row is read out, and the maximum value between every two columns in this row is written. Then, after a comparator is used to compare the written data with the read data, a maximum value of two rows and two columns can be obtained, which is the first value output after the original data is pooled. Because the step size of the pooling is 2, the valid output is taken when the input data is an even row and an even column. Through this pipeline structure, the comparison and output of the data are synchronized, which can greatly speed up the operation.

[0030] Furthermore, in step 8, the upsampling operation is used to enlarge the size of the input data. Its principle is to copy each data four times, so that the size of the output data becomes twice that of the input data. For example, if the input data is 22, then the output data is 44. Improve the resolution of the image or restore the details of the image.

[0031] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0032] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0033] It can be seen from the above technical scheme that the purpose of the present invention is to overcome the high power consumption of the existing FPGA-based face detection technology and the limited resources on the FPGA chip, and to place operations such as image preprocessing, command and data transmission scheduling on the PS side, and to place intensive computing tasks involving algorithms on the PL side. On this basis, a DMA module is used to achieve high-speed data interaction between the PL side and the PS side, and data is sent in the AXI-Stream format to achieve high-speed operation of the system.

[0034] In the present invention, the image preprocessing module obtains raw image data through an external camera and adjusts the image to a resolution of 416×416 pixels through the image preprocessing module. The image preprocessing includes image scaling, grayscale conversion, and normalization operations to ensure that the input image format matches the network model requirements and optimize processing efficiency.

[0035] In the present invention, the network parameter loading module reads the pre-trained convolutional neural network parameter file in the SD card, including convolution kernel weights, biases, and scaling factors, etc. The parameters are stored in the specified area of ​​DDR3 memory through the AXI-Stream interface, and parameter support is provided for subsequent calculation modules.

[0036] In the present invention, the quantization processing module performs INT8 quantization processing on the floating-point model parameters provided by the network parameter loading module, converting the parameters from 32-bit floating-point numbers to 8-bit integers, thereby significantly saving PS end memory resources. Quantization formula: Where r represents the original floating point parameter value, z is the zero point, s is the scaling factor, and the s conversion factor calculation formula is: where r max and r min Indicates the maximum and minimum values ​​of floating-point numbers, q max and q min Indicates the maximum and minimum values ​​represented by the fixed-point number. The calculation formula for converting a fixed-point number to zero point z is as follows:

[0037]

[0038] In the present invention, the data cache module includes an input cache and an output cache module. The input cache temporarily stores the feature map and convolution kernel parameters after quantization processing through the FIFO cache to ensure the continuity of the data flow; the output cache temporarily stores the result data after convolution calculation through the FIFO, and transmits it to the PS end or DDR memory through the AXI-Stream interface to provide efficient data support for subsequent modules. The multiplexer coordinates the data output of each module to avoid conflicts in calculation results.

[0039] In the present invention, the filling module generates a control signal through a row and column counter to determine the edge position of the feature map. The edge area is filled with zero values, and the non-edge area transmits actual data. The filling data is processed by the data selector and written into the FIFO cache to provide standardized input data for the parallel convolution module to ensure the boundary integrity of the convolution calculation.

[0040] In the present invention, the parallel convolution module obtains the standardized feature map from the input cache and the padding module, and uses the sliding window method to perform convolution operation on the input feature map. The convolution kernel and the input image area complete the pixel product and accumulation operation to generate the output feature map. The module adopts an 8-channel parallel processing mechanism to significantly improve the operation efficiency, and transmits the operation results to the data cache module through the AXI-Stream interface to achieve efficient data flow.

[0041] In the present invention, the maximum pooling module obtains data from the feature map output by the parallel convolution module and performs a dimensionality reduction operation on it. Pooling retains the significant information of the input feature map by comparing adjacent pixels and taking the maximum value. The module adopts a pipeline structure with a step size of 2 to accelerate the calculation process and reduce redundant information, providing efficient input data for the subsequent upsampling module.

[0042] In the present invention, the upsampling module expands each pixel data into a 2×2 matrix and completes the upsampling by combining RAM cache and FIFO operation. The upsampling module restores the low-resolution feature map output by the pooling module to a high-resolution feature map, providing refined data support for subsequent feature extraction and processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is a schematic diagram of the overall structure of the present invention;

[0044] Figure 2 It is a schematic diagram of the eight-channel input parallel operation structure of the present invention;

[0045] Figure 3 This is a schematic diagram of the eight-channel output parallel operation structure of the present invention;

[0046] Figure 4 It is a schematic diagram of the design structure of the batch calculation module of the present invention;

[0047] Figure 5 It is a schematic diagram of the design structure of the maximum pooling layer of the present invention;

[0048] Figure 6 This is a schematic diagram of the upsampling structure of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments.

[0050] The following is a detailed description of the ZYNQ-based face detection acceleration design method according to an embodiment of the present invention in conjunction with the accompanying drawings. Figure 1 As shown, it includes a host computer, an image acquisition module and a ZYNQ board. It includes the following steps:

[0051] Step 1: In the Vivado project, add the configuration module of the camera OV5640. The OV5640 camera uses the DVP interface as the image input and supports 720P resolution. The configuration interface uses the SCCB protocol and is connected to the outside through the EMIO on the PS side. Two external clock sources are used in the project: a 50MHz clock is used to drive the camera, and the other 24MHz clock is used to connect the DVP interface and provide clock signals. Initialize and set the camera through the SCCB interface. Next, add the camera data conversion module to convert the data of the DVP interface into the data of the AXI Stream interface. Then, add the VDMA IP core and perform related configurations, mainly including General configuration and read and write channel settings. In the General configuration, set the Number of Channels to 1, indicating that only one frame buffer is used; the Stream Data Width is set to 16 bits, indicating that each pixel occupies 16 bits; the Memory Map Data Width is set to 32 bits, which is consistent with the data width of DDR3. In the WriteChannel configuration, enable the fixed frame storage address and set the Base Address to 0x10000000, indicating the starting address of the frame buffer. The Horizontal Size is set to 12802, indicating the number of bytes per row (each pixel occupies 2 bytes), the Vertical Size is set to 720, indicating the number of rows in the frame, and the Stride is set to 12802, indicating the number of bytes between two adjacent rows. In the Read Channel configuration, the settings are consistent with the write channel, enabling the fixed frame storage address, the Base Address is 0x10000000, the Horizontal Size is 12802, the Vertical Size is 720, and the Stride is 12802. Finally, the AXI Stream interface is converted to an AXI Full interface through the AXI Interconnect and connected to the ZYNQ IP to complete data transmission and processing.

[0052] Step 2: The PS side reads the network parameters (including weights, biases, scaling factors, etc.) file (bin file) obtained by Pytorch training from the SD card (storage device), including the weights and biases of the convolution kernel, and writes it to another part of DDR3 through VDMA. And transmit it to the PL side through the SD card and AXI-DMA.

[0053] Step 3: Use INT8 quantization to convert the parameters in the model from 32-bit floating point numbers to 8-bit integers. The amount of calculation becomes 1 / 4 of the original, requiring only one byte of space, saving memory consumption on the PS side. The quantization calculation formula is: Where r represents the original floating point parameter value, z is the zero point, s is the scaling factor, and the s conversion factor calculation formula is: where r max and r min Indicates the maximum and minimum values ​​of floating-point numbers, q max and q min Indicates the maximum and minimum values ​​represented by the fixed-point number. The calculation formula for converting a fixed-point number to zero point z is as follows: First, define a quant_int8 module to convert 24-bit signed integer data input to 8-bit signed integer data output. The input parameters of this module include mult (multiplication factor), shift (number of right shifts), and zero_point (zero point offset). The internal logic of the module is as follows: Use the mult_gen_0 module to multiply data_in and mult to obtain a 39-bit signed integer result mult_rslt. When the clock rises or the reset signal is valid, the mult_rslt is right-shifted through the always statement, and zero_point is added to obtain an 8-bit signed integer result data_out. In addition, define a quant_int8_8ch module to convert 8 channels of 24-bit signed integer data input to 8 channels of 8-bit signed integer data output. The input parameters of this module are the same as those of the quant_int8 module, but a quant_int8 module is instantiated for each channel to perform the conversion. At the same time, the module uses the shift_reg module to delay the data_in_vld signal to align it with the data_out_vld signal.

[0054] Step 4: Input cache module: configure the depth and bit width of FIFO to meet the transmission requirements of feature maps and convolution kernel parameters, receive data from the PS end through the AXI-Stream interface and store it in FIFO to ensure that the data flow is stored in order. Use registers to cache the input data, and control the logic to manage the read and write timing of FIFO to avoid delay problems and ensure the continuous data flow of modules such as convolution and pooling. Output cache module: configure FIFO to cache output data to ensure the effective storage of the output of each computing module. Use a multiplexer to coordinate the data flow so that only one module outputs to the AXI-Stream interface at a time to avoid data conflicts. Data transmission and handshake control: transfer the data in FIFO to the PS end or memory through the AXI-Stream interface. Use the handshake signals value and ready to synchronize data during the transmission process, and use the counters row_cnt and col_cnt to determine whether to send the last signal to ensure the end of the line mark and ensure the order and integrity of data transmission.

[0055] Step 5: In order to ensure that the size of the feature map remains unchanged or even increases, the input data is usually padded before the convolution layer of the YOLO network. The input data is first cached in the FIFO. Because the depth of the FIFO is 4096, it is impossible to cache the entire image data at one time. Through batch processing, 9 rows are cached each time. Specifically, 4096 divided by 416 is approximately equal to 9, so 9 rows of data are cached each time, and then the data is sent to the PL side through the PS side. The PS side controls the filling process through registers and passes the following information to the PL side: site_type: indicates the location information of the batch data currently transmitted. The 9-10 bit encoding indicates different location types. 0 indicates that the first row is included, 1 indicates the middle position, and 2 indicates that the last row is included. col_cnt: indicates the number of columns of the current batch data. row_cnt: indicates the number of rows sent in the current batch. Filling logic: If the first row of data is included, a row of 0 needs to be added before the first row, and a column of 0 needs to be added on the left and right sides. If the data in the middle position is processed, just add 0 on the left and right sides of each row. If the last row of data is included, add another row of 0 after the last row, and add 0 on both sides. The first and last rows will not be included at the same time. Position judgment and data transfer: The data type is judged according to the site_type in the register. 0 means the first row is included, 1 means the middle position, and 2 means the last row is included. At the same time, col_cnt and row_cnt are used to determine the number of columns and rows in the current cache to achieve batch transmission and correct filling.

[0056] Step 6: The core of the convolution operation is to use the convolution kernel as a sliding window and slide it step by step on the input image. At each position, the convolution kernel is multiplied and accumulated with the pixels in the corresponding area of ​​the image to generate a pixel value in the output feature map. After each operation, the sliding window moves one step to the right. When the convolution operation of a row is completed, the window moves down one step and continues to convolve the next row until all rows of the image are traversed. The specific operation process is shown in the following formula: where x i is the input data, w i is the convolution kernel parameter, b is the bias term, and n is the number of channels. In order to maximize the use of hardware resources, the convolution kernel is fully expanded in both row and column dimensions, and 8 input channel data and 8 output channel data are generated in the two dimensions [O] and [I] of the input channel and output channel, respectively, to achieve 8-way input and 8-way output parallel computing. Figure 2As shown in the figure, Input represents the feature map data of different input channels, which actually exists in the form of a 3*3 matrix after being cached by the window. Kernel is a 3*3 convolution kernel, and Conv is a 3*3 weight matrix. The input channels of the feature map are expanded so that multiple input channel data of the feature map and multiple channel data of the convolution kernel can be convolved at the same time. The calculation results can be directly added to obtain the output data, avoiding the caching of the intermediate product results. Figure 3 As shown, by expanding the output channel of the feature map, the data of different convolution kernels are simultaneously convolved with the same input channel data of the feature map, and the data of different output channels of the feature map can be obtained. This parallel method maximizes the degree of parallelism and accelerates the convolution reasoning process. The present invention adopts an 8-channel input and output parallel strategy. For convolution layers with a large number of input and output channels, it is impossible to complete all outputs at once, so they need to be performed in batches. Batch calculation is to divide the input or output channels of the convolution layer into several batches, such as Figure 4 As shown in the figure, only the convolution results of some channels are calculated each time, and the final output is obtained by accumulating the results of each batch. This can reduce the consumption of computing resources and improve the computing efficiency. The specific operation steps are as follows: the first batch directly writes the input data into the FIFO cache; the middle batch reads the data from the FIFO, adds it to the input data and writes it into the FIFO again; the last batch adds the input data to the data in the FIFO and directly outputs it to obtain the final convolution result of the current output channel.

[0057] Step 7: The main function of the maximum pooling layer is to reduce the dimension of the input feature map to speed up the operation, reduce the number of parameters, prevent overfitting, and enhance the scale and rotation invariance of the model. The implementation process is as follows: First, the input data is delayed by one beat and stored in a new register, so as to obtain the maximum value of two adjacent data in the same row in the same clock cycle. For example, A10 is the maximum value of the first column and the second column of the first row, and B10 is the maximum value of the first column and the second column of the second row. Write these maximum values ​​to the RAM in sequence, and read the maximum value cache of the previous row in the RAM at the same time to obtain the maximum value between every two columns. Next, the comparator compares the maximum value of the current row with the maximum value of the previous row to obtain a global maximum value in a 2×2 area, which is the first value output by the pooling operation. Since the pooling step size is 2, valid values ​​are output only for even rows and even columns of the input data. The operation process is as follows Figure 5 As shown in the figure, this pipeline structure enables data comparison and output synchronization, significantly improving the operation speed.

[0058] Step 8: The upsampling module increases the image size by copying each input pixel to multiple locations, usually by a factor of 2. The upsampling operation is implemented using a RAM cache, such as Figure 6As shown, in odd rows, a set of data is read from the RAM every other clock cycle and written into register REG1. The data in REG1 is delayed by one beat and stored in another register REG2, thereby generating the upsampled data of the current row. At this time, the values ​​of REG1 and REG2 represent two rows of data after upsampling of the same row. These two rows of upsampled data (REG1 and REG2) are written into the FIFO for cache so that they can be output in even rows. In even rows, new data is not read from the RAM, but the upsampled data cached in the odd rows is read out from the FIFO. The read data is sent through the AXI Stream interface to form the upsampled output of the even rows. The alternating output of odd and even rows achieves a 2x upsampling effect in the vertical and horizontal directions.

[0059] At this point, the low-power face detection acceleration design based on ZYNQ has been completed from step 1 to step 7.

[0060] Correspondingly, the ZYNQ-based face detection acceleration design method designed in the embodiment of the present invention includes an image acquisition module, a ZYNQ board, and a host computer.

[0061] The image acquisition module uses a CMOS camera to capture facial images and outputs original video stream data in color-restored RGB565 format.

[0062] The ZYNQ board includes a PS end and a PL end, where the PL end includes an input buffer module, a padding module, a convolution module, a pooling module, an upsampling module, and an output buffer module, and each module is connected in sequence. The PS end is used for overall task scheduling, reading the model weight parameters in the SD card, opening the DMA channel to complete the data interaction of the PL end, and storing the read data in the DDR3 parameters for the host computer to display.

[0063] In order to evaluate the effect of using generated data for low-power face detection, the model selected in this example is YOLOV3-tiny. The neural network parameter training is performed in Pytorch, using the Widerface face dataset. The optimizer selects the adaptive algorithm optimizer, and the learning rate is set to 0.0001. BatchSize is set to 256. After 300 epochs, the final recognition accuracy reaches more than 90%. The target image to be tested is stored in DDR3 through an SD card or an OV5640 camera, and the program is started on the PS side for testing. During the execution of the program, the SDK's time function is used to calculate the running time of the entire program. The experiment selected 100 images of different sizes in the custom dataset for inference testing, and the average inference time for a single image was 205.21ms. The performance parameters of the CPU and GPU platforms that deploy the same network model are compared, as shown in Table 1. The operating frequency and power consumption of the CPU and GPU are all based on the official product data. The inference speed of the face detection acceleration design based on ZYNQ reaches 4.88FPS, which is 4.48 times that of the CPU platform, and the power consumption is only 2.144W, and the power consumption ratio is 12.66, which is 97 times that of the CPU platform and 25 times that of the GPU platform. Table 2 shows the resource report of the Vivado software, in which the lookup table uses 4.86% of the available resources, FF uses 2.90% of the resources, BRAM uses 18.15% of the available resources, and DSP uses 14.65% of the available resources. It can be seen from the experiment that the accuracy of face detection under multiple angles can be seen that the detection frame of the system accelerated by the Zynq platform can be updated and effectively tracked according to the movement of the face. In the hardware design, the use of DSP resources is reduced. By analyzing the proportion of occupied resources, it can be seen that only a small amount of chip resources are used in the present invention, and the main resources consumed are multiplier resources and internal memory resources.

[0064] Table 1

[0065]

[0066] Table 2

[0067]

[0068] On the other hand, the present invention further discloses a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the processor executes the steps of the above method.

[0069] On the other hand, the present invention further discloses a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the above method.

[0070] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the ZYNQ-based face detection acceleration design methods in the above embodiments.

[0071] It is understandable that the system, device and storage medium provided in the embodiments of the present invention correspond to the method provided in the embodiments of the present invention, and the explanation, examples and beneficial effects of the relevant contents can refer to the corresponding parts in the above methods.

[0072] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0073] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0074] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0075] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A ZYNQ-based face detection acceleration design method, which is used to process face detection tasks, based on a host computer, an image acquisition module and a ZYNQ board, and is characterized in that: The image acquisition module uses a CMOS camera to capture facial images and outputs original video stream data in color-restored RGB565 format; The ZYNQ board includes a PS end and a PL end, wherein the PL end includes an input buffer module, a padding module, a convolution module, a pooling module, an upsampling module, and an output buffer module, and each module is connected in sequence; The PS side is used for overall task scheduling, reading the model weight parameters in the SD card, opening the DMA channel to complete the data interaction on the PL side, and storing the read data in DDR3 for display on the host computer.

2. The ZYNQ-based face detection acceleration design method according to claim 1, characterized in that: The following steps are also included: Step 1: The PS side obtains raw image data through an external camera or other data source. The image data is processed by the image preprocessing module and adjusted to the 416×416 pixel resolution required by the YOLOv3-tiny network model. Step 2: The PS reads the pre-trained convolutional neural network parameter file in the SD card and stores these parameters to the specified area of ​​DDR3 memory through the AXI-Stream interface; Step 3: To reduce the computational complexity of the convolutional neural network and reduce PL-side resource consumption, the 32-bit floating-point network parameters are quantized into 8-bit fixed-point numbers; Step 4: The input buffer module temporarily stores the input data transmitted from the PS end through the FIFO buffer. The control logic manages the data transmission through the AXI-Stream interface and uses registers to buffer the data to avoid transmission delays. Step 5: The filling module first generates a control signal through a row and column counter to determine the edge position, fills the edge area with zero values, and transmits actual data in the non-edge area; The data selector outputs the padded feature map according to the control signal and writes it into the FIFO buffer to ensure that the filling is completed before entering the convolution calculation. The final padded data is stored in DDR3 in sequence to provide formatted input for subsequent operations. Step 6: The convolution module performs convolution operation on the input feature map through the parallel convolution calculation module at the PL end, and uses the sliding window method for processing; the convolution calculation module uses 8-channel parallel operation, and processes the data of multiple input and output channels simultaneously through the parallel convolution module, realizing efficient data transmission and resource optimization, and maximizing the use of FPGA resources; Step 7: The pooling module is used to reduce the dimension of the feature map output by the convolution and reduce the spatial size of the feature map; Step 8: The upsampling module enlarges the spatial resolution of the feature map in order to restore spatial details in the neural network or increase the size of the image. It is used in the decoder part of the convolutional neural network (CNN). In face detection, it gradually restores the small-sized feature map to a larger size.

3. The ZYNQ-based face detection acceleration design method according to claim 1, characterized in that: In step 1, the PS adds the camera OV5640 configuration module through the Vivado project, connects the DVP interface and clock signal, initializes the camera using the SCCB interface, and converts the camera's DVP interface data into AXI Stream interface data to ensure the stability and integrity of image data transmission.

4. The ZYNQ-based face detection acceleration design method according to claim 1, characterized in that: In step 2, the PS side reads the network parameter file from the SD card through VDMA, transfers it to the DDR3 memory of the PL side through the AXI-DMA interface, and uses the AXI bus to efficiently transfer data to reduce memory bottlenecks and improve transmission efficiency.

5. The ZYNQ-based face detection acceleration design method according to claim 1, characterized in that: In step 3, INT8 quantization is used to convert the parameters in the model from 32-bit floating point numbers to 8-bit integers. The quantization calculation formula is: ,in Represented as a raw floating-point parameter value, It's zero o'clock. is the scaling factor, Conversion factor calculation formula: ,in and Represents the maximum and minimum values ​​of floating-point numbers. and Indicates the maximum and minimum values ​​represented by fixed-point numbers; fixed-point numbers are converted to zero The calculation formula is as follows: .

6. The ZYNQ-based face detection acceleration design method according to claim 1, characterized in that: Step 4 also includes an output buffer module for temporarily storing the calculated output data, caching it through FIFO, and then transmitting it to the PS end or DDR memory through the AXI-Stream interface. The multiplexer is used to control the data output of multiple modules to ensure that there is no conflict between the calculation results.

7. The ZYNQ-based face detection acceleration design method according to claim 1, characterized in that: In step 5, FIFO is used to buffer input data, the row and column counter counts the image rows and columns and generates control signals, and the data selector outputs FIFO data or zero value according to the signal. The image data is stored in DDR from upper left to lower right, and the filling is completed before the data enters FIFO.

8. The ZYNQ-based face detection acceleration design method according to claim 1, characterized in that: In step 6, multiple input channel data of the feature map and multiple output channel data of the convolution kernel are simultaneously convolved to avoid caching of intermediate product results, and the calculation results are directly added and output, which improves the calculation efficiency and data processing capability of the system; In addition, convolution calculations are performed in batches, processing only part of the input or output channel data each time; FIFO is used to cache intermediate results and generate the final convolution output by accumulation, thereby reducing system resource usage and improving computing efficiency.

9. The ZYNQ-based face detection acceleration design method according to claim 1, characterized in that: In step 7, the maximum pooling layer compares and outputs data synchronously with the pipeline structure through register delay. The step size of the pooling operation is 2. Adjacent pixel data are compared in the same clock cycle, and the maximum value is selected and written into RAM to speed up the pooling operation.

10. The ZYNQ-based face detection acceleration design method according to claim 1, characterized in that: In step 8, the upsampling operation realizes data reconstruction through a dual register structure. The REG1 and REG2 registers respectively store different rows of data of the same feature map. Upsampling is realized by alternating delayed output. The odd-numbered rows of data are written into the FIFO, and the even-numbered rows of data are output through the FIFO, realizing lossless expansion of the feature map resolution.

Images