An edge computing platform and system based on FPGA real-time target recognition detection

Abstract

The application discloses an edge computing platform and system based on FPGA real-time target recognition detection, and relates to the field of microelectronic chips.The edge computing platform comprises an interconnected FPGA and DDR, and an ISP module, a pre-processing module, a VDMA module, an inference accelerator, a CPU and a character superposition module are arranged on the FPGA.The inference accelerator is used for deploying a preset convolutional neural network model, reading any frame of second image and weight parameter data about convolution kernels in the convolutional neural network model from the DDR, accelerating the execution of the algorithm of the convolutional neural network model by using a sphygmic array cluster module, and generating an inference result vector output to the DDR;the sphygmic array cluster module is integrated with a Winograd fast convolution algorithm and a multi-channel sphygmic array.Compared with the prior art, the application realizes the accelerated operation of the convolutional neural network model, and thus improves the inference efficiency.

Description

Technical Field

[0001] This invention relates to the field of microelectronic chip technology, and more specifically, to an edge computing platform and system based on FPGA real-time target recognition and detection. Background Technology

[0002] Currently, convolutional neural networks (CNNs) are being used more and more widely, achieving leapfrog development in areas such as image classification, face recognition, autonomous driving, and speech recognition. However, with the development of machine vision and edge computing, the increasing amount of data and convolutional layers required by CNNs, along with high-bandwidth transmission and intensive computation, pose a significant challenge to CPUs (Central Processing Units) that use serial computing. Existing technologies have adopted GPUs (Graphics Processing Units) to meet the real-time requirements of neural network inference computation through batch parallel processing. However, the drawbacks of high energy consumption, high latency, and low efficiency limit the use of GPUs in mobile edge computing applications. They cannot meet the stringent requirements of edge computing for devices in terms of high-speed computing, low latency, low power consumption, and high integration. Based on this, FPGAs (Field-Programmable Gate Arrays), with their advantages of high speed, low latency, and low energy consumption, have become the preferred solution for accelerating neural networks at the mobile edge.

[0003] Existing FPGA-based real-time target recognition and detection technologies primarily address the inference speed issue, typically using metrics such as gigabit operations per second (GOPS) and frames per second (FPS). Key factors influencing these metrics include the inference efficiency of the inference accelerator with its embedded neural network model within the FPGA and the communication bandwidth of the DDR (Double Data Rate Synchronous Dynamic Random Access Memory). The DDR's communication bandwidth is generally determined by the device's manufacturing process. Regarding inference accelerator efficiency, current technologies propose using the Winograd fast convolution algorithm to improve inference efficiency; however, this method offers limited improvement and cannot meet the demands of increasingly large data volumes in edge computing scenarios with limited resources. Summary of the Invention

[0004] To overcome the shortcomings of low inference efficiency in the prior art, this invention provides an edge computing platform and system based on FPGA for real-time target recognition and detection.

[0005] To solve the above-mentioned technical problems, the technical solution of the present invention is as follows:

[0006] In the first aspect, an edge computing platform based on FPGA real-time target recognition and detection includes interconnected FPGA and DDR, wherein the FPGA is equipped with an ISP module, a preprocessing module, a VDMA (Video Direct Memory Access) module, an inference accelerator, a CPU and a character overlay module;

[0007] The ISP module is used to obtain the original real-time image, perform image preprocessing operations to obtain a first image in RGB format, and store the first image in DDR and character overlay module respectively through the VDMA module;

[0008] The preprocessing module is used to read the first image from the DDR, perform image preprocessing operations to obtain a second image of a specified size, and write it into the DDR in parallel according to the primary color channels;

[0009] The inference accelerator is used to deploy a preset convolutional neural network model, and also to read any frame of the second image and the weight parameter data of the convolutional kernel in the convolutional neural network model from the DDR, accelerate the execution of the convolutional neural network model algorithm using the systolic array cluster module, and generate an inference result vector output to the DDR; wherein, the systolic array cluster module integrates the Winograd fast convolution algorithm and a multi-channel systolic array; the inference result vector is the output result obtained by executing the algorithm of the convolutional neural network model on any frame of the second image;

[0010] The CPU is used to make predictions based on the inference result vector and output the target recognition and detection results to the character overlay module; the target recognition and detection results include target type and corresponding target location information.

[0011] The character overlay module is used to overlay the first image with its corresponding target recognition and detection result and then output it.

[0012] In a second aspect, a real-time target recognition and detection system includes the edge computing platform described in the first aspect, and further includes a camera module and a display module; wherein the output end of the camera module is connected to the input end of the ISP module, and the input end of the display module is connected to the output end of the character overlay module.

[0013] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0014] This invention discloses an edge computing platform and system for real-time target recognition and detection based on FPGA. It optimizes the top-level architecture of the FPGA. The platform achieves primary color separation of the first image through a pre-processing module, and integrates the Winograd fast convolution algorithm and a multi-channel systolic array in the systolic array cluster module. Pre-processing of the convolutional neural network model is implemented in hardware, while post-processing is implemented in software, achieving efficient collaborative processing between SoC hardware and software. Compared to existing technologies, this invention accelerates the operation of the convolutional neural network model, thereby improving inference efficiency. Attached Figure Description

[0015] Figure 1 This is a schematic diagram of the edge computing platform described in Example 1;

[0016] Figure 2 This is a schematic diagram of the microstructure of the inference accelerator described in Example 1;

[0017] Figure 3 This is a schematic diagram of the structure of the YOLO v4 Tiny model described in Example 1;

[0018] Figure 4 This is a schematic diagram of the upsampling operation process described in Example 1;

[0019] Figure 5 This is a schematic diagram of the convolution operation transformation process in Example 1;

[0020] Figure 6 This is a schematic diagram of matrix partitioning in Example 1;

[0021] Figure 7 This is a schematic diagram of the data flow of the pulsating array in Example 1;

[0022] Figure 8 This is a schematic diagram of the internal structure of the PE in Example 1.

[0023] Figure 9 This is a schematic diagram of the quantization operation in Example 1;

[0024] Figure 10 This is a schematic diagram of the system described in Example 2. Detailed Implementation

[0025] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.

[0026] The accompanying drawings are for illustrative purposes only and should not be construed as limiting the scope of this patent.

[0027] To better illustrate this embodiment, some parts in the accompanying drawings may be omitted, enlarged, or reduced, and do not represent the actual product dimensions.

[0028] It will be understood by those skilled in the art that certain well-known structures and their descriptions may be omitted in the accompanying drawings.

[0029] To facilitate understanding and implementation of this invention, the relevant concepts are explained as follows:

[0030] The Winograd fast convolution algorithm performs matrix transformations on both the input data matrix and the convolution kernel matrix, then performs a Hadamard product, and finally transforms the result to obtain the convolutional output. The Hadamard product, also known as the element-wise product, is a matrix operation that multiplies corresponding elements of two matrices.

[0031] In the Winograd fast convolution algorithm, the calculation formulas for 1D convolution and 2D convolution with a stride of 1 are as follows:

[0032] For 1D convolution: Y = A T [[Gg]⊙[B T d]];

[0033] For 2D convolution: = A T [GgG] T ]⊙[B T dB]]A;

[0034] In the formula, G represents the convolution kernel transformation matrix, g is the convolution kernel, A and B represent the input transformation matrices, ⊙ represents the Hadamard product, d represents the input data matrix (i.e., feature data), and Y represents the output matrix (i.e., the operation result).

[0035] Its overall implementation process includes:

[0036] First, the input data matrix d is transformed; then the convolution kernel matrix g is transformed.

[0037] Then, the Hadamard product operation is performed;

[0038] Finally, A is used to perform an inverse transformation on the output matrix.

[0039] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0040] Example 1

[0041] This embodiment proposes an edge computing platform based on FPGA real-time target recognition and detection. (See reference...) Figure 1 It includes interconnected FPGA and DDR, wherein the FPGA is equipped with an ISP module, a preprocessing module, a VDMA module, an inference accelerator, a CPU and a character overlay module;

[0042] The ISP module is used to obtain the original real-time image, perform image preprocessing operations to obtain a first image in RGB format, and store the first image in DDR and character overlay module respectively through the VDMA module;

[0043] The preprocessing module is used to read the first image from the DDR, perform image preprocessing operations to obtain a second image of a specified size, and write it into the DDR in parallel according to the primary color channels;

[0044] The inference accelerator is used to deploy a preset convolutional neural network model, and also to read any frame of the second image and the weight parameter data of the convolutional kernel in the convolutional neural network model from the DDR, accelerate the execution of the convolutional neural network model algorithm using the systolic array cluster module, and generate an inference result vector output to the DDR; wherein, the systolic array cluster module integrates the Winograd fast convolution algorithm and a multi-channel systolic array; the inference result vector is the output result obtained by executing the algorithm of the convolutional neural network model on any frame of the second image;

[0045] The CPU is used to make predictions based on the inference result vector and output the target recognition and detection results to the character overlay module; the target recognition and detection results include target type and corresponding target location information.

[0046] The character overlay module is used to overlay the first image with its corresponding target recognition and detection result and then output it.

[0047] It should be noted that this embodiment provides a hardware and software architecture for a system-on-a-chip. The primary color separation of the first image is achieved through a preprocessing module. By integrating the Winograd fast convolution algorithm and a multi-channel systolic array into the systolic array cluster module, the preprocessing of the convolutional neural network model is implemented in hardware and the postprocessing is implemented in software. This optimizes the top architecture of the FPGA chip and achieves efficient collaborative processing between the SoC hardware and software.

[0048] In some examples, the FPGA is implemented using Xilinx's ZYNQ-7000 series XC7Z045.

[0049] In some examples, the CPU predicts an inference result vector with dimensions of 13×13×85×3, and finally outputs the target type and corresponding location information; where 3 represents the number of anchor boxes, and 85 represents the target type [x,y,w,h,conf,class1]. prob class2 prob ...class80 prob ].

[0050] In a preferred embodiment, the image preprocessing operations include serial-to-parallel conversion, color space conversion, gamma sharpening, and / or color format conversion.

[0051] In some examples, the color space conversion transforms the original real-time image from the yCbCr space to the RGB space.

[0052] In some examples, the color format is converted from RGB565 to RGB888, which converts the color format of the original real-time image.

[0053] In a preferred embodiment, the convolutional neural network model includes several hidden layers, which are pooling layers, resize layers, or convolutional layers corresponding to different convolutional modes; wherein, the last hidden layer is a convolutional layer.

[0054] The inference accelerator also includes an accelerator master controller, an input cache module, a weight parameter cache module, a convolution accumulation module, a batch processing module, a pooling processing module, a resize module, and an output cache module. (See attached document.) Figure 2 ;

[0055] The accelerator master controller is configured to receive and respond to the start trigger signal of the CPU, and to perform continuous inference acceleration on any frame of the second image; it is also configured to return an end flag signal to the CPU when the continuous inference acceleration on any frame of the second image ends; and it is also configured to generate control signals according to the algorithm of the hidden layer corresponding to the currently executed convolutional neural network model, so as to control the orderly operation of each module in the inference accelerator and control the data exchange between the inference accelerator and the DDR.

[0056] The input buffer module is used to read slice data of the second image or intermediate result from the DDR through the first DMA read interface, and perform rearrangement and zero-padding operations to obtain feature data and output it to the pulsating array cluster module in a parallel multi-channel manner.

[0057] The weight parameter caching module is used to read slice data of weight parameters from the DDR through the second DMA read interface and output it in a parallel multi-channel manner.

[0058] The pulsating array cluster module is used to acquire slice data of weight parameters and feature data in batches; it is also used to perform Winograd algorithm conversion according to the convolution mode and perform pulsating array multiplication operation based on multiple pulsating arrays, or directly perform pulsating array multiplication operation through multiple pulsating arrays.

[0059] The convolution accumulation module is used to receive the convolution results from the pulsating array cluster module, cache them, and accumulate them in batches. The accumulated results are output to the batch processing module in a parallel multi-channel manner.

[0060] The batch processing module is used to perform bias operations, activation operations, and quantization operations on the accumulated results to obtain quantization results and output them.

[0061] The pooling processing module is used to acquire the quantization results in a parallel multi-channel manner, perform caching and pooling operations to implement the pooling layer algorithm, and output the pooling results.

[0062] The output cache module is used to obtain the quantization result or the pooling result in a parallel multi-channel manner according to the currently executed hidden layer algorithm, rearrange and cache the result, obtain the intermediate result or inference result vector, and output it to the DDR through the first DMA write interface.

[0063] The Resize module is used to obtain intermediate results from the DDR, perform upsampling operations, and return the upsampled intermediate results to the DDR to implement the Resize layer algorithm.

[0064] It should be noted that the inference accelerator only receives the CPU's start trigger signal during operation. After completing the inference operation for one frame of image, it issues an end flag signal, and sequentially cycles to continuously accelerate the inference of real-time images. The implementation of the convolutional neural network model does not require CPU participation, but the data generated during the process is stored in DDR. Those skilled in the art should understand that the data output to DDR after executing the corresponding algorithm of the entire convolutional neural network model for any frame of the second image (i.e., after executing the corresponding algorithm for the last hidden layer) is the inference result vector, while the data output to DDR before executing the corresponding algorithm of the convolutional neural network model (i.e., the corresponding algorithm for any hidden layer before the last hidden layer) is an intermediate result.

[0065] This preferred embodiment further optimizes the microstructure of the inference accelerator. It should be noted that for the same frame of the second image, one continuous inference represents the implementation of one complete convolutional neural network model (i.e., the corresponding algorithm operations of all hidden layers). Depending on the currently implemented hidden layer algorithm, the data input to the output buffer module may be either a quantization result or a pooling result. That is, the pooling processing module can be bypassed, depending on whether pooling operations (i.e., implementing the pooling layer algorithm) are required on the data results obtained from the convolutional layer algorithm (convolution operation).

[0066] In some examples, the convolutional neural network model is a model involving convolutional kernel operations of size 3*3 with a stride of 1;

[0067] In some examples, the convolutional neural network model is a model involving convolutional kernel operations of size 2*2 with a stride of 1;

[0068] In some examples, the convolutional neural network model is a model involving convolutional kernel operations of size 1*1 with a stride of 1;

[0069] In some examples, the convolutional neural network model is the YOLOv4 Tiny model, whose structure is as follows: Figure 3 As shown, it includes 21 convolutional layers, 3 pooling layers, and 1 resize layer;

[0070] During a continuous inference process, after implementing the algorithm of the 11th hidden layer (which is a convolutional layer), a pooling operation is required (i.e., the 12th hidden layer). Correspondingly, at this time, the data (i.e., the quantization result) flows from the batch processing module into the pooling processing module (i.e., the pooling result is obtained) and then into the output buffer module. If no pooling operation is required after implementing the 11th hidden layer (which is a convolutional layer), then the data flows directly from the batch processing module into the output buffer module, bypassing the pooling processing module.

[0071] Those skilled in the art will understand that the weight parameters are parameters of the convolutional neural network model, specifically including convolution kernel parameters and stride. In some examples, the weight parameters employ post-training linear quantization to convert floating-point data into 8-bit fixed-point data stored in DDR, thereby reducing memory usage and improving transmission speed.

[0072] In some examples, the Resize module performs upsampling by padding for intermediate results with a size of 13*13 before upsampling. See [link / reference]. Figure 4 The size of the intermediate result after upsampling is 26*26.

[0073] It should be noted that the number of input / output channels of the convolution accumulation module, batch processing module, and pooling module is adapted to the number of input channels of the systolic array cluster module to maximize the data processing rate. In some examples, the systolic array cluster module uses eight parallel input channels, and the convolution accumulation module, batch processing module, and pooling module all adopt a parallel eight-channel design.

[0074] In some examples, the pooling processing module performs max pooling operations in parallel eight channels, utilizing Block RAM resources on the FPGA to implement data caching.

[0075] In one optional embodiment, the convolutional neural network model may have a first convolutional mode, a second convolutional mode, or a third convolutional mode.

[0076] Wherein, when the convolution mode is the first convolution mode, the convolutional layer uses a convolutional kernel with a size of 3*3 and a stride of 2;

[0077] When the convolution mode is the second convolution mode, the convolutional layer uses a convolutional kernel with a size of 3*3 and a stride of 1;

[0078] When the convolution mode is the third convolution mode, the convolutional layer uses a convolutional kernel with a size of 1*1 and a stride of 1.

[0079] Furthermore, the pulsating array cluster module includes a fast algorithm preprocessing module, a fast algorithm postprocessing module, and at least two pulsating arrays;

[0080] Wherein, when the convolution mode is the second convolution mode:

[0081] According to the block segmentation requirements of the Winograd fast convolution algorithm, the sliced ​​data of the feature data and weight parameters are input into the fast algorithm preprocessing module in batches, and converted into corresponding Winograd domain data respectively, to obtain the first input matrix of the feature data in the Winograd domain and the first weight matrix of the corresponding weight parameters in the Winograd domain; the first input matrix is ​​scaled according to a preset first scaling factor to form a second input matrix; the first weight matrix and the second input matrix are input into the systolic array in a pipeline manner for systolic array multiplication operation to obtain the first output matrix in the Winograd domain; the first output matrix is ​​input into the fast algorithm postprocessing module for accumulation, scaled according to a preset second scaling factor, and converted to the spatial domain to obtain the second output matrix as the convolution result;

[0082] When the convolution mode is either the first convolution mode or the third convolution mode:

[0083] Using a pipeline approach, the sliced ​​data of the feature data and weight parameters are directly input into the systolic array in batches for systolic array multiplication operations, and the third output matrix is ​​obtained as the convolution result.

[0084] It should be noted that when the convolution mode is the second convolution mode, the data stream sequentially passes through the fast algorithm preprocessing module, the systolic array, and the fast algorithm postprocessing module; when the convolution mode is the first or third convolution mode, the data stream bypasses the fast algorithm preprocessing module and the fast algorithm postprocessing module, and is directly processed by the systolic array before being output to the convolution accumulation module. Those skilled in the art should understand that when performing the corresponding operation in the second convolution mode, the fast algorithm preprocessing module mainly implements the transformation of the input feature data matrix d and the convolution kernel matrix g in the Winograd fast convolution algorithm (i.e., generating the first input matrix and the first weight matrix), the systolic array mainly implements the Hadamard product operation, i.e., matrix multiplication, and the fast algorithm postprocessing module mainly implements the inverse transformation of the output matrix (i.e., the inverse transformation of the first output matrix).

[0085] In some examples, the two-dimensional Winograd fast convolution algorithm F(2×2, 3×3) is implemented through the systolic array cluster module, i.e., Y = A. T [GgG] T ]⊙[B T dB]]A;

[0086] Where 3×3 represents the size of the kernel convolution, and 2×2 represents the output data size of the Winograd fast convolution algorithm. The transformation matrices are set as follows:

[0087]

[0088]

[0089]

[0090] In some examples, the 2D Winograd fast convolution algorithm is used. For 4*4 feature data and a 3*3 convolution kernel with a stride of 1, the convolution operation is converted into matrix multiplication, as follows: Figure 5 As shown;

[0091] Simultaneously, the matrix is ​​divided into blocks according to the block requirements to form block matrices K0, K1, K2, K3, W0, W1, and W2, as follows: Figure 6 As shown, the two-dimensional Winograd fast convolution algorithm is transformed into two nested one-dimensional Winograd algorithms, reducing the number of multiplication operations from 36 to 16, which reduces the computational complexity of multiplication by 2.25 times.

[0092] The one-dimensional matrix multiplication after block partitioning can be expressed as:

[0093]

[0094] Where M1 = (K0 - K2)W0;

[0095]

[0096]

[0097] M4 = (K1 - K3)W2;

[0098] Because the Winograd fast convolution algorithm involves division by 2 (such as the operation between M2 and M3), there is a possibility that the data may not be divisible by 2, thus generating truncation errors. These errors ultimately affect the accuracy of the target recognition and detection results by influencing the precision of the inference results. This embodiment introduces a first scaling factor and a second scaling factor to avoid the influence of truncation errors and generate a computational result without truncation errors (i.e., the second output matrix). Those skilled in the art should understand that the first scaling factor and the second scaling factor have a corresponding relationship and can be set by those skilled in the art according to actual conditions.

[0099] In some examples, the first scaling factor is preset to 4, and the second scaling factor is preset to 1 / 4.

[0100] In some examples, the multi-channel pulsation array specifically refers to two pulsation arrays;

[0101] In other examples, the multi-channel pulsation array is specifically four pulsation arrays, which are set by those skilled in the art based on the logic resources, storage resources and computing resources of the FPGA.

[0102] Preferably, each of the pulse arrays includes 4*4 PEs (Processing Elements), and each PE includes 16 parallel 8*8-bit hardware multipliers to enable 16 multiplication operations to be completed in one clk pulse.

[0103] It should be noted that in the pulsating array, data is input in a pipelined manner. The data stream of the second input matrix flows from left to right, and the data stream of the first weight matrix flows from top to bottom. The 16-bit multiplication result is stored in the internal register of the PE. (See reference...) Figure 7 The diagram shows the data flow of the pulsating array. d represents the data flow of the second input matrix, and w represents the data flow of the first weight matrix.

[0104] Optionally, when the convolution mode is the second convolution mode, the first weight matrix and the second input matrix are piped into the systolic array for systolic array multiplication, see [reference]. Figure 8 ,include:

[0105] The fast algorithm preprocessing module inputs the first weight matrix into the four PEs in the first row of the pulsating array, and then flows it from top to bottom into each PE in the column direction;

[0106] The fast algorithm preprocessing module inputs the second input matrix into the four PEs in the first column of the pulsating array, and the matrix flows from left to right into each PE in the row direction.

[0107] Furthermore, the input buffer module includes an input buffer controller, a data preprocessing module, a zero-padding operation module, a data forwarding module, a first data storage device, and a data reading module;

[0108] The input buffer controller is used to respond to the control signals of the accelerator master controller and control the orderly operation of each module in the input buffer module;

[0109] The data preprocessing module reads slice data of the second image or intermediate result from the DDR in a parallel pipeline manner through four first DMA read interfaces, truncates the slices according to the preset slice size requirements, and performs data rearrangement operation according to the input requirements of the pulse array cluster module to obtain feature data and output it.

[0110] The zero-padding module is used to receive feature data, perform zero-padding on the feature data according to the convolution mode, and output the data in a parallel multi-channel manner: when the convolution mode is the first convolution mode or the second convolution mode, the feature data is zero-padding and then output; when the convolution mode is the third convolution mode, the feature data is not zero-padding and is output directly.

[0111] The forwarding module is used to receive the feature data output by the zero-padding operation module and forward it to different storage addresses of the first data memory according to the size of the feature data.

[0112] The first data storage device uses ping-pong buffering to store feature data and outputs it in a parallel multi-channel manner;

[0113] The data reading module is used to output feature data to the pulsating array cluster module.

[0114] It should be noted that the accelerator master controller triggers the first DMA read interface and the second DMA read interface to initiate burst read operations in sequence through the counter, which can avoid conflicts and arbitration between DMAs and improve the bandwidth utilization of DDR.

[0115] In some examples, the first data memory is implemented using Block RAM resources on the FPGA.

[0116] In some examples, the first data storage uses a parallel eight-channel method to output data to the data reading module.

[0117] In an optional embodiment, the weight parameter caching module includes a weight parameter writing module, a weight memory, and a weight parameter reading module;

[0118] The weight parameter writing module is used to respond to the control signal of the accelerator master controller and read the slice data of the weight parameters from the DDR through the second DMA read interface.

[0119] The weight memory is used to store slice data of weight parameters in response to the control signals of the accelerator master controller, using a ping-pong buffer operation.

[0120] The weight parameter reading module is used to respond to the control signal of the accelerator master controller, read the slice data of the weight parameters and output them to the pulse array cluster module in batches in a parallel multi-channel manner.

[0121] In some examples, the weighted memory is implemented using Block RAM resources on the FPGA.

[0122] In a preferred embodiment, the image preprocessing operation includes separating the three primary colors of the first image data, scaling the size of the first image, and / or converting the pixel value data type of the first image, so that the pixel value is converted from an unsigned integer 0-255 to an 8-bit signed Q0 fixed-point number of -127-127;

[0123] The preprocessing module reads the first image from the DDR through a third DMA read interface, and transmits the second image data to the DDR through three third DMA write interfaces corresponding to the three primary color channels R, G, and B.

[0124] It should be noted that those skilled in the art should understand that converting pixel values ​​from unsigned integers (0-255) to 8-bit signed Q0 fixed-point numbers (-127-127), i.e., the decimal point being the 0th bit in the 8-bit data, can reduce memory usage and improve processing speed. Furthermore, reading a frame of the first image through one DMA read interface and caching the second image data (R, G, B primary color channels) into DDR through three DMA write interfaces, using parallel operations of multiple DMA interfaces, can improve the processing speed of the preprocessing module. This preferred embodiment, through a hardware-based preprocessing module, can improve the real-time frame rate (FPS).

[0125] In some examples, for a first image with a size of 1080*720, the preprocessing module scales the first image to 416*416 using nearest neighbor interpolation.

[0126] In an optional embodiment, in the batch processing module, the bias addition operation uses the following expression:

[0127] y = ∑(x i ×w i +)

[0128] In the formula, xi is the input activation; y is the output activation; wi is the weight; and b represents the preset bias parameter.

[0129] The activation operation uses ReLU as the activation function.

[0130] The quantization operation specifically involves linearly quantizing the data to 8 bits via a right-shift operation. (See [link to relevant documentation]). Figure 9 .

[0131] It should be noted that this embodiment reduces memory usage and speeds up data transmission and processing by quantizing the data to 8 bits.

[0132] In one optional embodiment, the output buffer module includes an output buffer controller, a data write module, a second data storage device, and a data read module;

[0133] The output buffer controller is used to respond to the control signal of the accelerator master controller and control the orderly operation of each module in the output buffer module.

[0134] The data writing module is used to receive the quantization result or the pooling result, rearrange them to obtain an intermediate result or inference result vector, and write it into the second data memory in a parallel multi-channel manner.

[0135] The second data storage is used to store intermediate results or inference result vectors using a ping-pong caching operation;

[0136] The data read module is used to read intermediate results or inference result vectors from the second data memory and output them to the DDR through the first DMA write interface.

[0137] In some examples, the second data memory is implemented using Block RAM resources on the FPGA.

[0138] Example 2

[0139] This embodiment proposes a real-time target recognition and detection system, see reference. Figure 10 The system includes the edge computing platform described in Embodiment 1, and further includes a camera module and a display module; wherein the output end of the camera module is connected to the input end of the ISP module, and the input end of the display module is connected to the output end of the character overlay module.

[0140] In some examples, the camera module includes a visible light camera and / or an infrared camera.

[0141] It is understood that the options in Embodiment 1 above also apply to this embodiment, so they will not be described again here.

[0142] Example 3

[0143] This embodiment proposes a robot, including a robot body and the edge computing platform described in Embodiment 1.

[0144] In some examples, the robot is in the form of an unmanned aerial vehicle.

[0145] It is understood that the options in Embodiment 1 above also apply to this embodiment, so they will not be described again here.

[0146] The same or similar labels correspond to the same or similar parts;

[0147] The terms used to describe positional relationships in the accompanying drawings are for illustrative purposes only and should not be construed as limiting this patent.

[0148] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. For those skilled in the art, other variations or modifications can be made based on the above description. The functional modules or units can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An edge computing platform based on FPGA for real-time target recognition and detection, characterized in that, It includes interconnected FPGA and DDR, wherein the FPGA is equipped with an ISP module, a preprocessing module, a VDMA module, an inference accelerator, a CPU, and a character overlay module; The ISP module is used to obtain the original real-time image, perform image preprocessing operations to obtain the first image in RGB format, and store the first image in DDR and the character overlay module respectively through the VDMA module. The preprocessing module is used to read the first image from the DDR, perform image preprocessing operations to obtain a second image of a specified size, and write it into the DDR in parallel according to the primary color channels; The inference accelerator is used to deploy a preset convolutional neural network model, and also to read any frame of the second image and the weight parameter data of the convolutional kernel in the convolutional neural network model from the DDR, accelerate the execution of the convolutional neural network model algorithm using the systolic array cluster module, and generate an inference result vector output to the DDR; wherein, the systolic array cluster module integrates the Winograd fast convolution algorithm and a multi-channel systolic array; the inference result vector is the output result obtained by executing the algorithm of the convolutional neural network model on any frame of the second image; The CPU is used to make predictions based on the inference result vector and output the target recognition and detection results to the character overlay module; the target recognition and detection results include target type and corresponding target location information. The character overlay module is used to overlay the first image with its corresponding target recognition and detection result and then output it.

2. The edge computing platform based on FPGA real-time target recognition and detection according to claim 1, characterized in that, The convolutional neural network model includes several hidden layers, which are pooling layers, resize layers, or convolutional layers corresponding to different convolution modes; wherein, the last hidden layer is a convolutional layer. The inference accelerator also includes an accelerator master controller, an input cache module, a weight parameter cache module, a convolution accumulation module, a batch processing module, a pooling processing module, a resize module, and an output cache module; The accelerator master controller is configured to receive and respond to the start trigger signal of the CPU, and to perform continuous inference acceleration on any frame of the second image; it is also configured to return an end flag signal to the CPU when the continuous inference acceleration on any frame of the second image ends; and it is also configured to generate control signals according to the algorithm of the hidden layer corresponding to the currently executed convolutional neural network model, so as to control the orderly operation of each module in the inference accelerator and control the data exchange between the inference accelerator and the DDR. The input buffer module is used to read slice data of the second image or intermediate result from the DDR through the first DMA read interface, and perform rearrangement and zero-padding operations to obtain feature data and output it to the pulsating array cluster module in a parallel multi-channel manner. The weight parameter caching module is used to read slice data of weight parameters from the DDR through the second DMA read interface and output it in a parallel multi-channel manner. The pulsating array cluster module is used to acquire slice data of weight parameters and feature data in batches; it is also used to perform Winograd algorithm conversion according to the convolution mode and perform pulsating array multiplication operation based on multiple pulsating arrays, or directly perform pulsating array multiplication operation through multiple pulsating arrays. The convolution accumulation module is used to receive the convolution results from the pulsating array cluster module, cache them, and accumulate them in batches. The accumulated results are output to the batch processing module in a parallel multi-channel manner. The batch processing module is used to perform bias operations, activation operations, and quantization operations on the accumulated results to obtain quantization results and output them. The pooling processing module is used to acquire the quantization results in a parallel multi-channel manner, perform caching and pooling operations to implement the pooling layer algorithm, and output the pooling results. The output cache module is used to obtain the quantization result or the pooling result in a parallel multi-channel manner according to the currently executed hidden layer algorithm, rearrange and cache them to obtain intermediate results or inference result vectors, and output them to the DDR through the first DMA write interface; wherein, the output obtained by executing the last hidden layer algorithm is the inference result vector, and the output obtained by executing the other hidden layer algorithms are all intermediate results; The Resize module is used to obtain intermediate results from the DDR, perform upsampling operations, and return the upsampled intermediate results to the DDR to implement the Resize layer algorithm.

3. The edge computing platform based on FPGA real-time target recognition and detection according to claim 2, characterized in that, In the convolutional neural network model, the convolution mode is a first convolution mode, a second convolution mode, or a third convolution mode; Wherein, when the convolution mode is the first convolution mode, the convolutional layer uses a convolutional kernel with a size of 3*3 and a stride of 2; When the convolution mode is the second convolution mode, the convolutional layer uses a convolutional kernel with a size of 3*3 and a stride of 1; When the convolution mode is the third convolution mode, the convolutional layer uses a convolutional kernel with a size of 1*1 and a stride of 1.

4. The edge computing platform based on FPGA real-time target recognition and detection according to claim 3, characterized in that, The pulsating array cluster module includes a fast algorithm preprocessing module, a fast algorithm postprocessing module, and at least two pulsating arrays; Wherein, when the convolution mode is the second convolution mode: According to the block segmentation requirements of the Winograd fast convolution algorithm, the sliced ​​data of the feature data and weight parameters are input into the fast algorithm preprocessing module in batches, and converted into corresponding Winograd domain data respectively, to obtain the first input matrix of the feature data in the Winograd domain and the first weight matrix of the corresponding weight parameters in the Winograd domain; the first input matrix is ​​scaled according to a preset first scaling factor to form a second input matrix; the first weight matrix and the second input matrix are input into the systolic array in a pipeline manner for systolic array multiplication operation to obtain the first output matrix in the Winograd domain; the first output matrix is ​​input into the fast algorithm postprocessing module for accumulation, scaled according to a preset second scaling factor, and converted to the spatial domain to obtain the second output matrix as the convolution result; When the convolution mode is either the first convolution mode or the third convolution mode: Using a pipeline approach, the sliced ​​data of the feature data and weight parameters are directly input into the systolic array in batches for systolic array multiplication operations, and the third output matrix is ​​obtained as the convolution result.

5. The edge computing platform based on FPGA real-time target recognition and detection according to claim 4, characterized in that, Each of the pulse arrays includes 4*4 PEs, and each PE includes 16 parallel 8*8-bit hardware multipliers to enable 16 multiplication operations to be performed with one clk pulse.

6. The edge computing platform based on FPGA real-time target recognition and detection according to claim 5, characterized in that, When the convolution mode is the second convolution mode, the step of pipelinedly inputting the first weight matrix and the second input matrix into the systolic array for systolic array multiplication includes: The fast algorithm preprocessing module inputs the first weight matrix into the four PEs in the first row of the pulsating array, and then flows it from top to bottom into each PE in the column direction; The fast algorithm preprocessing module inputs the second input matrix into the four PEs in the first column of the pulsating array, and the matrix flows from left to right into each PE in the row direction.

7. The edge computing platform based on FPGA real-time target recognition and detection according to claim 3, characterized in that, The input buffer module includes an input buffer controller, a data preprocessing module, a zero-padding operation module, a data forwarding module, a first data storage device, and a data reading module; The input buffer controller is used to respond to the control signals of the accelerator master controller and control the orderly operation of each module in the input buffer module; The data preprocessing module reads slice data of the second image or intermediate result from the DDR in a parallel pipeline manner through four first DMA read interfaces, truncates the slices according to the preset slice size requirements, and performs data rearrangement operation according to the input requirements of the pulse array cluster module to obtain feature data and output it. The zero-padding module is used to receive feature data, perform zero-padding on the feature data according to the convolution mode, and output the data in a parallel multi-channel manner: when the convolution mode is the first convolution mode or the second convolution mode, the feature data is zero-padding and then output; when the convolution mode is the third convolution mode, the feature data is not zero-padding and is output directly. The forwarding module is used to receive the feature data output by the zero-padding operation module and forward it to different storage addresses of the first data memory according to the size of the feature data. The first data storage device uses ping-pong buffering to store feature data and outputs it in a parallel multi-channel manner; The data reading module is used to output feature data to the pulsating array cluster module.

8. The edge computing platform based on FPGA real-time target recognition and detection according to claim 2, characterized in that, The weight parameter caching module includes a weight parameter writing module, a weight memory, and a weight parameter reading module; The weight parameter writing module is used to respond to the control signal of the accelerator master controller and read the slice data of the weight parameters from the DDR through the second DMA read interface. The weight memory is used to store slice data of weight parameters in response to the control signals of the accelerator master controller, using a ping-pong buffer operation. The weight parameter reading module is used to respond to the control signal of the accelerator master controller, read the slice data of the weight parameters and output them to the pulse array cluster module in batches in a parallel multi-channel manner.

9. An edge computing platform based on FPGA real-time target recognition and detection according to any one of claims 1-8, characterized in that, The image preprocessing operation includes separating the three primary colors of the first image, scaling the size of the first image, and / or converting the pixel value data type of the first image, so that the pixel value is converted from an unsigned integer 0-255 to an 8-bit signed Q0 fixed-point number of -127-127; The preprocessing module reads the first image from the DDR through a third DMA read interface, and transmits the second image to the DDR through three third DMA write interfaces corresponding to the three primary color channels R, G, and B.

10. A real-time target recognition and detection system, characterized in that, The edge computing platform includes any one of claims 1-9, and further includes a camera module and a display module; wherein the output end of the camera module is connected to the input end of the ISP module, and the input end of the display module is connected to the output end of the character overlay module.

Images