Real-time target detection method and system

Abstract

The present application relates to a kind of real-time target detection method, system belongs to image recognition technical field, solve the problem of slow target detection network detection speed.The method comprises: receiving image to be detected, after pre-processing, obtain the preset format of feature map to be detected, store feature map to be detected in PS end DDR;The preset weight stored in advance in PS end DDR is loaded into PL end DDR;According to the available DSP quantity of PL end, the convolution kernel quantity of multiple convolution layers included in target detection network and the input feature map of multiple convolution layers determine convolution layer PE array size;Multiple convolution layers in target detection network are convoluted using convolution layer PE array, and the target detection result after target detection network operation is completed is obtained;According to target detection result, the position range and category of each target in image to be detected are determined, and displayed on visual device for user to view.The speed of image detected by target detection network is improved.

Description

Technical Field

[0001] This invention relates to the field of image recognition technology, and in particular to a real-time target detection method and system. Background Technology

[0002] Automatic driving system refers to a train operation system in which the work performed by the train driver is fully automated and highly centrally controlled. Automatic driving system has functions such as automatic train wake-up and start-up and hibernation, automatic entry and exit from the depot, automatic cleaning, automatic driving, automatic stopping, automatic door opening and closing, and automatic fault recovery. It also has multiple operating modes such as normal operation, degraded operation, and operation interruption.

[0003] The primary function of autonomous driving systems is to accurately identify objects around the vehicle in real time to ensure safe and correct control decisions. Object detection networks based on deep learning-based object detection and semantic segmentation algorithms have become crucial in the field of autonomous driving, as fast inference speed is key to ensuring the safety of autonomous driving.

[0004] Current object detection networks have slow inference speeds and are difficult to maintain, which hinders the development of autonomous driving. Summary of the Invention

[0005] Based on the above analysis, the embodiments of the present invention aim to provide a real-time target detection method and system to solve the problems of slow detection speed and difficulty in maintenance of target detection networks in the prior art.

[0006] On one hand, embodiments of the present invention provide a real-time target detection method, the method comprising:

[0007] Receive the image to be detected, and after preprocessing, obtain the feature map to be detected in a preset format. Store the feature map to be detected in the DDR of the PS terminal.

[0008] Load the preset weights pre-stored in the PS-side DDR into the PL-side DDR. The preset weights are the weight data pre-trained based on the object detection network.

[0009] The size of the convolutional layer PE array is determined based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers included in the object detection network, and the input feature maps of the multiple convolutional layers; the size of the convolutional layer PE array is C*K*L, where C is the execution depth, K is the number of execution convolutional kernels, and L is the parallelism of the execution columns;

[0010] The convolutional layer PE array is used to perform convolution operations on multiple convolutional layers in the target detection network to obtain the target detection result after the target detection network is completed; during the convolution operation, the feature map to be detected and the preset weights are scheduled to the PL end to participate in the operation.

[0011] The location range and category of each target in the image to be detected are determined based on the target detection results and displayed on a visualization device for users to view.

[0012] A further improvement to the above method, the step of determining the size of the convolutional layer PE array based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers of the target detection network, and the input feature maps of the multiple convolutional layers, includes:

[0013] The parallelism of multiple execution depths and multiple execution columns is determined based on the input feature map size of the multiple convolutional layers included in the object detection network.

[0014] The number of execution convolution kernels is determined based on the number of convolution kernels in the multiple convolutional layers included in the object detection network;

[0015] The size of the convolutional layer PE array is determined based on the number of available DSPs at the PL end, the multiple execution depths, the parallelism of multiple execution columns, and the number of multiple execution convolution kernels.

[0016] A further improvement to the above method, the step of determining the size of the convolutional layer PE array based on the number of available DSPs at the PL end, multiple execution depths, the parallelism of multiple execution columns, and the number of multiple execution convolutional kernels, includes:

[0017] From multiple execution depths, multiple execution column parallelisms, and multiple execution convolution kernel numbers, select one execution depth, one execution column parallelism, and one execution convolution kernel number, respectively, such that the selected execution depth, execution column parallelism, execution convolution kernel number, and available DSP number satisfy the following conditions:

[0018] C*K*L≤2*N;

[0019] Where C*K*L represents the size of the convolutional layer PE array, N represents the number of available DSPs at the PL end, C represents the execution depth, K represents the number of execution convolutional kernels, and L represents the parallelism of the execution columns.

[0020] Based on a further improvement to the above method, the step of using a convolutional layer PE array to perform convolution operations on multiple convolutional layers in the target detection network includes:

[0021] For each of the plurality of convolutional layers, the input feature map H1*W1*P1 of the current convolutional layer is divided into columns according to the parallelism L of the execution columns and the execution stride S of the convolutional kernel of the current convolutional layer. There are 1 region, and the size of the input feature map in each region is H1*(L*S)*P1;

[0022] Perform the following steps on each region sequentially, from the first column to the last column of the input feature map of the current convolutional layer:

[0023] Based on the row X of the current convolutional kernel and the stride S of the current convolutional kernel, the input feature map H1*(L*S)*P1 of the current region is divided into... Each sub-region contains an input feature map of size X*(L*S)*P1; the kernel size of the current convolutional layer is X*Y*P1, where X is the row, Y is the column, and P1 is the depth.

[0024] Convolution operations are performed on each sub-region sequentially from the first row to the last row of the input feature map of the current convolutional layer.

[0025] Based on a further improvement to the above method, the step of performing convolution operations on each sub-region sequentially from the first row to the last row of the input feature map of the current convolutional layer includes:

[0026] Based on the execution depth C of the convolutional layer PE array, the current sub-region X*(L*S)*P1 is divided into P1 / C spaces, and the input feature map size of each space is X*(L*S)*C.

[0027] Based on the input feature map of the current convolutional layer, each space is processed sequentially from the depth of the first layer to the depth of the last layer as follows:

[0028] The number of kernel executions M / K is determined based on the number of kernels M in the current convolutional layer and the number of kernels K in the PE array of the convolutional layer.

[0029] Each time, the K convolutional kernels of the current convolutional layer and their corresponding preset weights are scheduled to perform convolution operations on the input feature map X*(L*S)*C included in the current space. The kernels are executed M / K times until all M convolutional kernels of the current convolutional layer have been executed.

[0030] Based on a further improvement to the above method, the step of scheduling the K convolutional kernels of the current convolutional layer and their corresponding preset weights to perform convolution operations on the input feature map X*(L*S)*C in the current space each time includes:

[0031] Perform X*Y cycles of convolution operations on the corresponding input feature map in the current space in the direction from the first row to the Xth row and from the first column to the Yth column of the convolution kernel of the current convolutional layer.

[0032] Within each cycle, based on the execution stride of the convolutional kernel, the weight data of the preset weights of one row, one column, and depth C from the K convolutional kernels are selected, multiplied by the corresponding current space including the input feature map data of one row, L columns, and depth C, and then summed. Each convolutional kernel obtains L intermediate results.

[0033] Based on the further improvement of the above method, a first image cache space and a second image cache space are preset in the DDR of the PS end; the first image cache space is used to store multiple feature maps to be detected; each feature map to be detected is quantized by 8 bits and reordered in the depth, row and column directions, and each reordered feature map to be detected is mapped and stored in the second image cache space.

[0034] Based on the above method, the preset weights are obtained through training using the following steps:

[0035] The object detection network was trained using the publicly available KITTI dataset, and the weight data after training was obtained.

[0036] After training, the weight data is quantized to 8 bits, and the quantized weight data is stored in the SD card in the order of depth, row and column.

[0037] The weight data in the SD card is loaded and saved to the PS-side DDR as the preset weight.

[0038] On the other hand, embodiments of the present invention provide a real-time target detection system, the system comprising:

[0039] The image data receiving module is used to receive the image to be detected, obtain the feature map to be detected in a preset format after preprocessing, and store the feature map to be detected in the PS terminal DDR.

[0040] The weight data loading module is used to load the preset weights pre-stored in the PS-side DDR into the PL-side DDR. The preset weights are weight data pre-trained based on the object detection network.

[0041] The PE array determination module is used to determine the size of the convolutional layer PE array based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers included in the object detection network, and the input feature maps of the multiple convolutional layers; the size of the convolutional layer PE array is C*K*L, where C is the execution depth, K is the number of execution convolutional kernels, and L is the parallelism of the execution columns;

[0042] The PE array calculation module is used to perform convolution operations on multiple convolutional layers in the target detection network using the convolutional layer PE array to obtain the target detection result after the target detection network has completed the operation; during the convolution operation, the feature map to be detected and the preset weights are scheduled to the PL end to participate in the operation.

[0043] The detection result display module is used to determine the location range and category of each target in the image to be detected based on the target detection results, and display it on the visualization device for users to view.

[0044] Based on further improvements to the above system, the PE array determination module includes:

[0045] The parallelism determination module for depth columns is used to determine the parallelism of multiple execution depths and multiple execution columns based on the size of the input feature maps of the multiple convolutional layers included in the object detection network.

[0046] The kernel number determination module is used to determine the number of multiple execution kernels based on the number of kernels in the multiple convolutional layers included in the object detection network;

[0047] The PE array determination submodule is used to determine the size of the convolutional layer PE array based on the number of available DSPs at the PL end, multiple execution depths, the parallelism of multiple execution columns, and the number of multiple execution convolution kernels.

[0048] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0049] 1. Improved the detection speed of existing target detection networks, enabling them to identify images to be detected more quickly.

[0050] 2. Makes the maintenance of the target detection network more convenient, more portable, and facilitates the iterative upgrade of the target detection algorithm.

[0051] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from the description and drawings, which are particularly pointed out. Attached Figure Description

[0052] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0053] Figure 1 A schematic flowchart of a real-time target detection method provided for an embodiment of the present invention;

[0054] Figure 2 A schematic diagram of a target detection network structure based on the YOLO V3 tiny algorithm is provided for an embodiment of the present invention;

[0055] Figure 3 A schematic diagram of convolution operation of a PE array of convolutional layers is provided for an embodiment of the present invention;

[0056] Figure 4 This is a schematic diagram of the structure of a real-time target detection system provided as an embodiment of the present invention. Detailed Implementation

[0057] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0058] A specific embodiment of the present invention discloses a real-time target detection method, such as... Figure 1 As shown.

[0059] Step S101: Receive the image to be detected, and after preprocessing, obtain the feature map to be detected in a preset format. Store the feature map to be detected in the PS terminal DDR.

[0060] Specifically, the image to be detected can be received through an image acquisition device, such as a camera, camcorder, scanner, or other device with a camera function (mobile phone, tablet, etc.). Since the images to be detected acquired through different image acquisition devices may not meet the input format requirements of the object detection network, preprocessing is required. Preprocessing methods can include demosaicing, gamma correction, or video scaling. After preprocessing, the image to be detected will meet the input format requirements of the object detection network.

[0061] It is worth noting that, such as Figure 2 As shown, a target detection network based on the YOLO V3 tiny algorithm is presented. This network comprises 23 layers, where `layer` represents the layer level, `filters` represents the number of convolutional kernels, `size / strd` represents the kernel size / stride, `input` represents the input feature map, and `output` represents the output feature map. The input image must meet the preset format of 416*416*3, i.e., 416 rows, 416 columns, and 3 depths of feature map data.

[0062] The preset format can be the input format requirement for the object detection network. The feature map corresponding to the preprocessed image to be detected can be saved to the Double Data Rate (DDR) synchronous dynamic random access memory of the processing system (PS).

[0063] Preferably, a first image cache space and a second image cache space are preset in the DDR of the PS end; the first image cache space is used to store multiple feature maps to be detected; each feature map to be detected is quantized by 8 bits and reordered in the depth, row and column directions, and each reordered feature map to be detected is mapped and stored in the second image cache space.

[0064] For example, after preprocessing, the image to be detected becomes 416*416*3 RGB format data, with each data byte being 8 bits. This data forms the AXI-Stream video stream data and is written to DDR. In DDR, each address stores one byte of data. Addresses 0x01000000, 0x01500000, and 0x02000000 are the starting addresses of the DDR addresses used for three-frame buffering, and 0x01000000, 0x01500000, and 0x02000000 are the first image buffer space. An image contains 416*416*3 = 519168 bytes of data. Storing an image into address 0x01000000 means storing the first byte of the image into 0x01000000, the second byte into 0x01000001, and so on, until all 519168 bytes are stored.

[0065] Three-frame buffering means that the image is stored in three different DDR addresses. In this embodiment of the invention, the first captured image frame is stored in DDR with the starting address 0x01000000, the second frame is stored in 0x01500000, and the third frame is stored in 0x02000000. Three-frame buffering is a reasonable design because single-frame buffering has a drawback: when the data source is continuously input, the frame buffer may store the result of two or more frames of image data superimposed, thus requiring three-frame buffering.

[0066] The acquired image data is saved to the DDR in the order of depth->width->height. That is, the first row and first column of RGB data is saved first, then the first row and second column of RGB data are saved, and so on, until the first row and 416th column are saved. Then the second row and first column are saved, and so on. In this embodiment of the invention, when the convolutional layer PE array performs convolution operations, it needs to retrieve data row by row; that is, the image needs to be reordered to the order of depth->height->width before being stored in the DDR.

[0067] The three captured images are reordered and placed into the DDR of the PS end with the starting address as 0x02500000, 0x03000000, and 0x03500000. 0x02500000, 0x03000000, and 0x03500000 are the second image buffer space.

[0068] Step S102: Load the preset weights pre-stored in the PS-side DDR into the PL-side DDR. The preset weights are the weight data pre-trained based on the object detection network.

[0069] Specifically, the pre-trained weight data of the object detection network is pre-stored in the DDR of the PS terminal. After obtaining the feature map to be detected in a preset format, the preset weights are loaded into the Programmable Logic (PL) terminal.

[0070] Preferably, the preset weights can be obtained through training using the following steps:

[0071] The object detection network was trained using the publicly available KITTI dataset, and the weight data after training was obtained.

[0072] After training, the weight data is quantized to 8 bits, and the quantized weight data is stored in the SD card in the order of depth, row and column.

[0073] The weight data in the SD card is loaded and saved to the PS-side DDR as the preset weight.

[0074] Understandably, using the publicly available KITTI dataset to train the YOLO V3 tiny algorithm, and separating the trained weight and bias data into two binary files, allows for the fusion of CONV and BN layers on the network weights. The fused new convolutional layer is then quantized using 8-bit fixed-point quantization. The quantized weight and bias data are rearranged according to memory access order and saved as a bin file to an SD card. The Vitis development software reads the weight data into address 0x06000000 in the DDR of the Processor System (PS). All weight data in the PS DDR is then loaded into the Programmable Logic (PL) DDR (starting address 0x00000000), and remapped to the AXI bus address (AXI address signal -0x06000000) when reading the weight data.

[0075] Step S103: Determine the size of the convolutional layer PE array based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers included in the object detection network, and the input feature maps of the multiple convolutional layers; the size of the convolutional layer PE array is C*K*L, where C is the execution depth, K is the number of execution convolutional kernels, and L is the parallelism of the execution columns.

[0076] Specifically, the logic resources at the PL end can be used to complete the inference process of the object detection network. This requires determining the number of available DSPs at the PL end; to increase the inference speed of the object detection network, the number of available DSPs at the PL end should be utilized as much as possible.

[0077] Specifically, by Figure 2It can be seen that the object detection network includes 13 convolutional layers (conv), namely layers 0, 2, 4, 6, 8, 10, 12, 13, 14, 15, 18, 21 and 22. The number of convolutional kernels in each convolutional layer is 16, 32, 64...255 respectively. The input feature map formats of each convolutional layer are 416*416*3, 208*208*16, 104*104*32...26*26*256 respectively.

[0078] The size of the convolutional layer PE array is determined based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers included in the object detection network, and the input feature maps of the multiple convolutional layers. The size of the convolutional layer PE array is C*K*L, where C is the execution depth, K is the number of execution convolutional kernels, and L is the parallelism of the execution columns.

[0079] Preferably, determining the size of the convolutional layer PE array based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers of the target detection network, and the input feature maps of the multiple convolutional layers includes:

[0080] The parallelism of multiple execution depths and multiple execution columns is determined based on the input feature map size of the multiple convolutional layers included in the object detection network.

[0081] The number of execution convolution kernels is determined based on the number of convolution kernels in the multiple convolutional layers included in the object detection network;

[0082] The size of the convolutional layer PE array is determined based on the number of available DSPs at the PL end, the multiple execution depths, the parallelism of multiple execution columns, and the number of multiple execution convolution kernels.

[0083] It is understandable that, such as Figure 2 As shown, layers 2, 4, 6, and 8 of the target detection network are selected for convolutional operations on the PE array of convolutional layers, and the number of available DSPs at the PL end is 1000.

[0084] The number of convolutional kernels in the 2-layer, 4-layer, 6-layer, and 8-layer object detection network are 32, 64, 128, and 256, respectively. To facilitate the calculation of convolutional kernels, multiple execution convolutional kernels of 32, 16, 8, 4, and 2 can be calculated at the same time. The number of each execution convolutional kernel can be the common divisor of the smallest number of convolutional kernels in the selected convolutional layers.

[0085] Based on the 208 columns in the 208*208*16 input feature map of layer 2, the 104 columns in the 104*104*32 input feature map of layer 4, the 52 columns in the 52*52*64 input feature map of layer 6, and the 26 columns in the 26*26*128 input feature map of layer 8, the parallelism of multiple execution columns of 13 columns and 26 columns can be selected. The parallelism of the selected multiple execution columns is the common divisor of the columns in the input feature map with the smallest number of columns.

[0086] Based on the depth of 16 in the 2-layer input feature map 208*208*16, the depth of 32 in the 4-layer input feature map 104*104*32, the depth of 64 in the 6-layer input feature map 52*52*64, and the depth of 128 in the 8-layer input feature map 26*26*128, multiple execution depths such as 32, 16, and 8 can be selected. The selected multiple execution depths are the common divisors of the depths in the feature maps with the smallest depth in each input feature map.

[0087] Preferably, determining the size of the convolutional layer PE array based on the number of available DSPs at the PL end, multiple execution depths, the parallelism of multiple execution columns, and the number of multiple execution convolutional kernels includes:

[0088] From multiple execution depths, multiple execution column parallelisms, and multiple execution convolution kernel numbers, select one execution depth, one execution column parallelism, and one execution convolution kernel number, respectively, such that the selected execution depth, execution column parallelism, execution convolution kernel number, and available DSP number satisfy the following conditions:

[0089] C*K*L≤2*N;

[0090] Where C*K*L represents the size of the convolutional layer PE array, N represents the number of available DSPs at the PL end, C represents the execution depth, K represents the number of execution convolutional kernels, and L represents the parallelism of the execution columns.

[0091] It is understandable that, since convolution operations utilize the logic resources of the PD end to complete the convolution operation, the selected execution depth, the parallelism of the execution columns, the number of execution convolution kernels, and the number of available DSPs must satisfy the following condition: C*K*L≤2*N.

[0092] Specifically, in order to better utilize the number of available DSPs at the PL end, one execution depth, one parallelism of one execution column, and one number of execution convolution kernels can be selected from multiple execution depths, multiple execution column parallelisms, and multiple execution convolution kernel numbers. Under the premise of satisfying the condition C*K*L≤2*N, the combination that maximizes C*K*L can be selected. At this time, the computing resources of the available DSPs can be fully utilized, the convolution operation of the object detection network can be completed more quickly, and the detection speed of the object detection network can be significantly improved.

[0093] Compared with existing technologies, determining the size of the convolutional layer PE array based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers of the object detection network, and the input feature map can improve the applicability of the object detection network on different hardware.

[0094] Step S104: Use the convolutional layer PE array to perform convolution operations on multiple convolutional layers in the target detection network to obtain the target detection result after the target detection network has completed the operation; during the convolution operation, the feature map to be detected and the preset weights are scheduled to the PL end to participate in the operation.

[0095] Specifically, the convolutional layer PE array is a configuration method for performing convolution operations in object detection networks. When executing with this convolutional layer PE array, each execution requires selecting L columns of input feature maps with a depth of C and selecting K convolution kernels to participate in the convolution operation.

[0096] The convolutional layer PE array is used to perform convolution operations on multiple convolutional layers in the object detection network. For example, one can choose... Figure 2 Layers 2, 4, and 6 of the 12 convolutional layers in the network participate in convolution operations. Other layers in the object detection network can be processed using different methods. After the object detection network completes inference, the completed object detection results are obtained.

[0097] Understandably, when performing inference on an object detection network, it is necessary to schedule the feature map to be detected and the corresponding weight data of the preset weights.

[0098] Step S105: Determine the location range and category of each target in the image to be detected based on the target detection results, and display them on the visualization device for the user to view.

[0099] Specifically, the object detection network can output the location and category of objects in an image. Based on the object detection results, it outlines the objects on the image with bounding boxes and displays the object category information near the bounding boxes. After this processing, the image is sent to a visualization device, where the outlined objects and their categories are displayed for the user to view. It is understandable that in the process of autonomous driving, the vehicle's onboard system can determine the next action based on the object detection results.

[0100] During implementation, a target detection network is pre-selected, and the weight data of this trained network is stored in the DDR at the PS end. The convolutional layers in the target detection network that need to be processed by the PE array provided according to this embodiment of the invention are determined. During actual target detection, the target detection network performs inference in real time based on the received image to be detected, and obtains the target detection result.

[0101] Compared with existing technologies, the real-time target detection method provided by this invention can perform convolution operations on the convolutional layers of existing target detection networks based on the convolutional layer PE array, making full use of the number of available DSPs at the PL end and improving the detection speed of the target detection network. Furthermore, the convolutional layer PE array determined according to the target detection network participates in multiple convolutional layer operations of the target detection network, making the target detection network easier to maintain, more portable, and facilitating iterative upgrades of the target detection algorithm.

[0102] Furthermore, the step of using a convolutional layer PE array to perform convolution operations on multiple convolutional layers in the target detection network includes:

[0103] For each of the plurality of convolutional layers, the input feature map H1*W1*P1 of the current convolutional layer is divided into columns according to the parallelism L of the execution columns and the execution stride S of the convolutional kernel of the current convolutional layer. There are 1 region, and the size of the input feature map in each region is H1*(L*S)*P1;

[0104] Perform the following steps on each region sequentially, from the first column to the last column of the input feature map of the current convolutional layer:

[0105] Based on the row X of the current convolutional kernel and the stride S of the current convolutional kernel, the input feature map H1*(L*S)*P1 of the current region is divided into... Each sub-region contains an input feature map of size X*(L*S)*P1; the kernel size of the current convolutional layer is X*Y*P1, where X is the row, Y is the column, and P1 is the depth.

[0106] Convolution operations are performed on each sub-region sequentially from the first row to the last row of the input feature map of the current convolutional layer.

[0107] Specifically, such as Figure 3 As shown, the convolutional layer PE array C*K*L is 13*16*4, the input feature map of the current convolutional layer is 416*416*8, the output feature map is 416*416*32, the number of convolutional kernels of the current convolutional layer is 32, the kernel size is 3*3*8, and the execution stride of the kernel of the current convolutional layer is 1.

[0108] Based on the parallelism of the execution column (13) and the execution stride of the current convolutional layer kernel (1), the input feature map of the current convolutional layer (416*416*8) is divided into 32 regions, each region containing an input feature map of size 416*13*8.

[0109] For each of the 32 regions, based on the 3 rows of the current convolution kernel and the execution stride of the current convolutional layer's kernel of 1, the input feature map of the current region, which is 416*13*8, is divided into 416 sub-regions, each of which includes an input feature map of size 3*13*8.

[0110] Convolution operations are performed on each sub-region sequentially from the first row to the last row of the input feature map of the current convolutional layer.

[0111] Preferably, the step of performing convolution operations on each sub-region sequentially from the first row to the last row of the input feature map of the current convolutional layer includes:

[0112] Based on the execution depth C of the convolutional layer PE array, the current sub-region X*(L*S)*P1 is divided into P1 / C spaces, and the input feature map size of each space is X*(L*S)*C.

[0113] Based on the input feature map of the current convolutional layer, from the depth of the first layer to the depth of the last layer, each space is processed as follows:

[0114] The number of kernel executions M / K is determined based on the number of kernels M in the current convolutional layer and the number of kernels K in the PE array of the convolutional layer.

[0115] Each time, the K convolutional kernels of the current convolutional layer and their corresponding preset weights are scheduled to perform convolution operations on the input feature map X*(L*S)*C included in the current space. The kernels are executed M / K times until all M convolutional kernels of the current convolutional layer have been executed.

[0116] Specifically, within each sub-region, the input feature map size is 3*13*8. Based on the execution depth of the convolutional layer PE array of 4, the current sub-region 3*13*8 is divided into 2 spaces, each space containing an input feature map size of 3*13*4. These 2 spaces are processed sequentially according to the direction from the first layer depth to the last layer depth of the current convolutional layer's input feature map.

[0117] Since the current convolutional layer has 32 convolutional kernels and the PE array of the convolutional layer has 16 execution kernels, the number of kernel executions is 2. Each time, only 16 convolutional kernels of the current convolutional layer and their corresponding preset weights can be scheduled to perform convolution operations on the current space 3*13*4. Execution is performed 2 times, which can complete the execution of all 32 convolutional kernels of the current convolutional layer.

[0118] Preferably, the step of scheduling the K convolutional kernels of the current convolutional layer and their corresponding preset weights to perform convolution operations on the input feature map X*(L*S)*C in the current space each time includes:

[0119] Perform X*Y cycles of convolution operations on the corresponding input feature map in the current space in the direction from the first row to the Xth row and from the first column to the Yth column of the convolution kernel of the current convolutional layer.

[0120] Within each cycle, based on the execution stride of the convolutional kernel, the weight data of the preset weights of one row, one column, and depth C from the K convolutional kernels are selected, multiplied by the corresponding current space including the input feature map data of one row, L columns, and depth C, and then summed. Each convolutional kernel obtains L intermediate results.

[0121] Specifically, such as Figure 3 As shown, the corresponding input feature map in the current space is subjected to 9 cycles of convolution operations in the direction of the first row to the third row and the first column to the third column of the convolution kernel of the current convolutional layer.

[0122] Within each cycle, the weight data of the preset weights of one row, one column, and 4 depth from the 16 convolutional kernels are multiplied and summed with the feature map data of one row, 13 columns, and 4 depth in the current space of 3*13*4. Each convolutional kernel obtains 13 intermediate results.

[0123] For example:

[0124] In the first cycle, the weight data of the first row, first column, and 4 depth of the current 16 convolutional kernels are multiplied with the feature map data of the first row, 13 columns, and 4 depth of the current space. This process requires 13×16×4 operations. Then, the corresponding values ​​of each convolutional kernel are added together to obtain 13 intermediate results of the 16 convolutional kernels.

[0125] In the second cycle, the weight data of the second row, first column, and 4 depth of the current 16 convolutional kernels are multiplied with the feature map data of the second row, 13 columns, and 4 depth of the current space. This process requires 13×16×4 operations. Then, the corresponding values ​​of each convolutional kernel are added together to obtain 13 intermediate results of the 16 convolutional kernels.

[0126] In the third cycle, the weight data of the third row, first column, and 4 depth of the current 16 convolutional kernels are multiplied with the feature map data of the third row, 13 columns, and 4 depth of the current space. This process requires 13×16×4 operations. Then, the corresponding values ​​of each convolutional kernel are added together to obtain 13 intermediate results of the 16 convolutional kernels.

[0127] In the fourth cycle, the weight data of the first row, second column, and depth 4 of the current 16 convolutional kernels are multiplied with the feature map data of the first row, 13 columns, and depth 4 of the current space. This process requires 13×16×4 operations. Then, the corresponding values ​​of each convolutional kernel are added together to obtain 13 intermediate results of the 16 convolutional kernels.

[0128] In the 5th cycle, the weight data of the second row, second column, and 4 depth of the current 16 convolutional kernels are multiplied with the feature map data of the first row, 13 columns, and 4 depth of the current space. This process requires 13×16×4 operations. Then, the corresponding values ​​of each convolutional kernel are added together to obtain 13 intermediate results of the 16 convolutional kernels.

[0129] In the 6th cycle, the weight data of the third row, second column, and 4 depth of the current 16 convolutional kernels are multiplied with the feature map data of the first row, 13 columns, and 4 depth of the current space. This process requires 13×16×4 operations. Then, the corresponding values ​​of each convolutional kernel are added together to obtain 13 intermediate results of the 16 convolutional kernels.

[0130] In the 7th cycle, the weight data of the first row, third column, and 4 depth of the current 16 convolutional kernels are multiplied with the feature map data of the first row, 13 columns, and 4 depth of the current space. This process requires 13×16×4 operations. Then, the corresponding values ​​of each convolutional kernel are added together to obtain 13 intermediate results of the 16 convolutional kernels.

[0131] In the 8th cycle, the weight data of the second row, third column, and 4 depth of the current 16 convolutional kernels are multiplied with the feature map data of the first row, 13 columns, and 4 depth of the current space. This process requires 13×16×4 operations. Then, the corresponding values ​​of each convolutional kernel are added together to obtain 13 intermediate results of the 16 convolutional kernels.

[0132] In the 9th cycle, the weight data of the third row, third column, and 4 depth of the current 16 convolutional kernels are selected and multiplied with the feature map data of the first row, 13 columns, and 4 depth of the current space. This process requires 13×16×4 operations. Then, the corresponding values ​​of each convolutional kernel are added together to obtain 13 intermediate results of the 16 convolutional kernels.

[0133] It is worth noting that the 13 columns in the first 3 periods are columns 1-13, the 13 columns in the fourth to sixth periods are columns 2-14, and the 13 columns in the seventh to ninth periods are columns 3-15.

[0134] Compared with the prior art, the real-time target detection method provided by the embodiments of the present invention determines the PE array of different sizes of convolutional layers according to the number of available DSPs at the PL end, the number of convolutional kernels of multiple convolutional layers included in the target detection network, and the input feature map, which can improve the applicability of the target detection network on different hardware.

[0135] One specific embodiment of the present invention discloses a real-time target detection system, such as... Figure 4 As shown.

[0136] The image data receiving module 401 is used to receive the image to be detected, obtain the feature map to be detected in a preset format after preprocessing, and store the feature map to be detected in the PS terminal DDR.

[0137] The weight data loading module 402 is used to load the preset weights pre-stored in the PS-end DDR into the PL-end DDR. The preset weights are weight data pre-trained based on the target detection network.

[0138] The PE array determination module 403 is used to determine the size of the convolutional layer PE array based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers included in the object detection network, and the input feature maps of the multiple convolutional layers; the size of the convolutional layer PE array is C*K*L, where C is the execution depth, K is the number of execution convolutional kernels, and L is the parallelism of the execution columns;

[0139] The PE array calculation module 404 is used to perform convolution operations on multiple convolutional layers in the target detection network using the convolutional layer PE array to obtain the target detection result after the target detection network has completed the operation; during the convolution operation, the feature map to be detected and the preset weights are scheduled to the PL end to participate in the operation.

[0140] The detection result display module 405 is used to determine the location range and category of each target in the image to be detected based on the target detection results, and display it on a visualization device for the user to view.

[0141] Furthermore, the PE array determination module 403 includes:

[0142] The parallelism determination module for depth columns is used to determine the parallelism of multiple execution depths and multiple execution columns based on the size of the input feature maps of the multiple convolutional layers included in the object detection network.

[0143] The kernel number determination module is used to determine the number of multiple execution kernels based on the number of kernels in the multiple convolutional layers included in the object detection network;

[0144] The PE array determination submodule is used to determine the size of the convolutional layer PE array based on the number of available DSPs at the PL end, multiple execution depths, the parallelism of multiple execution columns, and the number of multiple execution convolution kernels.

[0145] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0146] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A real-time target detection method, characterized in that, The method includes: Receive the image to be detected, and after preprocessing, obtain the feature map to be detected in a preset format. Store the feature map to be detected in the DDR of the PS terminal. Load the preset weights pre-stored in the PS-side DDR into the PL-side DDR. The preset weights are the weight data pre-trained based on the object detection network. The size of the convolutional layer PE array is determined based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers included in the object detection network, and the input feature maps of the multiple convolutional layers; the size of the convolutional layer PE array is C*K*L, where C is the execution depth, K is the number of execution convolutional kernels, and L is the parallelism of the execution columns; The convolutional layer PE array is used to perform convolution operations on multiple convolutional layers in the target detection network to obtain the target detection result after the target detection network is completed; during the convolution operation, the feature map to be detected and the preset weights are scheduled to the PL end to participate in the operation. The method of using a convolutional layer PE array to perform convolution operations on multiple convolutional layers in the object detection network includes: for each of the multiple convolutional layers, dividing the input feature map H1*W1*P1 of the current convolutional layer into columns according to the parallelism L of the execution columns and the execution stride S of the convolution kernel of the current convolutional layer. W1 / (L*S) There are 3 regions, each containing an input feature map of size H1*(L*S)*P1. Following the direction from the first column to the last column of the current convolutional layer's input feature map, each region is divided into the following steps: Based on the row X of the current convolutional kernel and the execution stride S of the current convolutional kernel, the input feature map H1*(L*S)*P1 of the current region is divided into 3 regions. H1 / S Each sub-region contains an input feature map of size X*(L*S)*P1; the kernel size of the current convolutional layer is X*Y*P1, where X is the row, Y is the column, and P1 is the depth; convolution operations are performed on each sub-region sequentially from the first row to the last row of the input feature map of the current convolutional layer. The location range and category of each target in the image to be detected are determined based on the target detection results and displayed on a visualization device for users to view.

2. The real-time target detection method according to claim 1, characterized in that, The step of determining the size of the convolutional layer PE array based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers of the object detection network, and the input feature maps of the multiple convolutional layers includes: The parallelism of multiple execution depths and multiple execution columns is determined based on the input feature map size of the multiple convolutional layers included in the object detection network. The number of execution convolution kernels is determined based on the number of convolution kernels in the multiple convolutional layers included in the object detection network; The size of the convolutional layer PE array is determined based on the number of available DSPs at the PL end, the multiple execution depths, the parallelism of multiple execution columns, and the number of multiple execution convolution kernels.

3. The real-time target detection method according to claim 2, characterized in that, The determination of the convolutional layer PE array size based on the number of available DSPs at the PL end, multiple execution depths, the parallelism of multiple execution columns, and the number of multiple execution convolutional kernels includes: From multiple execution depths, multiple execution column parallelisms, and multiple execution convolution kernel numbers, select one execution depth, one execution column parallelism, and one execution convolution kernel number, respectively, such that the selected execution depth, execution column parallelism, execution convolution kernel number, and available DSP number satisfy the following conditions: C*K*L≤2*N; Where C*K*L represents the size of the convolutional layer PE array, N represents the number of available DSPs at the PL end, C represents the execution depth, K represents the number of execution convolutional kernels, and L represents the parallelism of the execution columns.

4. The real-time target detection method according to claim 1, characterized in that, The step of performing convolution operations on each sub-region sequentially from the first row to the last row of the input feature map of the current convolutional layer includes: Based on the execution depth C of the convolutional layer PE array, the current sub-region X*(L*S)*P1 is divided into P1 / C spaces, and the input feature map size of each space is X*(L*S)*C. Based on the input feature map of the current convolutional layer, each space is processed sequentially from the depth of the first layer to the depth of the last layer as follows: The number of kernel executions M / K is determined based on the number of kernels M in the current convolutional layer and the number of kernels K in the PE array of the convolutional layer. Each time, the K convolutional kernels of the current convolutional layer and their corresponding preset weights are scheduled to perform convolution operations on the input feature map X*(L*S)*C included in the current space. The kernels are executed M / K times until all M convolutional kernels of the current convolutional layer have been executed.

5. The real-time target detection method according to claim 4, characterized in that, The step of scheduling the K convolutional kernels of the current convolutional layer and their corresponding preset weights to perform convolution operations on the input feature map X*(L*S)*C in the current space includes: Perform X*Y cycles of convolution operations on the corresponding input feature map in the current space in the direction from the first row to the Xth row and from the first column to the Yth column of the convolution kernel of the current convolutional layer. Within each cycle, based on the execution stride S of the convolutional kernel, the weight data of the preset weights of one row, one column, and depth C from K convolutional kernels are selected, multiplied and then added to the corresponding current space including the input feature map data of one row, L columns, and depth C. Each convolutional kernel obtains L intermediate results.

6. The real-time target detection method according to any one of claims 1-5, characterized in that, The PS-side DDR has a first image cache space and a second image cache space. The first image cache space is used to store multiple feature maps to be detected. Each feature map to be detected is quantized by 8 bits and reordered in the depth, row and column directions. The reordered feature map to be detected is then mapped and stored in the second image cache space.

7. The real-time target detection method according to any one of claims 1-5, characterized in that, The preset weights are obtained through training using the following steps: The object detection network was trained using the publicly available KITTI dataset, and the weight data after training was obtained. After training, the weight data is quantized to 8 bits, and the quantized weight data is stored in the SD card in the order of depth, row and column. The weight data in the SD card is loaded and saved to the PS-side DDR as the preset weight.

8. A real-time target detection system, characterized in that, The system includes: The image data receiving module is used to receive the image to be detected, obtain the feature map to be detected in a preset format after preprocessing, and store the feature map to be detected in the PS terminal DDR. The weight data loading module is used to load the preset weights pre-stored in the PS-side DDR into the PL-side DDR. The preset weights are weight data pre-trained based on the object detection network. The PE array determination module is used to determine the size of the convolutional layer PE array based on the number of available DSPs at the PL end, the number of convolutional kernels in the multiple convolutional layers included in the object detection network, and the input feature maps of the multiple convolutional layers; the size of the convolutional layer PE array is C*K*L, where C is the execution depth, K is the number of execution convolutional kernels, and L is the parallelism of the execution columns; The PE array calculation module is used to perform convolution operations on multiple convolutional layers in the target detection network using the convolutional layer PE array to obtain the target detection result after the target detection network has completed the operation; during the convolution operation, the feature map to be detected and the preset weights are scheduled to the PL end to participate in the operation. The method of using a convolutional layer PE array to perform convolution operations on multiple convolutional layers in the object detection network includes: for each of the multiple convolutional layers, dividing the input feature map H1*W1*P1 of the current convolutional layer into columns according to the parallelism L of the execution columns and the execution stride S of the convolution kernel of the current convolutional layer. W1 / (L*S) There are 3 regions, each containing an input feature map of size H1*(L*S)*P1. Following the direction from the first column to the last column of the current convolutional layer's input feature map, each region is divided into the following steps: Based on the row X of the current convolutional kernel and the execution stride S of the current convolutional kernel, the input feature map H1*(L*S)*P1 of the current region is divided into 3 regions. H1 / S Each sub-region contains an input feature map of size X*(L*S)*P1; the kernel size of the current convolutional layer is X*Y*P1, where X is the row, Y is the column, and P1 is the depth; convolution operations are performed on each sub-region sequentially from the first row to the last row of the input feature map of the current convolutional layer. The detection result display module is used to determine the location range and category of each target in the image to be detected based on the target detection results, and display it on the visualization device for users to view.

9. The real-time target detection system according to claim 8, characterized in that, The PE array determination module includes: The parallelism determination module for depth columns is used to determine the parallelism of multiple execution depths and multiple execution columns based on the size of the input feature maps of the multiple convolutional layers included in the object detection network. The kernel number determination module is used to determine the number of multiple execution kernels based on the number of kernels in the multiple convolutional layers included in the object detection network; The PE array determination submodule is used to determine the size of the convolutional layer PE array based on the number of available DSPs at the PL end, multiple execution depths, the parallelism of multiple execution columns, and the number of multiple execution convolution kernels.

Images