[0038] The embodiments of the present invention are described in detail below. This embodiment is implemented on the premise of the technical solution of the present invention, and provides a detailed implementation manner and a specific operation process, but the protection scope of the present invention is not limited to the following implementation. example.
[0039] like figure 1 and figure 2 As shown, this embodiment includes two parts: offline model training and online detection, which are described as follows:
[0040] 1. Offline training of deep learning network model part:
[0041] Using the image semantic segmentation annotation tool labelme, the outlines of all dangerous objects in each image are marked separately as the input of the deep learning network.
[0042] The convolutional neural network used to extract features has a total of 8 layers, including 5 convolutional layers and 3 max-pooling layers. Normalize the input image data, adjust the image size to 224*224, and use 8-layer convolutional neural network for feature extraction for each image.
[0043] An initialized image with a size of 224*224 is passed through a convolution filter with a size of 11*11 and a stride of 2 to generate 110*110*32 and enter the next layer.
[0044] The second layer is the maximum pooling layer, the size is 3*3, the step size is 2, the output of 55*55*32 is obtained, and the third layer is entered.
[0045] The third layer is the convolution layer, the convolution kernel is 5*5, the stride is 2, and the output of 26*26*86 is obtained.
[0046] The fourth layer is the maximum pooling layer with a size of 3*3 and a stride of 2, resulting in an output of 13*13*32.
[0047] The fifth layer is a convolutional layer, the size of the convolution kernel is 3*3, the stride is 1, and the output of 13*13*128 is obtained.
[0048] The sixth layer is the convolution layer, and the convolution kernel size and stride of the fifth layer are the same, and the output of 13*13*128 is obtained.
[0049] The seventh layer is the convolution layer, and the convolution kernel size and stride of the fifth layer are the same, and the output of 13*13*128 is obtained.
[0050] The eighth layer is the maximum pooling layer with a size of 3*3 and a stride of 2, resulting in a feature map of 13*13*128.
[0051] The region proposal network RPN, which generates candidate regions according to the feature map, includes three convolution layers. The size of the convolution kernel of the first layer is 3*3. The output results are divided into two channels, and the size of the two convolution kernels is 1*. A convolutional layer of 1, which generates the probability that the candidate region is the target and the bounding box coordinates of the candidate region. RPN adopts the anchor mechanism. On the feature map generated by the original image through the convolutional neural network, each point corresponds to the 9 anchors on the original image. After the region proposal network RPN generates the regression sum of the four coordinates of these 9 anchors as the target probability value.
[0052] Since the position of the candidate region is a floating-point number, and the pooled feature map requires a fixed size, it is necessary to use ROI Align to extract a fixed-size feature map for subsequent classification and bounding box regression operations. The process of ROI Align is to traverse each candidate area and divide the candidate area into k*k units. The floating-point boundary of the candidate area and the boundary of each unit are not quantized, and four fixed coordinates are calculated in each unit. position, the values of these four positions are calculated by bilinear interpolation, and then the maximum pooling operation is performed.
[0053] According to the feature map of fixed size, softmax is used to separate the candidate regions, and at the same time, the final accurate position of the bounding box is obtained by regression calculation again.
[0054] The Mask branch that generates masks is a fully convolutional network FCN, the input is the fixed-size feature maps generated by ROI Align, and the output is their masks. The Mask branch includes 4 convolutional layers, and finally generates a binary mask map of size 28*28*80. The small size property of the mask helps keep the mask branch network lightweight. During training, we downscale the real mask to 28x28 to compute the loss function, and in inference, upscale the predicted mask to the size of the ROI bounding box to give the final mask result, one mask per target membrane.
[0055] 2. FPGA-based online detection part:
[0056] The FPGA-based millimeter wave imaging dangerous goods detection part mainly applies the deep learning network model generated by offline training to real-time dangerous goods detection. After the FPGA is initialized, the trained deep learning network model parameters are loaded; when the person to be tested enters the millimeter-wave detection area, the millimeter-wave imaging device images the millimeter-wave reflected by the human body and collects real-time data. The size is 200*512 pixels; the FPGA takes the image data as input, passes it to the deep learning program framework in the FPGA, and uses the loaded network model parameters to detect dangerous objects; if there are dangerous objects in the currently processed image, mark the danger The category of the item starts to alarm, and the area with the mask value of 1 in the image is obtained at the same time, which is used to identify the location coordinates of the dangerous item. If no hazardous material is present, proceed to the next image.
[0057] Deep learning neural network has the advantages of high detection rate and fast detection speed, but the calculation amount is huge, and the effect of real-time detection cannot be achieved by using embedded processor or server CPU. In order to reduce the computational load of the FPGA, the millimeter-wave imaging algorithm is partially carried out in a piece of FPGA independent of the detection of dangerous objects, and the forward transfer operation in the network model is carried out in another piece of FPGA, thus realizing the acceleration of the deep learning network. The operation makes the whole detection process of dangerous goods have real-time performance.
[0058] After the FPGA used for millimeter wave imaging passes the imaging algorithm, it outputs a fixed image size millimeter wave image to the back-end FPGA, and the image resolution is 200*512; the image is output to the back-end FPGA through the high-speed bus for network model operation. The forward transfer process of the selected target detection network is mainly composed of multi-channel convolution operations. Taking advantage of the concurrent operation of FPGA, the convolution operations of each channel can be performed at the same time, thereby accelerating the neural network execution process.
[0059] Compared with the traditional CPU, the FPGA platform has better concurrency and faster processing speed; compared with the traditional GPU, it has finer concurrent operation granularity and concurrent execution efficiency; compared with the application-specific integrated circuit chip, it has better functions and can be customized sexual advantage.
[0060] The traditional image detection algorithm and the deep learning network based on other models are used for testing, and the effects are not as good as the technical solution of the present invention using the deep learning network. Specifically, the detection rate of dangerous goods based on the Mask R-CNN deep network model is replaced by the traditional image detection algorithm, and the detection rate is significantly reduced; the detection of dangerous goods based on the Mask R-CNN deep network model is replaced by other deep learning networks, which cannot Accurately describe the irregular contours of dangerous goods, thereby increasing the workload of manual judgment.
[0061] A comparison was made between using the FPGA platform and the traditional processor platform to run the Mask R-CNN-based deep network model to detect dangerous goods. Using the traditional processor-based platform, the detection time was much longer than the FPGA-based platform.
[0062] The above descriptions are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements and improvements made within the spirit and principles of the present invention shall be included in the protection of the present invention. within the range.
