A non-metallic lighter detection method in X-ray security inspection images based on deep learning

Abstract

The present invention discloses a method for detecting non-metallic lighters in X-ray security inspection images based on deep learning. The data enhancement and multi-scale training of the method use three different data enhancement methods and fuse the data enhancement data in a mix-up manner to improve the positioning capability of non-metallic lighters. The non-metallic lighter recognition model of X-ray security inspection images is applied, and the X-ray security inspection images in real scenes are input into the model, and the model directly outputs the non-metallic lighter detection results. The three data enhancement methods and multi-scale training improve the generalization ability of the model for detecting non-metallic lighters in X-ray security inspection images in real scenes. Improved channel attention is introduced into the feature extraction network to improve the model's extraction of effective features for non-metallic lighters. The DO-PAN network structure is designed in the feature fusion stage to strengthen the fusion of shallow features and deep features, improve the model's recognition capability for non-metallic lighters, and use the Focalloss loss function instead of the cross entropy loss function.

Description

Technical Field

[0001] The present invention belongs to the field of target detection and is a method for detecting non-metallic lighters in X-ray security inspection images based on deep learning. The method can be applied to an intelligent X-ray security inspection system to assist security personnel in detecting prohibited items such as non-metallic lighters. Background Art

[0002] X-ray baggage security inspection is of great significance to maintaining public safety. X-rays penetrate objects of different materials to different degrees, resulting in different imaging colors for different objects, thereby detecting prohibited items in the package. At present, baggage security inspection still relies on security inspectors to visually judge whether the package contains prohibited items based on X-ray security inspection images. However, with the continuous development of public transportation and the surge in the number of tourists traveling, security inspectors are under high pressure all year round. Realizing intelligent baggage security inspection will greatly help improve security inspection efficiency and reduce the workload of security inspectors. At present, target detection technology based on deep learning has made great breakthroughs, and the accuracy and generalization of traditional target detection algorithms have been greatly improved. However, there is currently no method for detecting non-metallic lighters in X-ray security inspection images.

[0003] The difficulties in identifying non-metallic lighters in X-ray security inspection images are as follows: 1. Non-metallic lighters are very small, while security inspection images generally have extremely high resolutions, and non-metallic lighters vary in size. Even the same non-metallic lighters have different relative sizes in different images. 2. Since X-ray security inspection imaging is only related to the thickness and density of items, non-metallic lighters appear yellow in X-ray security inspection images, which is similar to the color of most non-prohibited items such as plastics and food, resulting in a complex and confusing background for non-metallic lighters in real scenes. 3. In real scenes, due to the serious overlapping and occlusion of multiple items in luggage, it is difficult to distinguish non-metallic lighters even with the naked eye. Summary of the invention

[0004] The purpose of this invention is to propose an end-to-end non-metallic lighter detection model based on deep learning for the task of detecting non-metallic lighters in X-ray security inspection images. Figure 1 As shown in the figure, by inputting an X-ray security inspection image of any size, the model directly outputs the location information of non-metallic lighters.

[0005] In order to achieve the above objectives and in view of the real-time and accuracy requirements of the security inspection system, the technical solution adopted by the present invention is a deep learning-based X-ray security inspection image non-metallic lighter detection method, which is divided into two steps:

[0006] (1) Data enhancement and multi-scale training: The data used for model training comes from X-ray security images collected in real scenes at airport security checkpoints and subway security checkpoints, totaling 896 images. The data set consisting of 896 images is enhanced using three different data enhancement methods, and the data enhancement data is fused in a mix-up manner to improve the positioning capability of non-metallic lighters. In order to improve the model's ability to better recognize X-ray security images of any scale, a multi-scale training method is used during training, and the image input size is adjusted to any multiple of 32 between 320 and 608 every 10 iterations.

[0007] (2) End-to-end model: Apply the X-ray security image non-metallic lighter recognition model, input the X-ray security image in the real scene into the model, and the model directly outputs the non-metallic lighter detection result.

[0008] Furthermore, the dataset includes single package images and complex multi-package images. Labelimg is used to annotate the dataset images in PASCAL VOC format. In order to expand the dataset and improve the generalization performance of the model, three data enhancement methods, namely random cropping, random rotation, and random affine transformation, are used. The data are fused in a Mix-Up manner to effectively expand the dataset and enable the enhanced model to locate non-metallic lighters.

[0009] Furthermore, the process of the non-metallic lighter recognition model is as follows: an X-ray security inspection image of any size is input into a convolutional neural network, and after a series of convolution operations, the features of the non-metallic lighter are extracted, and then the feature vector is regressed and classified to output the coordinates and category information of the non-metallic lighter.

[0010] Furthermore, the non-metallic lighter recognition model is divided into two parts, the non-metallic lighter feature extraction network part, and the multi-scale feature fusion and model detection part. The non-metallic lighter feature extraction network part uses the CSPdarknet53 network in YOLOv4. Its network structure, the feature extraction network contains five cross-stage local networks, and each local network has 1, 2, 8, 8, and 4 residual modules respectively. The multi-scale feature fusion and model detection part uses an improved channel attention module to strengthen the feature extraction network. For the extraction of effective features of non-metallic lighters, global maximum pooling and global average pooling are used to jointly reflect the response of the channel.

[0011] M c (F) * =σ(MLP(δ(MLP(MaxPool(AvgPool(F))))))

[0012] Furthermore, the feature map is downsampled five times in the feature extraction network, and the Do-PAN feature fusion module is designed. Do-Conv is used to replace ordinary convolution to fuse shallow features with deep features, further improving the model's recognition accuracy for small targets such as non-metallic lighters.

[0013] Furthermore, on the three output feature maps of the model, each pixel predicts a 3*(4+1+1)-dimensional vector, where 3 means that if the center of the target is located at this pixel, the pixel will predict three bounding boxes, 4 means the coordinate vector, and 1 means the confidence vector and the category probability vector. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 It is an end-to-end recognition model diagram.

[0015] Figure 2 This is a diagram of the detection result of non-metallic lighters in the X-ray security inspection image of the model of the present invention.

[0016] Figure 3 It is an X-ray security inspection image in a real scene.

[0017] Figure 4 It is the result of data enhancement and data fusion.

[0018] Figure 5 It is a non-metal lighter feature extraction network.

[0019] Figure 6 This is a comparison chart of the visualization results of different attention mechanisms.

[0020] Figure 7 This is the Do-PAN module structure diagram.

[0021] Figure 8 This is a graph of simple package recognition results using different models.

[0022] Fig. 9 These are the complex package recognition result diagrams of different models. DETAILED DESCRIPTION

[0023] The present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0024] (1) Data enhancement: The present invention uses three data enhancement methods, namely random cropping (b), random rotation (c), and random affine transformation (d), to expand the total amount of the data set by three times, and uses the Mix-Up method to fuse the data.

[0025] (2) After the data is input into the neural network model, a single Tesla V100 GPU is used for training, the batch size is set to 4, and the SGD optimizer is used. The initial learning rate of this model is set to 0.0001, the weight decay rate is set to 0.0005, and the momentum is set to 0.9. The learning rate cosine decay method is used. In the first two epochs of training, the learning rate is smoothly increased from 0 to the initial learning rate by the learning rate warm-up method. The model began to converge after 40 epochs of training, and the training time was about 8 hours.

[0026] (3) After the neural network model is trained, an X-ray security inspection image of any size is input, and the model can output the non-metal lighter detection result.

[0027] A deep learning-based method for detecting non-metallic lighters in X-ray security inspection images. The method is divided into two steps:

[0028] (3) Data enhancement and multi-scale training: The data used for model training in the present invention comes from X-ray security images collected in real scenes such as airport security checkpoints and subway security checkpoints, totaling 896 images. The data set is enhanced using three different data enhancement methods, and the data enhancement data is fused in a mix-up manner to improve the model's ability to locate non-metallic lighters. In order to improve the model's ability to better recognize X-ray security images of any scale, the present invention uses a multi-scale training method during training, and adjusts the image input size to any multiple of 32 between 320 and 608 every 10 iterations.

[0029] (4) End-to-end model: Figure 2 The non-metallic lighter recognition model of X-ray security inspection images using the present invention is demonstrated. The model is input with X-ray security inspection images in real scenes, and the model directly outputs the non-metallic lighter detection results.

[0030] The core algorithm of the present invention

[0031] 1. Data preprocessing

[0032] The dataset of the present invention is collected from high-resolution X-ray security inspection images in real scenes such as airport security checkpoints and subway security checkpoints. Figure 3 As shown in the figure, the dataset of the present invention includes simple single-package images and complex multi-package images, totaling 896 images. Subsequently, professionals used Labelimg to annotate the dataset images in the PASCAL VOC format to expand the dataset and improve the generalization performance of the model. Figure 4As shown, the present invention uses three data enhancement methods, namely random cropping (b), random rotation (c), and random affine transformation (d), and then fuses these data in a Mix-Up manner (d), which not only effectively expands the data set but also enhances the model's ability to locate extremely small contraband such as non-metallic lighters.

[0033] 2. End-to-end X-ray non-metal lighter recognition model

[0034] The process of the non-metallic lighter recognition model is as follows: X-ray security inspection images of any size are input into the convolutional neural network, and after a series of convolution operations, the non-metallic lighter features are extracted. Then, the feature vectors are regressed and classified, and the coordinates and category information of the non-metallic lighter are output.

[0035] The model of the present invention is divided into two parts, the non-metal lighter feature extraction network part, and the multi-scale feature fusion and model detection part. In the feature extraction network part, the present invention uses the CSPdarknet53 network in YOLOv4, and its network structure is as follows Figure 5 As shown, the feature extraction network contains five cross-stage local networks, and each local network has 1, 2, 8, 8, and 4 residual modules respectively. In addition, the X-ray security inspection machine uses X-ray transmission imaging, and its imaging color is related to the material of the object. The color of non-metallic lighters is close to that of various security items such as clothing and beverage bottles. In addition, the actual security inspection items are randomly placed, resulting in the overlap of the image background and non-metallic lighters and difficult to distinguish, making recognition extremely difficult. The attention mechanism is often used in the field of computer vision, which enables the network to focus on key information and ignore irrelevant redundant features. In the non-metallic lighter recognition task, the features of non-metallic lighters are similar to those of background items. The present invention believes that the channel attention using global average pooling cannot accurately reflect the response of the channel to the detection target. In addition, non-metallic lighters are small in size and their position is irrelevant to the image structure. The introduction of the spatial attention mechanism will destroy the true distribution of features, which is not conducive to the model extracting effective features. Therefore, as shown in the following formula, the present invention proposes an improved channel attention module to strengthen the feature extraction network for the extraction of effective features of non-metallic lighters. This module uses global maximum pooling and global average pooling to jointly respond to the response of the channel. As shown Figure 6 As shown, the improved channel attention module designed by the present invention has better performance.

[0036] M c (F) * =σ(MLP(δ(MLP(MaxPool(AvgPool(F))))))

[0037] The feature map is downsampled five times in the feature extraction network. For non-metal lighters, the downsampling operation may cause the loss of non-metal lighter features. In order to improve the recognition accuracy of non-metal lighters, Figure 7 As shown, the present invention designs a Do-PAN feature fusion module, uses Do-Conv (depth-wise over parameterized convolutional layer) to replace ordinary convolution, fuses shallow features with deep features, and further improves the model's recognition accuracy for small targets such as non-metal lighters.

[0038] like Figure 7 As shown, on the three output feature maps of the model of the present invention, each pixel predicts a 3*(4+1+1)-dimensional vector, where 3 means that if the center of the target is located at the pixel, the pixel will predict three bounding boxes, 4 means the coordinate vector, and 1 means the confidence vector and the category probability vector.

[0039] The present invention uses the same data set to train different models for 100 epochs, and the experimental results are shown in the following table. Compared with SSD, Retina-Net, and YOLOv 4, the recognition accuracy of non-metallic lighters of the model proposed by the present invention is improved by 27%, 15%, and 9%, respectively. The inference time is increased by about 11 milliseconds compared with SSD and YOLOv 4, and reduced by 10 milliseconds compared with Retina-Net. The experimental results show that the non-metallic lighter recognition model designed by the present invention achieves the optimal recognition result while slightly reducing the real-time performance.

[0040] Table 1 Recognition results comparison table

[0041]

[0042] The core technologies of this patent include:

[0043] (1) The introduction of three data augmentation methods, namely random cropping, random affine transformation, and random rotation, as well as multi-scale training, improves the model's generalization ability for detecting non-metallic lighters in X-ray security inspection images in real scenarios.

[0044] (2) An end-to-end X-ray security image non-metallic lighter recognition model was designed. An improved channel attention was introduced into the feature extraction network to improve the model's extraction of effective features of non-metallic lighters. In the feature fusion stage, a DO-PAN network structure was designed to strengthen the fusion of shallow features and deep features, thereby improving the model's recognition ability for small targets such as non-metallic lighters. In addition, the Focal loss function was used instead of the cross entropy loss function.

Claims

1. A method for detecting non-metallic lighters in X-ray security inspection images based on deep learning, characterized in that: This method consists of two steps: (1) Data enhancement and multi-scale training: The data used for model training comes from X-ray security images collected in real scenes at airport security checkpoints and subway security checkpoints, totaling 896 images. The data set consisting of 896 images is enhanced using three different data enhancement methods, and the data enhancement data is fused in a mix-up manner to improve the positioning capability of non-metallic lighters. In order to improve the model's recognition capability for X-ray security images of any scale, a multi-scale training method is used during training, and the image input size is adjusted to any multiple of 32 between 320 and 608 every 10 iterations. (2) End-to-end model: Apply the non-metallic lighter recognition model for X-ray security inspection images. The model inputs the X-ray security inspection images in real scenes, and the model directly outputs the non-metallic lighter detection results. The non-metallic lighter recognition model is divided into two parts, the non-metallic lighter feature extraction network part, and the multi-scale feature fusion and model detection part; the non-metallic lighter feature extraction network part uses the CSPdarknet53 network in YOLOv4, and its network structure, the feature extraction network contains five cross-stage local networks, and each local network has 1, 2, 8, 8, and 4 residual modules respectively; the multi-scale feature fusion and model detection part uses an improved channel attention module to strengthen the feature extraction network, and for the extraction of effective features of non-metallic lighters, global maximum pooling and global average pooling are used to jointly reflect the response of the channel; The feature map is downsampled five times in the feature extraction network, and the Do-PAN feature fusion module is designed. Do-Conv is used to replace ordinary convolution to fuse shallow features with deep features, further improving the recognition accuracy of the model for small non-metal lighter targets. On the three output feature maps of the model, each pixel predicts a 3*(4+1+1)-dimensional vector, where 3 means that if the center of the target is located at this pixel, the pixel will predict three bounding boxes, 4 means the coordinate vector, and 1 means the confidence vector and the category probability vector.

2. The method for detecting non-metallic lighters in X-ray security inspection images based on deep learning according to claim 1, characterized in that: The dataset includes single package images and complex multi-package images. Labelimg is used to annotate the dataset images in PASCAL VOC format. In order to expand the dataset and improve the generalization performance of the model, three data enhancement methods, random cropping, random rotation, and random affine transformation, are used. These data are fused in a Mix-Up manner to effectively expand the dataset and enable the enhanced model to locate non-metallic lighters.

3. The method for detecting non-metallic lighters in X-ray security inspection images based on deep learning according to claim 1, characterized in that: The process of the non-metallic lighter recognition model is as follows: X-ray security inspection images of any size are input into the convolutional neural network, and after a series of convolution operations, the non-metallic lighter features are extracted. Then, the feature vectors are regressed and classified, and the coordinates and category information of the non-metallic lighter are output.

Images