Dual-mode network based multi-data set x-ray security image target detection method

Abstract

The application relates to a kind of multi-data set X-ray security image target detection methods based on dual-mode network, comprising: lattice appearance enhancer network connection basic feature extraction network forms target detection network;Through target detection network and K X-ray security image data set, the teacher model of K target detection network is trained;Common class mode learning network and unique class mode learning network of K X-ray security image data set are constructed;Using the teacher model of K target detection network after training, student model of target detection network is alternately carried out knowledge distillation under common class mode learning network and unique class mode learning network;Student model of target detection network after knowledge distillation is trained using back propagation algorithm;The student model of target detection network after training is used to carry out target detection to X-ray security image, and the position and corresponding category of contraband in X-ray security image are obtained. The position and corresponding category of contraband can be accurately detected, and contraband is marked.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image target detection, and particularly relates to a multi-dataset X-ray security image target detection method based on a dual-mode network. BACKGROUND

[0002] Target detection is an important problem in computer vision, which refers to identifying which regions in an image are target objects and giving the location, size and category of the target objects. Target detection is very widely used in computer vision applications, such as self-driving cars, drones, security monitoring, etc. In the application scenario of security monitoring, there is a kind of image which is imaged by X-ray, such as the image output by the security inspection instrument in the airport, station and subway station, which requires X-ray image target detection to solve the task of automatic detection of dangerous goods in the security inspection process.

[0003] Early X-ray image target detection uses methods based on hand-crafted features, and the effect of these methods is not ideal. In recent years, with the continuous development of deep learning technology, deep convolutional neural networks have been widely used in various computer vision tasks and have achieved very good results. In 2016, ​S, Kundegorski M E, Devereux M, et al. Transfer learning using convolutional neural networks for object classification within x-ray baggage security imagery [C] / / 2016 IEEE International Conference on Image Processing (ICIP). IEEE, 2016: 1057-1061.) first applied CNN to the field of X-ray security inspection image recognition, which was used for the whole process of feature extraction, representation and classification. They used transfer learning method based on AlexNet, and through layer by layer freezing AlexNet, fine-tuning the parameters of convolutional layer and fully connected layer, to a certain extent, alleviated the limited availability of X-ray security inspection image dataset, and achieved high performance. Miao (Miao C, Xie L, Wan F, et al. Sixray: A large-scale security inspection x-ray benchmark for prohibited item discovery in overlapping images [C] / / Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition. 2019: 2119-2128.) et al. published an X-ray image dataset SIXray, and proposed a method called class-balanced hierarchical refinement to solve the problem of class imbalance, and designed a class-balanced loss function to alleviate the noise introduced by easy negative samples. Wei (Wei Y, Tao R, Wu Z, et al. Occluded Prohibited Items Detection: An X-ray Security Inspection Benchmark and De-occlusion Attention Module [C]. ACM Multimedia 2020.) et al. proposed a high-quality dataset OPIXray designed for target detection tasks in security inspection scenarios, and all dangerous goods contained in the dataset were manually labeled by professional security inspectors in a certain airport. In addition, the paper also proposed a method for detecting occluded prohibited items in security inspection scenarios, and the core of the method is a de-occlusion attention module.The module can be inserted into most detectors as a plug-and-play module, and the purpose is to detect the hidden dangerous goods in the X-ray image. Zhao (Zhao R, Zhu L, Dou S, et al. Detecting Overlapped Objects in X-Ray Security Imagery by a Label-Aware Mechanism[J] / / IEEE TRANSACTIONS ON INFORMATION FORENSICS AND SECURITY, 2022: 999-1009.) and others proposed a new dataset CLCXray for X-ray security image overlap problem. For the overlap problem, the article proposes a label-aware mechanism similar to attention mechanism, which adjusts the feature map according to the label assignment information, and distinguishes different objects in the overlap area in the high-dimensional feature layer. The new method has achieved better results on CLCXray and existing datasets. The current research mainly focuses on the performance on a single dataset, and a single dataset often contains only a limited number of categories. In practical applications, the categories involved in contraband are often diverse and need to be updated constantly. In addition, the X-ray security image target detection model often needs to be deployed in stations, airports and subway stations, which puts forward certain requirements for the deployment cost and performance.

[0004] Aiming at the problems existing in the prior art, a multi-dataset X-ray security image target detection method based on a dual-mode network is provided. SUMMARY

[0005] Aiming at the problems existing in the prior art, the present application provides a multi-dataset X-ray security image target detection method based on a dual-mode network, which can effectively solve the problems existing in the prior art.

[0006] The present application provides a multi-dataset X-ray security image target detection method based on a dual-mode network, comprising:

[0007] Prepare K X-ray security image datasets and divide them into training sets and test sets, and unify the resolution of all images in the K X-ray security image datasets, wherein K is any positive integer;

[0008] Connect the Lattice appearance enhancer network to the basic feature extraction network to form a target detection network;

[0009] Train a teacher model of K target detection networks through the target detection network and the K X-ray security image datasets;

[0010] Construct a common class mode learning network and a unique class mode learning network for the K X-ray security image datasets;

[0011] The teacher model of the K target detection networks trained is used to alternately perform knowledge distillation on the student model of the target detection network under the common class mode learning network and the unique class mode learning network, to obtain the student model of the target detection network after knowledge distillation;

[0012] The student model of the target detection network after knowledge distillation is trained by using a back propagation algorithm, to obtain the target detection network after training;

[0013] The student model of the target detection network after training is used to perform target detection on the X-ray security inspection image, to obtain the position and corresponding category of the contraband in the X-ray security inspection image.

[0014] The Lattice appearance enhancement sub-network includes an edge perception module and a texture perception module; the edge perception module includes an edge detection module, a first weight learning module, and a first Lattice structure module; and the texture perception module includes a region information aggregation module, a second weight learning module, and a second Lattice structure module.

[0015] The Lattice appearance enhancement sub-network is used for appearance enhancement of an input image, and the appearance enhancement of the input image includes:

[0016] The input image X is input into the first weight learning module to obtain a first weight feature map F1, and the input image X is input into the edge detection module to obtain an edge image E;

[0017] The first weight feature map F1, the input image X, and the edge image E are input into the first Lattice structure module respectively to obtain an edge perception feature map F EAM ;

[0018] The edge image E is input into N 3*3 conv2d convolution layers, N batch normalization layers, and N ReLU activation function layers to obtain an edge feature map F E ;

[0019] The edge perception feature map F EAM is input into the region information aggregation module to obtain a region information aggregation feature map F RIA ;

[0020] The edge perception feature map F EAM is input into the second weight learning module to obtain a second weight feature map F2;

[0021] The second weight feature map F2, the edge feature map F E , and the region information aggregation feature map F RIA are input into the second Lattice structure module to obtain a texture perception feature map FTAM ;

[0022] texture-aware feature map F TAM After a 1*1 conv2d convolution layer, a batch normalization layer and a sigmoid activation function layer, the attention feature map F is obtained AT ;

[0023] The attention feature map F AT and the edge-aware feature map F EAM are element-wise multiplied to obtain the output appearance enhancement feature map F LAE .

[0024] The basic target detection network adopts a feature pyramid mode for extraction.

[0025] The common class mode learning network includes a domain discriminator and a class prototype alignment module, which are used to solve the feature distribution deviation problem between domains of the K X-ray security image datasets.

[0026] The student model of the target detection network is pre-trained through random combination of the K X-ray security image datasets.

[0027] The use of the trained K target detection network teacher models to alternately use the common class mode learning network and the unique class mode learning network for knowledge distillation of the target detection network student model includes:

[0028] Batch data {X, Y} is sampled from one of the X-ray security image datasets in each iteration;

[0029] A sample x i in the batch data {X, Y} is randomly input to the student model and its corresponding teacher model to extract features;

[0030] The extracted features are input into the domain discriminator to calculate the adversarial loss;

[0031] The extracted features are input into the class prototype alignment module to update the class feature prototype of the corresponding dataset;

[0032] The updated class feature prototype is used to calculate the prototype alignment loss;

[0033] The features extracted by the teacher model and the student model are subjected to feature knowledge distillation, so that the student model imitates the features of the current sample corresponding to the teacher model.

[0034] The student model of the target detection network after knowledge distillation is trained using a backpropagation algorithm, which includes:

[0035] Sampling batch data {X, Y} from one of the X-ray security image data sets in each iteration;

[0036] Randomly sampling a sample x from the batch data {X, Y} i Input to the student model to extract features;

[0037] The extracted features are forward propagated in the target detection network to obtain a prediction result

[0038] Calculate the prediction result Detection loss with real label y For updating the parameters of the target detection network.

[0039] Among them, the basic target detection network includes a regression subnetwork and a classification subnetwork.

[0040] Among them, the position and corresponding category of the contraband in the X-ray security image include:

[0041] The position of the contraband in the X-ray security image is labeled using the regression subnetwork to obtain the contraband target bounding box coordinate value in the X-ray security image;

[0042] The category of the contraband in the X-ray security image is predicted using the classification subnetwork to obtain the corresponding category of the contraband in the X-ray security image.

[0043] Therefore, the present application provides the following effects and / or advantages:

[0044] First, the traditional DOAM network is modified by using the Lattice filter structure, an edge perception module is added to strengthen the edge features of the target region in the image, thereby effectively solving the problem of overlapping and occlusion of object edges caused by the characteristics of X-ray images; a texture perception module is added to more effectively aggregate the effective information within the region, thereby alleviating the influence of the difference in X-ray image acquisition sources on feature extraction. By using the edge perception and texture perception modules, the appearance of the target object on the image is enhanced, which facilitates subsequent image feature extraction and improves the effectiveness of the extracted features.

[0045] Second, the student and teacher model mode is used to train the overall target detection network, one teacher model is trained for one data set, so that the teacher model extracts a certain type of effect better, and the parameters of the trained teacher model are migrated to the student model. The student model uses a lighter model, which not only conforms to the general principle of knowledge distillation, but also makes the network model obtained by the present application easier to deploy in the actual market environment.

[0046] Third, the common class branch and unique class branch (dual mode) learning network of multiple data sets is designed. For the common class branch learning network, a domain discriminator and a class prototype alignment module are used to solve the domain bias problem. For the unique class branch learning network, the knowledge distillation learning method is used for knowledge integration. Different classes of different data sets can be effectively integrated without significant performance loss.

[0047] Fourth, the basic feature extraction network solves the multi-scale problem of the target through the feature pyramid (FPN) extraction mode, and solves the class imbalance problem through the use of Focal Loss. The feature extraction function has the characteristics of fast speed, simple network structure and high precision. Further, the basic feature extraction network includes a regression sub-network and a classification sub-network, which can better identify the target according to the image features, and accurately mark the position and corresponding category of the prohibited items.

[0048] It should be understood that the above summary and the following detailed description of the present application are exemplary and explanatory, and are intended to provide further explanation of the application as claimed. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The method flowchart in the present application.

[0050] Figure 2 The overall network structure diagram in the present application.

[0051] Figure 3 The network structure diagram of the Lattice appearance enhancer network in the present application. DETAILED DESCRIPTION

[0052] In order to facilitate the understanding of those skilled in the art, the structure of the present application will be further described in detail in combination with the drawings: it should be understood that in the present embodiment, the order of the steps mentioned can be adjusted according to actual needs, even simultaneously or partially simultaneously, except for the order specified.

[0053] Please refer to Figure 1 and Figure 2 The embodiment of the present application provides a multi-data set X-ray security inspection image target detection method based on a dual mode network, which comprises the following steps:

[0054] S1 prepares K X-ray security inspection image data sets and divides them into training sets and test sets, and unifies the resolution of all images in the K X-ray security inspection image data sets, wherein K is any positive integer;

[0055] In this step, the training data set used is a commonly used data set for X-ray security image target detection, which is SIXray and OPIXray, respectively, both of which are collected in actual security scenes. SIXray is an X-ray image data set containing gun, straight knife, wrench, piler and scissor, which consists of 8929 X-ray images. The categories of OPIXray data set include folding knife, straight knife, scissor, utility knife and multi-tool knife, which consists of 8885 X-ray images. The images of the two data sets are divided into training and test sets. When training the student model later, the combination of the two data sets is used to simulate the multi-data set scene.

[0056] Specifically, scale transformation is performed on the K X-ray security image data sets to change the images to the same resolution.

[0057] In this step, since the image sizes in the SIXray and OPIXray two data sets are different, directly inputting the network is not conducive to the learning of the network model, so all the images need to be changed to the same resolution before being input into the network. Considering the size distribution of the image resolution in the data set, if the resolution of all images is transformed to the maximum value among them, the small resolution images will be severely distorted, and processing large resolution images will seriously affect the running speed of the network, so in this application we first down-sample the input images with large resolution to reduce the resolution of the images, thereby speeding up the running speed of the network. However, the resolution of the image cannot be too small, too small image will lose a lot of detail information and affect the accuracy of target detection. Therefore, considering the above factors, all images are transformed to 1333*800 before being input into the network. During training, the images will undergo additional random flipping and color transformation for data augmentation to enrich the diversity of training data.

[0058] S2 connects the Lattice appearance enhancer network to the base feature extraction network to form a target detection network;

[0059] Among them, please refer to Figure 3 , the Lattice appearance enhancer network includes an edge perception module and a texture perception module; wherein the edge perception module includes an edge detection module, a first weight learning module, a first Lattice structure module; the texture perception module includes a region information aggregation module, a second weight learning module, a second Lattice structure module.

[0060] In this step, the de-occlusion attention module DOAM is taken as the object of modification. It proposes to enhance the attention of the edges and textures of the X-ray image to solve the occlusion problem of the X-ray image. DOAM is composed of edge guidance, texture perception and attention generation modules in turn. The edge guidance module includes N 3*3 conv2d convolution layers, batch normalization layers and ReLU activation function layers. The texture perception module includes M 3*3 conv2d convolution layers, batch normalization layers, ReLU activation function layers, and a region information aggregation module. The input image is input into the edge guidance module to obtain an edge image. The edge image and the input image are connected and then input into the texture perception module. The output features of the edge guidance and texture perception modules are connected and then input into the feature map generation module. Then, a 1*1 conv2d layer is used, followed by a Sigmoid activation function layer. Then, the obtained feature map is multiplied with the image obtained by connecting the input image and the edge image to obtain the output feature map of DOAM. In the DOAM module, only the simplest connection operation is used for feature aggregation, which cannot effectively aggregate the features between modules and cannot select the weights. Therefore, DOAM is modified from these two points.

[0061] Further, a weight learning module is introduced to strengthen the model's weight for specific regions of the image, thereby improving the model's positioning ability for edge and texture enhancement. The first weight learning module and the second weight learning module are both composed of a 1*1 conv2d layer and a Sigmoid activation function layer. Then, a Lattice filter structure is introduced to replace the simple connection operation. The structure of the Lattice filter, also known as X-section, is a physical topology of a all-pass filter with a butterfly structure, which decomposes the input signal into multiple order representations. After modification, the Lattice appearance enhancement subnetwork (LAE) includes an edge perception module and a texture perception module, which respectively solve the problems of edge overlap occlusion and texture deviation caused by different X-ray image imaging devices and imaging environments.

[0062] Among them, please refer to Figure 3 , the Lattice appearance enhancement subnetwork is used for appearance enhancement of an input image, and the appearance enhancement of the input image includes:

[0063] S201 inputs the input image X into the first weight learning module to obtain the first weight feature map F1, and inputs the input image X into the edge detection module to obtain the edge image E;

[0064] S202 inputs the first weight feature map F1, the input image X and the edge image E into the first Lattice structure module respectively to obtain the edge perception feature map F EAM ;

[0065] The first weight feature map F1 is input into the first Lattice structure module respectively, and the edge-aware feature map F is obtained after the input image X and the edge image E are input into the first Lattice structure module respectively EAM Comprise:

[0066] S2021, the first weight feature map F1 is multiplied with the input image X element by element, and then added with the edge image E element by element, and the feature map L1 is obtained after the activation function ReLU;

[0067] S2022, the first weight feature map F ! Is multiplied with the edge image E element by element, and then added with the input image X element by element, and the feature map L2 is obtained after the activation function ReLU;

[0068] S2023, the feature map L1 and the feature map L2 are added element by element to obtain the edge-aware feature map F EAM .

[0069] S203, the edge image E is input into N 3*3 conv2d convolution layers, N batch normalization layers and N ReLU activation function layers to obtain the edge feature map F E ;

[0070] S204, the edge-aware feature map F EAM Is input into the region information aggregation module to obtain the region information aggregation feature map F RIA ;

[0071] S205, the edge-aware feature map F EAM Is input into the second weight learning module to obtain the second weight feature map F2;

[0072] S206, the second weight feature map F2, the edge feature map F E , and the region information aggregation feature map F RIA Are input into the second Lattice structure module to obtain the texture-aware feature map F TAM ;

[0073] Wherein, the second weight feature map F2, the edge feature map F E , and the region information aggregation feature map F RIA Are input into the second Lattice structure module to obtain the texture-aware feature map F TAM Comprise:

[0074] S2061, the second weight feature map F2 is multiplied with the region information aggregation feature map F RIA Element by element, and then added with the edge feature map F E Element by element, and the feature map L3 is obtained after the activation function ReLU;

[0075] S2062 multiply the second weight feature map F2 and the edge feature map F E element-wise after multiplication with the region information aggregation feature map F RIA element-wise addition, and an activation function ReLU to obtain the feature map L4.

[0076] S2063 element-wise addition of the feature map L3 and the feature map L4 to obtain the edge texture perception feature map F TAM .

[0077] S207 texture perception feature map F TAM After a 1*1 conv2d convolution layer, a batch normalization layer and a Sigmoid activation function layer, the attention feature map F AT is obtained.

[0078] S208 element-wise multiplication of the attention feature map F AT and the edge perception feature map F EAM to obtain the output appearance enhancement feature map F LAE .

[0079] Please refer to Figure 3 , in this step, first input the X-ray image into the edge perception module, respectively through the edge detection module and the first weight learning module two parallel branches. The first weight learning module obtains the first weight feature map F1, and the first weight feature map F1 is copied into two identical first weight feature maps F1 after inputting into the first Lattice structure module, and is multiplied element-wise with the input image X and the edge image E respectively. The second weight learning module obtains the second weight feature map F2, and the second weight feature map F2 is copied into two identical second weight feature maps F2 after inputting into the second Lattice structure module, and is multiplied element-wise with the region information aggregation feature map F RIA and the edge feature map F E respectively. The structure of the first weight learning module and the second weight learning module is the same, which includes a 1*1 conv2d convolution layer and a Sigmoid activation function layer.

[0080] Wherein, the basic target detection network adopts a feature pyramid mode to extract.

[0081] In this step, RetinaNet is used as the basic feature extraction network, which is a single-stage target detection method. It solves the problem of multi-scale target through the feature pyramid mode (FPN), and solves the class imbalance problem through the use of Focal Loss. It has the characteristics of fast speed, simple network structure and high precision. The present application considers the actual application scene of X-ray security image target detection, and uses ResNet101 and ResNet50 two kinds of basic feature extraction networks for the teacher model and the student model in knowledge distillation.

[0082] S3 trains K target detection network teacher models through the target detection network and K X-ray security image data sets;

[0083] In this step, for the two data sets SIXray and OPIXray used in the present application, the Lattice appearance enhancement sub-network in step S2 is used to connect the basic feature extraction network to form a target detection network, and the teacher models of the two data sets are trained. In the knowledge distillation learning mode, the teacher model is a pre-training model before formal training, which is used to guide the student model during formal training. The guidance is realized by constraining the feature distance between the teacher model and the student model. The teacher model usually has more parameters and longer training time than the student model. In the present application, the teacher model uses ResNet101 as the basic feature extraction network, and is trained for 120 rounds. The model training parameters of each data set are adjusted separately.

[0084] S4 constructs a common class mode learning network and a unique class mode learning network for K X-ray security image data sets;

[0085] The common class mode learning network includes a domain discriminator and a class prototype alignment module, which is used to solve the problem of feature distribution deviation between domains of the K X-ray security image data sets.

[0086] In this step, please refer to Figure 2, a target detection network of double-branch learning mode is designed. The network is composed of a common class branch and a unique class branch. In the common class branch, a domain discriminator and a class prototype alignment module are included to solve the feature distribution deviation problem existing among multiple data domains. The domain discriminator is composed of three fully connected layers and a Sigmoid activation function. The features from the feature pyramid (FPN) are input into the domain discriminator to determine the domain class of the features. Through the gradient reversal layer, gradient backpropagation is realized to play an adversarial learning role, so that the student model learns domain-independent features. The class prototype alignment module further improves the effectiveness of the features learned by the model and improves the classification accuracy after detection by explicitly aligning the features of the corresponding classes between the datasets. In the unique class branch, knowledge distillation is used for knowledge integration. Through knowledge distillation, different classes of different datasets can be effectively integrated without significant performance loss.

[0087] S5 uses the trained K target detection network teacher models to alternately perform knowledge distillation on the student model of the target detection network under the common class mode learning network and the unique class mode learning network, to obtain the knowledge-distilled student model of the target detection network;

[0088] Wherein, the student model of the target detection network is pre-trained through random combination of the K X-ray security inspection image datasets.

[0089] Wherein, the use of the trained K target detection network teacher models to alternately perform knowledge distillation on the student model of the target detection network under the common class mode learning network and the unique class mode learning network includes:

[0090] S501 samples batch data {X, Y} from one of the X-ray security inspection image datasets each iteration;

[0091] S502 randomly selects a sample x i from the batch data {X, Y} and inputs it to the student model and its corresponding teacher model to extract features;

[0092] S503 inputs the extracted features into the domain discriminator to calculate the adversarial loss;

[0093] In this step, the student model is trained using the teacher model obtained in step S3 and the combination of the SIXray and OPIXray 2 datasets using the double-branch mode learning network designed in step S4. Specifically, batch data {X, Y} is sampled from one of the datasets each iteration, and a sample x i is randomly selected from the batch data {X, Y} and input into the student network and its corresponding teacher network to extract features, and then the features extracted by the feature pyramid (FPN) are input into the domain discriminator In the specific implementation, the adversarial loss is calculated for updating the network parameters, and the calculation process is as follows:

[0094]

[0095] where the indicator function is equal to 1 only when i = k, and B is the number of samples in the batch data.

[0096] S504 inputs the extracted features to the category prototype alignment module to update the category feature prototype of the corresponding data set;

[0097] In this step, the features extracted by the feature pyramid network (FPN) are input to the category prototype alignment module to update the category prototype of the corresponding data set, and the calculation process is as follows:

[0098] p k,c ← αp k,c + (1-α)q X,c

[0099] where p k,c is the current value of the category prototype in a certain round, q X,c is the feature of the corresponding category in the feature pyramid in the round, and α is a weight parameter.

[0100] S505 calculates the prototype alignment loss using the updated category feature prototype;

[0101] In this step, the prototype alignment loss is calculated using the updated category feature prototype, so that the common class features extracted by the model for different data sets are subject to the same feature distribution, the feature difference between data sets is reduced, and the accuracy of target detection is improved. The calculation process is as follows:

[0102]

[0103] where C is the number of common classes, K is the number of data sets, and ‖·‖ F is the Frobenius norm.

[0104] S506 performs feature knowledge distillation on the features extracted by the teacher model and the student model, so that the student model imitates the features of the teacher model corresponding to the current sample.

[0105] In this step, the feature pyramid of the teacher model and the student model is subjected to feature knowledge distillation, so that the student model imitates the features of the teacher model corresponding to the current sample. The calculation process of the distillation loss is as follows:

[0106]

[0107] Wherein, B is the number of samples in the batch data, M is the number of feature maps output by the feature pyramid, T l (·) represents a feature transformation operation, Respectively, the student model and the teacher model, P l (·) represents a feature map output by the feature pyramid.

[0108] S6 trains the student model of the knowledge distilled target detection network using the back propagation algorithm to obtain the trained target detection network;

[0109] Wherein, the student model of the knowledge distilled target detection network is trained using the back propagation algorithm to obtain the trained target detection network, comprising:

[0110] S601 samples batch data {X, Y} from one of the X-ray security image data sets each time;

[0111] S602 randomly selects a sample x i Input to the student model to extract features;

[0112] S603 the extracted features are subjected to forward propagation in the target detection network to obtain the prediction result

[0113] S604 calculates the detection loss of the prediction result And the real label y Used to update the parameters of the target detection network.

[0114] Wherein, the basic target detection network comprises a regression subnetwork and a classification subnetwork.

[0115] Wherein, the prediction result Contains the target bounding box coordinate values obtained by the regression subnetwork and the corresponding class of the target in the bounding box predicted by the classification subnetwork.

[0116] In this step, batch data {X, Y} is sampled from one of the data sets each time, and for one of the samples x i , input the student network to extract features, and pass through the forward propagation of the target detection network D to obtain the prediction result Contains the target bounding box coordinate values obtained by the regression network and the corresponding class of the target in the bounding box predicted by the classification network. Then calculate the detection loss of the prediction result And the real label y Used to update the network parameters:

[0117]

[0118] Wherein, for a classification subnetwork, for a regression subnetwork.

[0119] In this step, during the training process, for data preprocessing, three data augmentation methods of random flipping, random scaling and random occlusion are used, and data normalization is performed on all training and test data. The target detection network starts training with the ImageNet pre-trained ResNet as the initialization after fine-tuning, with stochastic gradient descent as the optimizer, where the initial learning rate is 0.002, the weight decay parameter is 0.0001, and the momentum is 0.9. For the learning rate update strategy, the popular "poly" strategy is adopted, that is, for each iteration lr = baselr * (1-iter / max_iter)^power, where baselr is the initial learning rate, iter is the current iteration number, max_iter is the total iteration number, and power is the power of the polynomial (generally set to 0.9). The mini-batch of each iteration is set to 4, and the whole network is trained for 120 cycles.

[0120] S7 uses the trained student model of the target detection network to perform target detection on the X-ray security image to obtain the position and corresponding category of the prohibited article in the X-ray security image.

[0121] The position and corresponding category of the prohibited article in the X-ray security image are obtained, comprising:

[0122] S701 uses the regression subnetwork to label the position of the prohibited article in the X-ray security image to obtain the coordinate value of the prohibited article target bounding box in the X-ray security image;

[0123] S702 uses the classification subnetwork to predict the category of the prohibited article in the X-ray security image to obtain the corresponding category of the prohibited article in the X-ray security image.

[0124] In this step, the resolution of the input X-ray security image needs to be transformed to 1333*800, input into the trained target detection network, and after the appearance of the prohibited article in the X-ray security image is enhanced by the Lattice appearance enhancement subnetwork, the extracted appearance and texture features are input into the regression subnetwork in the basic feature extraction network to obtain the position bounding box coordinate value of the detected prohibited article, and then input into the classification subnetwork in the basic feature extraction network to predict the corresponding category of the prohibited article. The regression subnetwork can obtain the position of the prohibited article in the image and its position bounding box coordinate value, and the classification subnetwork can obtain the corresponding category of the prohibited article.

[0125] Table 1 is a performance comparison of the present application and other nine X-ray security image target detection methods on the SIXray+OPIXray dataset. Among them, mAP is the average mean accuracy, which is obtained by calculating the proportion of the detection results and the true value satisfying the intersection over union (IoU). mAP is the most stringent and challenging evaluation index, which is obtained by calculating the accuracy of IoU from 0.5 with an increment of 0.05 to 0.95, mAP50 and mAP75 are respectively calculated when the accuracy of IoU is 0.5 and 0.75, mAP75 is more stringent than mAP50, and the numerical value is generally lower than mAP50. The accuracy of the method proposed in the present application under the strict evaluation index and the commonly used evaluation index is significantly better than that of the existing general or X-ray security image target detection algorithm, so as to more accurately identify the contraband and ensure the reliability and safety of the practical application of the method.

[0126] Table 1

[0127] Method Precision (mAP) Precision (mAP50) Precision (mAP75) RetinaNet 46.1 83.5 45.0 RetinaNet+DOAM 47.3 84.7 45.9 SSD 37.7 75.8 33.9 SSD+DOAM 40.9 79.9 36.0 YOLOv3 42.5 84.2 38.3 YOLOv3+DOAM 43.6 85.2 39.0 FCOS 43.6 81.5 40.0 FCOS+DOAM 44.7 82.8 41.0 Unified 43.4 n / a n / a The method presented in the invention 51.4 89.2 49.3

[0128] In Table 1:

[0129] RetinaNet corresponds to the method proposed by T.-Y. Lin et al. (T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollar, “Focal loss for dense object detection,” in Proc. IEEE / CVF Int. Conf. Comput. Vis., 2017, pp. 2980-2988.);

[0130] SSD corresponds to the method proposed by W. Liu et al. (W. Liu, D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg, “SSD: Single shot multibox detector,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 21-37.);

[0131] YOLOv3 corresponds to the method proposed by J. Redmon et al. (J. Redmon and A. Farhadi, “YOLOv3: An incremental improvement,” arXiv:1804.02767, 2018.);

[0132] FCOS corresponds to the method proposed by Z. Tian et al. (Z. Tian, C. Shen, H. Chen, and T. He, “FCOS: Fully convolutional one-stage object detection,” in Proc. IEEE / CVF Int. Conf. Comput. Vis., 2019, pp. 9627-9636.);

[0133] Unified corresponds to the method proposed by X. Zhou et al. (X. Zhou, V. Koltun, and P. Krah enbuhl, “Simple multi-dataset detection,” in Proc. IEEE / CVF Conf. Comput. Vis. Pattern Recognit., 2022, pp. 7571-7580.);

[0134] DOAM corresponds to the method proposed by Y. Wei et al. (Y. Wei, R. Tao, Z. Wu, Y. Ma, L. Zhang, and X. Liu, “Occluded prohibited items detection: An X-ray security inspection benchmark and de-occlusion module,” in Proc. ACM Int. Conf. Multimedia., 2020, pp. 138-146.).

[0135] Those skilled in the art will appreciate that embodiments of the present application can be provided as methods, systems, or computer program products. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer-usable program code.

[0136] The present application is described in reference to the accompanying drawings, which use the legend: 100 - method, 200 - device (system), 300 - computer program product. Figure 1 one or more processes and / or blocks Figure 1 one or more blocks to perform a function specified in the flow or multiple flows and / or blocks.

[0137] These computer program instructions can also be stored in a computer- readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instructions which implement the flow Figure 1 one or more processes and / or blocks Figure 1 one or more blocks to perform a function specified in the flow or multiple flows and / or blocks.

[0138] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the flow Figure 1 one or more processes and / or blocks Figure 1 one or more blocks to perform a function specified in the flow or multiple flows and / or blocks.

[0139] It should be noted that the use of any of the following terms "comprising", "including", "containing", "characterized by" and the like, is intended to be open-ended and non-limiting. It should be noted that the use of the term "including" in the claims does not limit the scope of the claims to only the listed steps or elements. The term "including" is used to mean "comprising" or "consisting of," and allows for additional steps or elements to be present without further restricting the scope of the application. The use of the term "comprising" is intended to cover the terms "including" and "consisting of," and is intended to be used in a non-limiting sense. The use of the term "comprising" also covers any additional steps or elements that are present in addition to those listed in the claims. The term "consisting essentially of" is intended to mean that the additional steps or elements are present, but that the additional steps or elements do not materially affect the basic operation of the application. The term "consisting of" is intended to mean that the only steps or elements present are those listed in the claims. The use of the term "or" in the claims is intended to be used in a non-exclusive sense, such that "A or B" means "A or B or both".

[0140] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Such variations and modifications are intended to be covered by the claims.

[0141] Obviously, various modifications and changes can be made to the present application by those skilled in the art without departing from the spirit and scope of the present application. Accordingly, the present application intends to include all such modifications and changes as fall within the scope of the claims and their equivalents.

[0142] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connection", "connecting", "fixing" and the like should be understood in a broad sense, for example, can be fixed connection, or detachable connection, or integral; can be mechanical connection, or electrical connection; can be direct connection, or indirect connection through intermediate medium; can be internal communication of two elements, or interaction relationship between two elements. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0143] In the description of the present application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in any one or more embodiments or examples. In addition, those skilled in the art can combine and combine the different embodiments or examples described in the present application and the features of the different embodiments or examples without contradiction.

Claims

1. A method for target detection in multi dataset x-ray security images based on dual mode network, characterized in that, The application relates to an X-ray security inspection image target detection method and device. Preparation of K X-ray security inspection image data sets and division into a training set and a test set, unification of the resolutions of all images in the K X-ray security inspection image data sets, wherein K is an arbitrary positive integer; A Lattice appearance enhancer network is connected with a basic feature extraction network to form a target detection network; A teacher model of the K target detection networks is trained through the target detection network and the K X-ray security inspection image data sets; A common class pattern learning network and a unique class pattern learning network of the K X-ray security inspection image data sets are constructed; A student model of the target detection network is subjected to knowledge distillation under the common class pattern learning network and the unique class pattern learning network by using the trained teacher model of the K target detection networks in turn, and a student model of the target detection network after knowledge distillation is obtained; The student model of the target detection network after knowledge distillation is trained by using a back propagation algorithm, and a trained target detection network is obtained; The student model of the trained target detection network is used for target detection of an X-ray security inspection image, and the position and corresponding category of contraband in the X-ray security inspection image are obtained. The Lattice appearance enhancer network comprises an edge perception module and a texture perception module; wherein the edge perception module comprises an edge detection module, a first weight learning module and a first Lattice structure module; and the texture perception module comprises a region information aggregation module, a second weight learning module and a second Lattice structure module. The Lattice appearance enhancer network is used for appearance enhancement of an input image, and the appearance enhancement of the input image comprises: inputting the input image X into the first weight learning module to obtain a first weight feature map inputting the input image X into the edge detection module to obtain an edge image E; input the first weight feature map respectively, input image X, edge image E are input into the first Lattice structure module respectively to obtain edge perception feature map ; The edge image E is obtained after N 3*3 conv2d convolution layers, N batch normalization layers and N ReLU activation function layers ; edge-aware feature map input the region information aggregation module to obtain a region information aggregation feature map ; edge-aware feature map second weight feature map obtained after inputting the second weight learning module ; a second weight feature map , an edge feature map , a region information aggregation feature map obtaining a texture perception feature map after inputting the second Lattice structure module ; Texture-aware feature map After a 1*1 conv2d convolution layer, a batch normalization layer and a sigmoid activation function layer, the attention feature map is obtained ; attention feature map and edge-aware feature map after element-wise multiplication to obtain an output appearance enhancement feature map .

2. The dual-mode network based multi dataset x-ray security image target detection method of claim 1, wherein, The target detection network adopts a feature pyramid mode for extraction.

3. The dual-mode network based multi dataset x-ray security image target detection method of claim 1, wherein, The common class pattern learning network comprises a domain discriminator and a category prototype alignment module, and is used for solving the feature distribution deviation problem among the K X-ray security inspection image data sets.

4. The dual-mode network based multi dataset x-ray security image target detection method of claim 1, wherein, The student model of the target detection network is pre-trained through random combination of the K X-ray security inspection image data sets.

5. The dual-mode network based multi dataset x-ray security image target detection method of claim 3, wherein, The student model of the target detection network is subjected to knowledge distillation under the common class pattern learning network and the unique class pattern learning network by using the trained teacher model of the K target detection networks in turn, and a student model of the target detection network after knowledge distillation is obtained. Batch data {X, Y} are sampled from one of the X-ray security inspection image data sets in each iteration; randomly select one sample from the batch data {X, Y} input to the student model and its corresponding teacher model extract features; The extracted features are input into the domain discriminator to calculate an adversarial loss; The extracted features are input into the category prototype alignment module to update the category feature prototype of the corresponding data set; The updated category feature prototype is used to calculate a prototype alignment loss; The features extracted by the teacher model and the student model are subjected to feature knowledge distillation, so that the student model imitates the features of the teacher model corresponding to the current sample.

6. The dual-mode network based multi dataset x-ray security image target detection method of claim 1, wherein, The student model of the target detection network after knowledge distillation is trained by using a back propagation algorithm, and a trained target detection network is obtained. The target detection network comprises a regression subnetwork and a classification subnetwork. Randomly select one sample from the batch data {X, Y} Input to the student model to extract features; The extracted features are subjected to forward propagation in the target detection network to obtain a prediction result ; Computing the prediction result With the real label The detection loss For updating the target detection network parameters.

7. The dual-mode network based multi dataset x-ray security image target detection method of claim 1, wherein, The position and corresponding category of contraband in the X-ray security inspection image are obtained.

8. The dual-mode network based multi dataset x-ray security image target detection method of claim 1, wherein, ​ The regression sub-network is used to label the position of the prohibited article in the X-ray security image, to obtain the coordinate value of the prohibited article target bounding box in the X-ray security image; The classification sub-network is used to predict the category of the prohibited article in the X-ray security image, to obtain the corresponding category of the prohibited article in the X-ray security image.

Images