Target detection method for passive terahertz security inspection image based on modal conversion

Abstract

The application discloses a target detection method for passive terahertz security inspection images based on modal conversion, comprising the following steps: fusing a dangerous goods image under X-rays with a passive terahertz human body image, using a cycle generative adversarial network to perform modal style conversion on non-paired data, generating a passive terahertz security inspection image highly similar to a real image, expanding the number and types of a data set through a synthetic image, then training an improved YOLOv5 target detection network by using the data set, learning features of various types of dangerous goods, and finally detecting a real passive terahertz security inspection image by using the trained target detection network, so that the problems of a small number of samples and unbalanced samples in the training process of target detection of passive terahertz security inspection images are solved, the detection precision and generalization are improved, and the detection demand of various types of dangerous goods in an actual security inspection scene is met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of target detection, in particular to a target detection method for passive terahertz security inspection images based on modal conversion. BACKGROUND

[0002] Terahertz (THz) waves refer to electromagnetic waves with a frequency in the range of 0.1-10 THz (wavelength of 3000-30 pm), which have good material penetration characteristics and material fingerprint spectrum identification characteristics, and are therefore used for imaging. Passive terahertz imaging is achieved by detecting the radiation energy emitted by the object itself using a terahertz detector, and ultimately using the differences in the penetration characteristics of different materials to achieve imaging and identification. The energy of a single photon of a terahertz wave is only a few millielectron volts, which is much lower than that of an X-ray, and therefore does not damage the properties of the material due to ionization. Therefore, terahertz imaging can be used for human security inspection and can be used for security inspection in crowded large-scale venues. However, there is a lack of passive terahertz security inspection image data sets for target detection of terahertz security inspection images in the prior art, and therefore there is an urgent need for a target detection method for passive terahertz security inspection images based on modal conversion to solve the above problems. SUMMARY

[0003] The present application provides a target detection method for passive terahertz security inspection images based on modal conversion, which can solve the problems of insufficient and unbalanced samples in the target detection training process of passive terahertz security inspection images in the prior art, and improve the detection accuracy and generalization.

[0004] To achieve the above-mentioned purpose, the present application provides the following technical scheme: a target detection method for passive terahertz security inspection images based on modal conversion, comprising the following steps:

[0005] S1, image fusion of the obtained dangerous goods image under X-rays and the passive terahertz human body image;

[0006] S2, constructing a cycle generative adversarial network to perform modal style conversion of non-paired data and generate passive terahertz security inspection images;

[0007] S3, based on the passive terahertz security inspection images generated by different types of objects, selecting and labeling to produce a passive terahertz security inspection image data set;

[0008] S4, training an improved YOLOv5 target detection network using the passive terahertz security inspection image data set;

[0009] S5, detecting real passive terahertz security inspection images through the trained target detection network.

[0010] Preferably, in step S1, the X-ray dangerous goods image is processed and converted to a grayscale image, a mask of the processed image is established, the effective area with the goods is extracted, and a random position conforming to the two-dimensional normal distribution is generated according to the position statistics of the goods in the real data set, the superimposed area of the terahertz human body image is extracted, and the two images are fused according to the mask of the X-ray image:

[0011] Img add [i,j]=Img1[i,j]*mask[i,j]+Img2[i,j]*(1-mask[i,j]);

[0012] wherein Img add is the superimposed image, mask is the mask of the X-ray image, Img1 and Img2 respectively represent the X-ray image and the extracted area to be fused in the passive terahertz human body image; and Img add covers the corresponding area of the original terahertz human body image.

[0013] Preferably, the X-ray dangerous goods image is processed: the dangerous goods image is annotated with a data set, then the X-ray dangerous goods image is rotated to obtain object images at different angles, and then the images are scaled at a certain ratio.

[0014] Preferably, the grayscale image is contrast enhanced by histogram equalization, and the enhanced image is: wherein I is the image grayscale matrix, I max is the maximum grayscale level in I, and I min is the minimum grayscale level in I.

[0015] Preferably, in step S2, the cycle generative adversarial network includes two generators and two discriminators, wherein the generated passive terahertz security inspection image is denoted as X domain, the real terahertz image is denoted as Y domain, the generator for converting the X domain image to the Y domain image is denoted as G XY , the generator for converting the Y domain image to the X domain image is denoted as G YX , the discriminator for discriminating the X domain image is denoted as D X , and the discriminator for discriminating the Y domain image is denoted as D Y ; the cycle generative adversarial network simultaneously establishes the mapping of X→Y and Y→X.

[0016] Preferably, the loss function in the cycle generative adversarial network conversion process is:

[0017]

[0018] wherein, is the adversarial loss function in the X→Y cycle process is the adversarial loss function in the Y→X cycle process, and λ cyc, lambda idt is a proportionality coefficient, is a cycle consistency loss function, is a dentity loss;

[0019] The optimization objective is:

[0020] Preferably, the generator includes an encoder, a converter and a decoder, wherein the encoder extracts a feature vector from the input image, performs convolution, normalization and activation operation, the converter converts the feature vector of the source domain into the feature vector of the target domain, and the decoder restores the low-level features from the feature vector to generate an image; the discriminator is used to extract features from the image and determine whether the features are close to the image features of a certain domain.

[0021] Preferably, in step S3, the number and types of the data set are continuously expanded by the passive terahertz security image generated in step S5.

[0022] Preferably, in step S4, the YOLOv5 target detection network is built and improved, wherein an attention mechanism module CBAM that fuses channel attention and spatial attention is added at the end of the backbone network of the YOLOv5 target detection network, which focuses on the key positions in the output feature map, and a bidirectional feature fusion method is adopted for the feature extraction module: a weighted bidirectional feature pyramid network BiFPN, which realizes bidirectional fusion of features by using bidirectional cross-scale connection and weighted feature fusion.

[0023] Compared with the prior art, the present application has the following advantages: in the present application, the dangerous goods image obtained under X-rays and the passive terahertz human image are superimposed and fused by mask operation, and a cycle generative adversarial network is used for modal conversion to generate a synthetic terahertz security image that meets the characteristics of terahertz imaging and is close to a real image, wherein the cycle generative adversarial network trains two generators simultaneously and establishes the mappings of X→Y and Y→X, which can train a group of pictures with different contents and structures, and get rid of the limitation that the existing method is difficult to constrain the consistency of the generated image and the input image in terms of content and structure, and needs the structure alignment of the source picture and the target picture, that is, the paired data set; based on the passive terahertz security images generated by different types of goods, a terahertz security image data set with rich types and high clarity can be made, which can effectively improve the detection precision and the range of detected goods, solve the problems of few samples and unbalanced samples in the training process of passive terahertz security image target detection, improve the detection precision and generalization, and adapt to the detection needs of various dangerous goods in actual security scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0024] The accompanying drawings are included to provide a further understanding of the application, and are incorporated in and constitute a part of this specification, illustrate embodiments of the application, and together with the description serve to explain the application, and do not limit the application.

[0025] In the drawings:

[0026] Figure 1 is a flow chart of the target detection method of the present application;

[0027] Figure 2 is a statistical result chart of the anchor frame position and size of dangerous goods in the terahertz image of the present application;

[0028] Figure 3 is a structural diagram of the cycle generative adversarial network of the present application;

[0029] Figure 4 is an effect diagram and a real same category image of the synthesized terahertz picture of the present application;

[0030] Figure 5 is a structural diagram of the convolution block attention module of the present application;

[0031] Figure 6 is a structural diagram of the weighted bidirectional feature pyramid network of the present application;

[0032] Figure 7 is a structural diagram of the target detection effect of the present application. DETAILED DESCRIPTION

[0033] The preferred embodiments of the present application are described below in conjunction with the accompanying drawings, and it should be understood that the preferred embodiments described herein are only used to illustrate and explain the present application, and do not limit the present application.

[0034] Embodiment: As shown in the figure, the target detection method of the passive terahertz security image based on modal conversion includes the following steps: Figure 1

[0035] S1, fuse the obtained X-ray dangerous goods image with the passive terahertz human body image;

[0036] Among them, the X-ray security images are collected, and the pictures are classified according to the type of goods. In a specific embodiment, the X-ray images of knives, guns, scissors and lighters are collected, all of which are white background and the size is uniform as 256*256;

[0037] According to whether to carry dangerous goods, two categories of dangerous and safe are obtained. The pictures of dangerous goods are labeled using labelImg software, and the format is PASCAL VOC, which contains the label and position information of the target goods, and is divided into pistol, large gun, stick weapon and mobile phone; reference Figure 2 ​The relative size and relative position of the anchor box relative to the whole picture are counted.

[0038] The X-ray dangerous goods picture is rotated to obtain pictures of objects at different angles, and the missing background is filled with white. The picture is scaled according to a certain ratio. The ratio is selected according to the relative size of the anchor box counted above and the actual size of the object. Generally, a random number between [0.3, 0.42] is set. The picture processed above is converted from an RGB picture to a grayscale picture. Considering the difference between X-ray and terahertz imaging, the pictures of lighters and other objects are subjected to contrast enhancement operation. Histogram equalization is used for contrast enhancement. The enhanced image is: where I is the image grayscale matrix, I max is the maximum grayscale level in I, and I min is the minimum grayscale level in I.

[0039] A mask of the processed image is established to extract the effective area with the object, for example, the effective area is set to 1 and the invalid area is set to 0. In a specific embodiment, the area with a grayscale value greater than 210 is set to 0. The target coverage area in the terahertz human body picture is selected. Referring to the statistical results of the relative position of the anchor box center obtained above, the anchor box center position is regarded as a two-dimensional normal distribution. The lower left corner of the terahertz picture is regarded as the origin to establish a coordinate system. The distribution of the anchor box center position is described as:

[0040]

[0041] In this embodiment, μ1=0.58, μ2=0.42, σ1=σ2=1, ρ=-0.5, and a random number conforming to the distribution is generated as the anchor box center point. When the coverage area corresponding to the point exceeds the picture range, the point is taken again. The position of the corresponding area of the background human body image is extracted. According to the mask of the X-ray image, the two pictures are fused, Img add [i,j]=Img1[i,j]*mask[i,j]+Img2[i,j]*(1-mask[i,j]); where Img add is the superimposed picture, mask is the mask of the X-ray image, and Img1 and Img2 represent the regions to be fused extracted from the X-ray image and the passive terahertz human body image respectively; Img add covers the corresponding area of the original terahertz human body picture.

[0042] S2, construct a cycle generative adversarial network to perform modal style conversion of unpaired data and generate passive terahertz security inspection images.

[0043] where reference Figure 3As shown, a recurrent generative adversarial network (RGAN) consists of two generators and two discriminators. Ordinary GANs, which distinguish between the generated and target images, struggle to maintain content and structural consistency between the generated and input images. Therefore, structural alignment between the source and target images is required, necessitating paired datasets. RGAN overcomes this limitation. The generator comprises an encoder, a transformer, and a decoder. The encoder extracts feature vectors from the input image. In this embodiment, the encoder consists of three convolutional neural networks performing convolution, normalization, and activation operations, resulting in an encoder output of 256*64*64. The transformer converts the feature vectors from the source domain into feature vectors from the target domain. In this embodiment, it uses nine residual blocks, which mitigate gradient vanishing and adaptively adjust the network depth. The residual block output is also 256*64*64. The decoder reconstructs low-level features from the feature vectors, consisting of two deconvolutional layers and one convolutional layer, with an output of 3*256*256. The generator uses Tanh activation for the last convolutional layer and ReLU for the rest. The discriminator extracts features from the image and determines whether the features are similar to features in a certain domain. This embodiment uses a five-layer convolutional network to perform convolution, normalization, and activation operations, with LeakyReLU as the activation function.

[0044] In one specific embodiment, synchronization step S1 generates 1200 composite terahertz security inspection images as the source domain image set, denoted as X; 1200 passive terahertz images containing hazardous materials are selected as the target domain image set, denoted as Y; the generator for converting X-domain images to Y-domain images is denoted as G. XY Converting a Y-domain image to an X-domain image is denoted as G. YX The decision maker for identifying images in the X domain is D. X The decision maker for identifying images in the Y domain is D. Y ; A recurrent generative adversarial network simultaneously establishes mappings between X→Y and Y→X;

[0045] In the forward loop, input image X in the X domain, and then process it through G. XY generate Will Input decision device D Y Identify and calculate the adversarial loss, Input generator G YX get X and The content distribution within the loop is aligned to calculate the cycle consistency loss, constraining the output image to have the same content as the input image. In the reverse loop, the input image y in the Y domain is processed by the generator G. YX generate Will Input decision device D X To calculate the losses incurred in adversarial proceedings, Input generator G XY Generation y with is aligned, calculate its cycle consistency loss, training effect reference Figure 4 as shown;

[0046] Wherein, the loss function is composed of an adversarial loss Cycle consistency loss Identity loss; the adversarial loss describes the good and bad of the discrimination result in one-way propagation, using the least square loss, expressed as:

[0047] The training target is:

[0048] Cycle consistency loss Compare the input image and the image generated after passing through the two generators, describe the consistency of the generated image and the original image in content, which is an important point in the cycle generative adversarial network, ensuring that the source domain picture and the synthesized picture are structurally aligned and content similar in training. In this embodiment, L1 loss is used, which helps to restore the low frequency part of the image, expressed as:

[0049]

[0050] Identity loss is added To describe the continuity of the image, so that the generated image is close to the input image, expressed as:

[0051]

[0052] The total loss function is expressed as:

[0053]

[0054] Wherein, is the adversarial loss function in the forward cycle process, is the adversarial loss function in the reverse cycle process, λ cyc , λ idt are proportionality coefficients.

[0055] The optimization target is:

[0056] According to the loss calculation result, the gradient descent method is used to optimize the parameters. In a specific embodiment, the learning rate is initially 0.0002, and after half of the training rounds, it is linearly reduced to 0; wherein, the generator and the discriminator both use the Adam optimizer with good convergence performance, and the two generators are optimized simultaneously, and the discriminator is optimized separately. The optimization strategy, the momentum is β1=0.5, β2=0.999.

[0057] S3. Passive terahertz security inspection images generated based on different types of items are selected and labeled to create a passive terahertz security inspection image dataset;

[0058] In one specific embodiment, 1000 synthetic terahertz security inspection images are generated, divided into four categories: pistols, knives, sharp scissors, and lighters. LabelImg is used for annotation, and a passive terahertz security inspection image dataset is created; it includes the labels and location information of dangerous items in the images.

[0059] S4. Train the improved YOLOv5 target detection network using a passive terahertz security inspection image dataset;

[0060] A YOLOv5 object detection network was built and improved. The YOLOv5 network consists of an input module, backbone, neck, and prediction head. The input module employs data augmentation techniques such as Mosaic, cutout, and copy-paste to perform adaptive anchor box calculation and adaptive image scaling. The backbone uses CSPDarknet, which consists of three CSP modules and one SPP module. (Refer to...) Figure 5 As shown, a CBAM (Convergent Focused Attention) module, which integrates channel attention and spatial attention, is added to the end of the backbone to focus on key positions in the output feature map. The neck adopts a Feature Pyramid Network (FPN) + PAN (Panel Pane) structure. The FPN works from top to bottom, transferring and fusing high-level feature information through upsampling. The PAN adds a bottom-up pyramid after the FPN to perform secondary feature fusion, transferring the localization features from the lower layers to the upper layers. In this embodiment, a more efficient bidirectional feature fusion method is used: a weighted bidirectional feature pyramid network (BiFPN). It uses bidirectional cross-scale connections and weighted feature fusion to achieve bidirectional feature fusion. (Referencing...) Figure 7 The diagram shows the BiFPN structure, which, compared to the FPN+PAN structure, reduces the number of parameters and lowers the computational cost.

[0061] S5. Detect real passive terahertz security inspection images using the trained object detection network. See the test example for reference. Figure 6 As shown, high detection accuracy is obtained.

[0062] Finally, it should be noted that the above only describes the preferred examples of the present application, and is not intended to limit the present application. Although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art will appreciate that modifications can be made to the technical solutions described in the foregoing embodiments, or some of the technical features thereof can be replaced equivalently, without departing from the spirit and principle of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for target detection in passive terahertz security screening images based on modal conversion, characterized in that, Comprise the following steps: S1, the image fusion of the obtained dangerous goods image under X-ray and passive terahertz human body image is carried out; S2, a cycle generative adversarial network is constructed, and the modal style conversion of unpaired data is carried out to generate passive terahertz security inspection image; S3, based on the passive terahertz security inspection images generated by different kinds of goods, the passive terahertz security inspection image dataset is made through selection and marking; S4, the improved YOLOv5 target detection network is trained by using the passive terahertz security inspection image dataset; S5, the real passive terahertz security inspection image is detected through the trained target detection network.

2. The modal conversion based passive terahertz security inspection image target detection method according to claim 1, characterized in that: In step S1, the X-ray dangerous goods image is processed and converted to a gray picture, a mask of the processed image is established, the effective area with goods is extracted, and the random position conforming to the two-dimensional normal distribution is generated according to the position statistics of goods in the real data set, the superimposed area generated by the terahertz human body image is extracted, and the two images are fused according to the mask of the X-ray image: Img add [i,j] = Img1[i,j] * mask[i,j] + Img2[i,j] * (1 - mask[i,j]); wherein Img add is the superimposed image, mask is the mask of the X-ray image, Img1 and Img2 represent the regions to be fused extracted from the X-ray image and the passive terahertz human body image respectively; and Img add covers the corresponding region of the original terahertz human body image.

3. The modal conversion based passive terahertz security inspection image target detection method according to claim 2, characterized in that: The X-ray dangerous goods image is processed: the dangerous goods image is annotated, then the X-ray dangerous goods picture is rotated to obtain the object picture at different angles, and then the picture is scaled according to a certain proportion.

4. The modal conversion based passive terahertz security inspection image target detection method according to claim 2 or 3, characterized in that: For gray-scale image, the contrast enhancement is carried out by histogram equalization, and the enhanced image is: where I is the image gray matrix, I max is the maximum gray level in I, and I min is the minimum gray level in I.

5. The modal conversion based passive terahertz security inspection image target detection method of claim 1, wherein: In step S2, the cycle generative adversarial network includes two generators and two discriminators, wherein the generated passive terahertz security inspection image is denoted as an X domain, a real terahertz image is denoted as a Y domain, a generator for converting an X domain image into a Y domain image is denoted as G XY , a generator for converting a Y domain image into an X domain image is denoted as G YX , a discriminator for discriminating an X domain image is denoted as D X , and a discriminator for discriminating a Y domain image is denoted as D Y ; the cycle generative adversarial network simultaneously establishes X→Y and Y→X mappings.

6. The modal conversion based passive terahertz security inspection image target detection method according to claim 5, characterized in that: The loss function in the cycle generative adversarial network conversion process is: wherein, is the adversarial loss function in the X→Y cycle process, is the adversarial loss function in the Y→X cycle process, λ cyc , λ idt is a proportional coefficient, is the cycle consistency loss function, is the dentity loss; The optimization objective is:

7. The modal conversion based passive terahertz security inspection image target detection method according to claim 5, characterized in that: The generator includes an encoder, a converter and a decoder, wherein the encoder extracts a feature vector from an input image, performs convolution, normalization and activation operations, the converter converts the feature vector of the source domain into the feature vector of the target domain, the decoder restores low-level features from the feature vector and generates an image; the discriminator is used to extract features from the image and judge whether the features are close to the features of a certain domain image.

8. The modal conversion based passive terahertz security inspection image target detection method of claim 1, wherein: In step S3, the passive terahertz security inspection image generated in step S5 is used to continuously expand the number and types of the dataset.

9. The modal conversion based passive terahertz security inspection image target detection method of claim 1, wherein: In step S4, the YOLOv5 target detection network is built and improved, wherein the attention mechanism module CBAM of fusion channel attention and spatial attention is added at the end of the backbone network of the YOLOv5 target detection network, which focuses on the key positions in the output feature map, and the feature extraction module adopts a bidirectional feature fusion method: BiFPN, which realizes bidirectional fusion of features by using bidirectional cross-scale connection and weighted feature fusion.

Images