Dynamic three-dimensional imaging radar foreign matter detection method based on adaptive knowledge distillation

Abstract

The invention relates to a dynamic three-dimensional imaging radar foreign matter detection method based on adaptive knowledge distillation, and the method comprises the steps: carrying out the data collection and preprocessing based on a millimeter wave scanning image; labeling and dividing a data set; the preprocessed data set is sent into the improved detection model for iterative training, loss is calculated through model forward propagation in each iteration, gradient descent is carried out through back propagation, and model training is completed after optimal model parameters are obtained; and performing foreign matter detection on the millimeter wave image to obtain a detection result. According to the embodiment of the invention, new foreign matters can be effectively detected, and meanwhile, good detection performance of old foreign matters can be kept.

Description

Technical Field

[0001] The present disclosure relates to the field of millimeter wave image target detection, and in particular to a dynamic three-dimensional imaging radar foreign body detection method based on adaptive knowledge distillation. Background Art

[0002] Compared with optical images, millimeter wave images have a single channel and contain less feature information. Foreign objects can be easily confused with clothing wrinkles. Traditional deep learning models lack attention to foreground related to foreign objects, which affects the detection accuracy of the model to a certain extent.

[0003] In actual security inspection tasks, some unexpected types of foreign objects often appear, such as a wide variety of contraband. Traditional deep learning models rely on labeled images for training. Directly putting new samples into the trained model will cause the model's performance on the old tasks to drop sharply. Fusion training of new and old samples will greatly increase the time and computational costs. Summary of the invention

[0004] The present disclosure intends to provide a dynamic three-dimensional imaging radar foreign object detection method based on adaptive knowledge distillation. The model can effectively detect new types of foreign objects while maintaining good detection performance for old types of foreign objects.

[0005] According to one of the solutions of the present disclosure, a dynamic three-dimensional imaging radar foreign body detection method based on adaptive knowledge distillation is provided, comprising:

[0006] Based on millimeter wave scanning images, data collection and preprocessing are performed;

[0007] Label and partition datasets;

[0008] The preprocessed data set is sent to the improved detection model for iterative training. In each iteration, the loss is calculated through the model forward propagation, and the gradient is descended through back propagation. After the optimal model parameters are obtained, the model training is completed.

[0009] Perform foreign body detection on the millimeter wave image to obtain the detection result.

[0010] In some embodiments, data collection and preprocessing are performed based on the millimeter wave scanning image, including:

[0011] Through the millimeter wave to be detected object omnidirectional multi-angle imaging processing, each rotation angle α collects β images, complete a complete scanning process will accumulate M = α × β images, the collected images are continuous in time, forming a set of time series, this time series is represented by X = {x1, x2…x t}(t≤T) represents.

[0012] In some embodiments, the model training includes teacher model training, including at least:

[0013] Step S312: Input the dataset image into the Resnet-50 network, pass through a convolution layer with a convolution kernel size of 7×7, then pass through a BN layer for normalization, and finally pass through a maximum pooling layer to form a feature map group P1;

[0014] Step S313: input P1 into a Conv Block to obtain a feature map group P2;

[0015] Step S314: input P2 into an Identity Block to obtain a feature map group P3;

[0016] Step S315: Pass P3 through one Identity Block to obtain feature map group P4, pass P4 through one Conv Block to obtain feature map group P5, pass P5 through three Identity Blocks to obtain feature map group P6, pass P6 through one Conv Block to obtain feature map group P7, pass P7 through five Identity Blocks to obtain feature map group P8, pass P8 through one Conv Block to obtain feature map group P9, pass P9 through two Identity Blocks to obtain feature map group P 10 ;

[0017] Step S316: P 10 After the average pooling layer and the Flatten layer, the feature map group P is obtained by passing through the BN layer and the Softmax layer. 11 ;

[0018] Step S317: P 11 Input into the FPN network, P is connected through top-down and lateral connections. 11 Fusion is performed to obtain a multi-scale feature map group P 12 ;

[0019] Step S318: P 12 Input into the detection head, and design multi-scale anchor boxes. Configure a classification head for each anchor box to obtain the logit value and a regression head to obtain the predicted Top-1 value.

[0020] Step S319: Use the Softmax function to extract the number of pairs C from the classification head T Transformed into a probability distribution, the output probability distribution of the teacher model is expressed as:

[0021]

[0022] Where the subscript T represents the teacher model, and t represents the temperature factor used to soften the output probability distribution;

[0023] Step S3110: Use statistical features to select important responses of the classification head: Calculate the confidence score of each classification score response node obtained by the teacher model and find its average value μ′ c and standard deviation σ′ c , and the elastic threshold T′ is obtained c , traverse all nodes and select the response nodes with confidence scores greater than the threshold as distillation nodes, where T′ c The way to obtain is:

[0024] T′ c =μ′ c +λ1σ′ c (14)

[0025] Finally, we get the category response C' of the teacher model. T ;

[0026] Step S3111: Use the Softmax function to represent each edge of each bounding box e as a probability distribution, thus obtaining a probability matrix of the bounding box B:

[0027] B T =[p1,p2,p3,p4]∈R n×4 (15)

[0028] Step S3112: Use statistical distribution information to select important responses of the regression head: First, select the Top-1 value of each predicted distribution and calculate its average value μ' B and standard deviation σ' B , and the elastic threshold T′ is obtained B :

[0029] T′ B =μ' B +λ2σ' B (16)

[0030] Step S3113: traverse all distributions, select candidate boxes whose confidence scores are greater than the threshold, and calculate the IoU value between the candidate boxes; i represents the score of the i-th candidate box, b M represents the candidate box with the highest probability score, b i represents the i-th candidate box, T′ B represents the screening threshold; IoU(b M ,b i ) represents the intersection-and-union ratio, and the response b M and b i If the overlap between IoU(b M,b i ) is less than T′ B , then keep this candidate box and send it to the next round of comparison. If IoU(b M ,b i ) is greater than or equal to T′ B , then reduce b i Confidence level;

[0031]

[0032] Step S3114: Repeat step S3113 to finally obtain the regression response B of the teacher model from the selected bounding box T .

[0033] In some embodiments, wherein the teacher model training further comprises:

[0034] Step S3115: Modify the training hyperparameters, set learning_rate, batch_size, max_iters, and keep other parameter settings as the network default values. Start training, and after repeating the preset number of rounds, a trained teacher detection network is obtained.

[0035] In some embodiments, the model training includes student model training, including at least:

[0036] Step S322: Obtain feature map group P according to the process from step S311 to step S316 13 , and P 13 The input FPN is fused through top-down and lateral connections to obtain a multi-scale feature map group P 14 ;

[0037] Step S323: Using the parameter-free masking module, define P 13 The high-level feature is F∈R C×H×W , where C, H, and W represent the number of channels, height, and width of F, respectively;

[0038] Step S324: Use k to represent the foreign body target in the input image, and derive its vector representation by averaging the feature vectors of the region of interest of k Use (h,w) to enumerate the H×W spatial domain and a binary mask M k ∈{0,1} H×W To activate the region of interest of k, we get the final expression:

[0039]

[0040] Among them, M k (h,w) is 1 only when the kth foreign object covers (h,w), Nk Represents the total number of eigenvectors in its region of interest:

[0041]

[0042] Step S325: Average the vector representations of all foreign object types and obtain the prototype F of each type of foreign object from each input image. P ∈R C By measuring the similarity between each type of foreign object prototype and all pixels embedded in the original feature F, a pseudo mask M is obtained. IM ∈R H×W ;

[0043] Step S326: Planarize the spatial dimensions of F, that is, reshape F from C×H×W to HW×C;

[0044] Step S327: In each type of foreign body prototype F P Perform matrix multiplication between and F to obtain an H×W similarity map, loop through all types of foreign objects, and combine the similarity map with element addition to obtain an H×W non-standard mask, and perform min-max normalization on this mask to obtain the final pseudo mask M IM ∈[0,1] H×W ;

[0045] Step S328: Using threshold The pseudo mask M IM Segmented into M representing the foreground F and M for background B ,when When , all pixels are embedded as foreground. When M F and M B Acting on P 14 superior;

[0046] Step S329: P 14 Send it to the detection head and design multi-scale anchor frames. Configure a classification head for each anchor frame to obtain the logit value and a regression head to obtain the predicted Top-1 value.

[0047] Step S3210: Use the Softmax function to transform the logit number C S Transformed into a probability distribution, the output probability distribution of the student model is expressed as:

[0048]

[0049] Where the subscript S represents the student model, and t represents the temperature factor, which is used to soften the output probability distribution;

[0050] Step S3211: Use the new foreign object and repeat step S3110 to obtain the category response C' of the teacher model T and the category response C' of the student model S , and C' T Passed to the student model, the incremental distillation loss in the classification head is:

[0051]

[0052] Where m represents one of the category responses;

[0053] Step S3212: Repeat steps S3111 to S3114 to obtain B S , and extract the selected bounding box B from the teacher model T Then use KL-Divergence loss to pass it to the student model S:

[0054]

[0055] Step S3213: The incremental positioning response of the regression head is obtained as:

[0056]

[0057] Step S3214: The total loss of the model obtained from step S312 to step S3213 is:

[0058] L all =L1+α1L cls (C T ,C S )+α2L bbox (B T ,B S ) (twenty four)

[0059] Where α1 and α2 represent the balancing factors that balance the weights of different loss terms.

[0060] In some embodiments, the student model training further comprises:

[0061] Step S3215: Keep the training hyperparameters, learning_rate, batch_size, and max_iters of the student model consistent with those of the teacher model, start training, and after repeating the preset number of rounds, obtain a trained student target detection network.

[0062] In some embodiments, performing foreign body detection on the millimeter wave image to obtain a detection result includes:

[0063] The data set is input into the trained student target detection network, and the detection results are output through the model. The location of the foreign object is framed in the image and formed into a file. The file content contains the category of the foreign object and the coordinate information of the human body where it is located.

[0064] In some embodiments, labeling and dividing the data set includes:

[0065] The collected images are classified according to the types of prohibited items, and each type is represented by Y = {X1, X2…X i}(i≤N), where N represents the number of samples of a single type of contraband, thus forming the total sample data set D={Y1,Y2…T j}(j≤M), where M represents the total number of prohibited items;

[0066] Divide into training set, validation set and test set according to preset ratio;

[0067] Use rectangular boxes to label the training data set.

[0068] In some embodiments, pre-processing comprises:

[0069] Use a mean filter to reduce the noise of the image;

[0070] Use image sharpening to enhance the edge information of the image.

[0071] In some embodiments, the detection is performed based on millimeter wave security inspection equipment, and the foreign matter includes at least prohibited items located in any part of the object to be detected.

[0072] The dynamic three-dimensional imaging radar foreign body detection method based on adaptive knowledge distillation in various embodiments of the present disclosure is based on at least millimeter wave scanning images, data collection and preprocessing are performed; data sets are annotated and divided; the preprocessed data sets are sent to the improved detection model for iterative training, and the loss is calculated by forward propagation of the model in each iteration, and gradient descent is performed by back propagation to complete the training of the model after obtaining the optimal model parameters; foreign body detection is performed on the millimeter wave image to obtain the detection result, so as to address the problems existing in the existing methods in the field of millimeter wave image target detection and utilize the characteristics of knowledge distillation, use Resnet-50 as the backbone network, FPN as the detector of the neck network as the reference network for contraband detection, use a typical teacher-student model and combine the elastic response knowledge distillation selection strategy to enable the model to learn more valuable responses, introduce a parameter-free masking module, and apply the mask generated by it to the multi-scale feature map generated by FPN, so that the student model pays more attention to the prospects related to contraband. The Soft-NMS algorithm is introduced to improve the problem of model mistaken deletion when multiple candidate boxes overlap.

[0073] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In the drawings, which are not necessarily drawn to scale, similar reference numerals in different views may represent similar components. Similar reference numerals with letter suffixes or similar reference numerals with different letter suffixes may represent different instances of similar components. The drawings generally illustrate various embodiments by way of example and not limitation, and together with the description and claims, serve to explain the disclosed embodiments.

[0075] Figure 1 A schematic diagram of the overall structure of model training according to an embodiment of the present disclosure is shown;

[0076] Figure 2 A schematic diagram of the Resnet-50 structure of model training in an embodiment of the present disclosure is shown;

[0077] Figure 3 A schematic diagram of a Conv Block structure diagram of a model training according to an embodiment of the present disclosure is shown;

[0078] Figure 4 A schematic diagram of the Identity Block structure of model training according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution of the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings of the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the described embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0080] Millimeter waves are electromagnetic waves with a wavelength of 1 mm to 10 mm and a frequency range of 26.5 to 300 GHz. They can effectively detect all kinds of contraband hidden under human clothing without direct contact with the human body. Millimeter waves have no safety impact on the human body, so they are widely used in human security inspection tasks in public places such as airports and railway stations.

[0081] At present, methods based on deep learning have become the mainstream in the field of target detection in millimeter wave images. Algorithms based on deep learning build deep network structures and train network learning to extract higher-dimensional feature information. They mainly include anchor-based methods and anchor-free methods. Anchor-based methods are divided into single-stage algorithms and two-stage algorithms. Single-stage algorithms use the idea of ​​regression to directly predict classification confidence and position coordinates. The most representative ones are the YOLO series of algorithms and the SSD algorithm. The two-stage algorithm first generates a series of region candidate boxes from the input image, and then generates the predicted classification and position of the final target from these candidate boxes. The main representatives are Fast-RCNN and Faster-RCNN. The anchor-free method removes the anchor boxes containing prior knowledge, thereby reducing the number of network hyperparameters and directly detecting the corners or centers of the target border. The main representatives are Transfomer, etc.

[0082] In the actual task of target detection, some new samples often appear, which requires the model to be updated. Recently, incremental learning methods have been gradually applied to the field of target detection. Incremental learning methods can continuously learn new knowledge from data streams while maintaining old knowledge. At present, incremental learning methods are roughly divided into three types: regularization-based, knowledge distillation-based, and knowledge replay-based. Regularization-based methods prevent old parameters from changing significantly during incremental learning by imposing constraints on important parameters. Knowledge distillation-based methods use teacher models and student models to perform distillation in the teacher model for training old tasks, thereby compressing knowledge into the student model. After the student model inherits the parameters of the teacher model, it can be updated on new tasks. The algorithm based on knowledge replay uses some of the retained old category samples to train the model together with the new category samples.

[0083] In combination with the above-mentioned background technology, the present disclosure records the corresponding solutions in the form of embodiments to solve the defects in the prior art, but it is not intended to limit the scope of protection of the patent right claimed by the present disclosure.

[0084] As one of the solutions, the embodiment of the present disclosure provides a dynamic three-dimensional imaging radar foreign body detection method based on adaptive knowledge distillation, including:

[0085] Based on millimeter wave scanning images, data collection and preprocessing are performed;

[0086] Label and partition datasets;

[0087] The preprocessed data set is sent to the improved detection model for iterative training. In each iteration, the loss is calculated through the model forward propagation, and the gradient is descended through back propagation. After the optimal model parameters are obtained, the model training is completed.

[0088] Perform foreign body detection on the millimeter wave image to obtain the detection result.

[0089] In view of the above contents, the present disclosure at least aims to propose a dynamic three-dimensional imaging radar foreign body detection method based on adaptive knowledge distillation. In view of the problems existing in the prior art, the characteristics of knowledge distillation are utilized, Resnet-50 is used as the backbone network, FPN is used as the detector of the neck network as the reference network for contraband detection, and a typical teacher-student model is used in combination with the elastic response knowledge distillation selection strategy to enable the model to learn more valuable responses. A parameter-free masking module is introduced, and the mask generated by it is applied to the multi-scale feature map generated by FPN, so that the student model pays more attention to the prospects related to contraband. The Soft-NMS algorithm is introduced to improve the problem of model mistaken deletion when multiple candidate boxes overlap.

[0090] The overall structure of the embodiment of the present disclosure is as follows: Figure 1 The overall structural diagram of the model training of an embodiment of the present disclosure is shown. By way of example, the following takes steps S1 to S3 as an example, takes millimeter wave security inspection equipment, and foreign objects at least including prohibited items located at any part of the object to be detected, such as the human body as an example, to further illustrate the dynamic three-dimensional imaging radar foreign object detection method based on adaptive knowledge distillation of the present disclosure.

[0091] S1: Data collection and preprocessing.

[0092] S11: Randomly place the prohibited items on any part of the body, and use the millimeter wave body security inspection equipment to scan the person carrying the prohibited items; the millimeter wave equipment collects images of the front and back sides of the human body, and a total of two images are collected in one scan.

[0093] S12: The millimeter wave security inspection equipment performs all-round and multi-angle imaging processing on the human body. Each time it rotates by an angle of α, β images are collected. After completing a complete scanning process, a total of M = α × β images are collected. The collected images are continuous in time and constitute a set of time series. This time series is represented by X = {x1, x2…x t}(t≤T) represents.

[0094] S13: Preprocessing the collected data image, the preprocessing process includes: using a mean filter to reduce noise on the image, using an image sharpening operation to enhance edge information of the image to make the image clearer.

[0095] In some specific embodiments, the present disclosure may include labeling and dividing a data set, including:

[0096] The collected images are classified according to the types of prohibited items, and each type is represented by Y={X1,X2…X i}(i≤N), where N represents the number of samples of a single type of contraband, thus forming the total sample data set D={Y1,Y2…T j}(j≤M), where M represents the total number of prohibited items;

[0097] Divide into training set, validation set and test set according to preset ratio;

[0098] Use rectangular boxes to label the training data set.

[0099] Step S2: Label and divide the dataset.

[0100] Step S21: Classify the collected images according to the types of prohibited items, and use Y = {X1, X2...X i}(i≤N), where N represents the number of samples of a single type of contraband, thus forming the total sample data set D={Y1,Y2…T j}(j≤M), where M represents the total number of prohibited items.

[0101] Step S22: select class a samples in the total data set as initial contraband, class b samples as incremental contraband, a+b=D, and divide them into training set, validation set and test set in a ratio of 8:1:1.

[0102] Step S23: Manually label the training data set using a rectangular frame. The specific steps are as follows:

[0103] Step S231: input image;

[0104] Step S232: If the image contains contraband, proceed to step S233, otherwise return to step S321;

[0105] Step S233: Determine the type and location of the prohibited items. Each type of prohibited item is marked with a different label box and name. The label name is expressed as: label = {l1, l2…l i}(i≤M).

[0106] Step S24: Repeat the above steps until all images in the data set are labeled, and then input the data set into the model for training.

[0107] Step S3: Model training. The preprocessed data set is sent to the improved contraband detection model for iterative training. In each iteration, the loss is calculated through forward propagation of the model, and the gradient is descended through back propagation. After the optimal model parameters are obtained, the model training is completed.

[0108] Step S31: Teacher model training.

[0109] Step S311: setting initialization parameters, the number of prohibited goods categories nc and the names of prohibited goods categories classes;

[0110] Step S312: Figure 2 As shown in the figure, the dataset image is input into the Resnet-50 network, passes through a convolution layer with a convolution kernel size of 7×7, and then passes through a BN layer for normalization. Finally, it passes through a maximum pooling layer to form a feature map group P1.

[0111] Step S313: Input P1 into a Conv Block to obtain a feature map group P2. The Conv Block structure is shown in the figure Figure 3 As shown, the specific steps are:

[0112] Step S3131: P1 passes through a convolution layer with a convolution kernel size of 1×1 and a BN layer, and then passes through a ReLu activation function to obtain a feature map group P 1_1 ;

[0113] Step S3132: P 1_1 Input the convolution layer and BN layer with a convolution kernel size of 3×3, and then pass the ReLu activation function to obtain the feature map group P 1_2 ;

[0114] Step S3133: P 1_2 Input the convolution layer with a convolution kernel size of 3×3 and the BN layer to obtain the feature map group P 1_3 ;

[0115] Step S3134: P1 is passed through a convolution layer with a convolution kernel size of 1×1 and a BN layer to obtain a feature map group P 1_4 ;

[0116] Step S3135: P 1_3 and P 1_4 After accumulation, the ReLu activation function is used to obtain P2;

[0117] Step S314: Input P2 into an Identity Block to obtain a feature map group P3. The Identity Block structure diagram is as follows: Figure 4 As shown, the specific steps are:

[0118] Step S3141: P2 passes through a convolution layer with a convolution kernel size of 1×1 and a BN layer, and then passes through a ReLu activation function to obtain a feature map group P 2_1 ;

[0119] Step S3142: P 2_1 After the convolution layer with a convolution kernel size of 3×3 and the BN layer, the feature map group P is obtained by the ReLu activation function. 2_2 ;

[0120] Step S3143: P 2_2 After the convolution layer with a convolution kernel size of 3×3 and the BN layer, the feature map group P is obtained. 2_3 ;

[0121] Step S3145: P2 and P 2_3 After accumulation, the ReLu activation function is used to obtain the feature map group P3;

[0122] Step S315: Pass P3 through one Identity Block to obtain feature map group P4, pass P4 through one Conv Block to obtain feature map group P5, pass P5 through three Identity Blocks to obtain feature map group P6, pass P6 through one Conv Block to obtain feature map group P7, pass P7 through five Identity Blocks to obtain feature map group P8, pass P8 through one Conv Block to obtain feature map group P9, pass P9 through two Identity Blocks to obtain feature map group P 10 ;

[0123] Step S316: P 10 After the average pooling layer and the Flatten layer, the feature map group P is obtained by passing through the BN layer and the Softmax layer. 11 ;

[0124] Step S317: P 11 Input into the FPN network, P is connected through top-down and lateral connections. 11 Fusion is performed to obtain a multi-scale feature map group P 12 ;

[0125] Step S318: P 12 Input into the detection head, and design multi-scale anchor boxes. Configure a classification head for each anchor box to obtain the logit value and a regression head to obtain the predicted Top-1 value.

[0126] Step S319: Use the Softmax function to extract the number of pairs C from the classification head T Transformed into a probability distribution, the output probability distribution of the teacher model is expressed as:

[0127]

[0128] Where the subscript T represents the teacher model, and t represents the temperature factor used to soften the output probability distribution;

[0129] Step S3110: Select important responses of the classification head using statistical features: Calculate the confidence score of each classification score response node obtained by the teacher model and find its average value μ' c and standard deviation σ′c , and the elastic threshold T′ is obtained c , traverse all nodes and select the response nodes with confidence scores greater than the threshold as distillation nodes, where T′ v The way to obtain is:

[0130] T′ c =μ′ c +λ1σ′ c (26)

[0131] Finally, we get the category response C' of the teacher model. T ;

[0132] Step S3111: Use the Softmax function to represent each edge of each bounding box e as a probability distribution, thus obtaining a probability matrix of the bounding box B:

[0133] B T =[p1,p2,p3,p4]∈R n×4 (27)

[0134] Step S3112: Use statistical distribution information to select important responses of the regression head: First, select the Top-1 value of each prediction distribution and calculate its average value μ' B and standard deviation σ' B , and the elastic threshold T′ is obtained B :

[0135] T′ B =μ' B +λ2σ' B (28)

[0136] Step S3113: traverse all distributions, select candidate boxes whose confidence scores are greater than the threshold, and calculate the IoU value between the candidate boxes; i represents the score of the i-th candidate box, b M represents the candidate box with the highest probability score, b i represents the i-th candidate box, T′ B represents the screening threshold; IoU(b M ,b i ) represents the intersection-and-union ratio, and the response b M and b i If the overlap between IoU(b M ,b i ) is less than T′ B , then keep this candidate box and send it to the next round of comparison. If IoU(b M ,b i ) is greater than or equal to T′ B , then reduce b i confidence level;

[0137]

[0138] Step S3114: Repeat step S3113 to finally obtain the regression response B of the teacher model from the selected bounding box T ;

[0139] Step S3115: Modify the training hyperparameters, set learning_rate, batch_size, max_iters, and keep other parameter settings as the network default values. Start training, and after repeating the preset number of rounds, a trained teacher detection network is obtained.

[0140] Step S32: training the student model.

[0141] Step S321: the student model also uses Resnet-50 as the backbone network and FPN as the detector of the neck network;

[0142] Step S322: Repeat steps S311 to S316 to obtain feature map group P 13 , and P 13 The input FPN is fused through top-down and lateral connections to obtain a multi-scale feature map group P 14 ;

[0143] Step S323: Using the parameter-free masking module, define P 13 The high-level feature is F∈R C×H×W , where C, H, and W represent the number of channels, height, and width of F, respectively;

[0144] Step S324: Use k to represent the contraband target in the input image, and derive its vector representation by averaging the feature vectors of the region of interest of k Use (h,w) to enumerate the H×W spatial domain and a binary mask M k ∈{0,1} H×W To activate the region of interest of k, we get the final expression:

[0145]

[0146] Among them, M k (h,w) is 1 only when the kth contraband covers (h,w), N k Represents the total number of eigenvectors in its region of interest:

[0147]

[0148] Step S325: Average the vector representations of all prohibited items and obtain the prototype F of each type of prohibited item from each input image. P ∈R C , by measuring the similarity between each type of contraband prototype and all pixels embedded in the original feature F, a pseudo mask M is obtained. IM ∈R H×W , the specific steps are:

[0149] Step S326: Planarize the spatial dimensions of F, that is, reshape F from C×H×W to HW×C;

[0150] Step S327: In each type of contraband prototype F P Perform matrix multiplication between and F to obtain an H×W similarity map, loop through all types of contraband, and combine their similarity maps with element addition to obtain an H×W non-standard mask, and perform min-max normalization on this mask to obtain the final pseudo mask M IM ∈[0,1] H×W ;

[0151] Step S328: Using threshold The pseudo mask M IM Segmented into M representing the foreground F and M for background B ,when When , all pixels are embedded as foreground. When M F and M B Acting on P 14 superior;

[0152] Step S329: P 14 Send it to the detection head and design multi-scale anchor frames. Configure a classification head for each anchor frame to obtain the logit value and a regression head to obtain the predicted Top-1 value.

[0153] Step S3210: Use the Softmax function to transform the logit number C S Transformed into a probability distribution, the output probability distribution of the student model is expressed as:

[0154]

[0155] Where the subscript S represents the student model, and t represents the temperature factor, which is used to soften the output probability distribution;

[0156] Step S3211: Use the new class of contraband and repeat step S3110 to obtain the category response C' of the teacher model T and the category response C' of the student model S , and C'T Passed to the student model, the final incremental distillation loss in the classification head is:

[0157]

[0158] Where m represents one of the category responses;

[0159] Step S3212: Repeat steps S3111 to S3114 to obtain B S , and extract the selected bounding box B from the teacher model T T Then use KL-Divergence loss to pass it to the student model S:

[0160]

[0161] Step S3213: The incremental positioning response of the regression head is finally obtained as:

[0162]

[0163] Step S3214: The total loss of the model obtained from steps S312-S3213 is:

[0164] L all =L1+α1L cls (C T ,C S )+α2L bbox (B T ,B S ) (36)

[0165] Where α1 and α2 represent the balancing factors for balancing the weights of different loss terms;

[0166] Step S3215: Keep the training hyperparameters, learning_rate, batch_size, and max_iters of the student model consistent with those of the teacher model, start training, and after repeating the preset number of rounds, obtain a trained student target detection network.

[0167] Step S33: performing contraband detection on the millimeter wave human body security inspection image.

[0168] Step S331: The pre-processed image is made into a data set according to step S2 to form a millimeter wave human body security inspection image data set;

[0169] Step S332: Input the untrained data set into the trained student target detection network, and finally output the detection result through the model, frame the location of the prohibited items in the image, and form a file. The file content includes the category of the prohibited items and the coordinate information of the human body position.

[0170] The dynamic three-dimensional imaging radar foreign body detection method based on adaptive knowledge distillation in each embodiment of the present disclosure has the following beneficial effects compared with the prior art:

[0171] 1. The teacher-student model is adopted, and the elastic selection strategy is used to retain and transfer the most valuable response knowledge, effectively alleviating the "catastrophic forgetting" problem, so that the model can effectively detect new types of contraband while maintaining good detection performance for old types of contraband;

[0172] 2. Introduce the Soft-NMS algorithm to remove candidate boxes with high overlap by penalizing the confidence of the candidate boxes, thereby improving the detection performance when multiple contraband items overlap;

[0173] 3. Combined with the parameter-free masking module, the foreground related to contraband and the background unrelated to contraband are decoupled, so that the model pays more attention to the foreground area and improves the detection performance of the model.

[0174] In some specific embodiments, the dynamic three-dimensional imaging radar foreign object detection device based on adaptive knowledge distillation disclosed in the present invention includes one or more processing modules configured to execute the specific implementation methods of steps S1 to S3 in the above text.

[0175] Specifically, one of the inventive concepts disclosed herein is to collect and preprocess data based on at least millimeter wave scanning images; to annotate and divide data sets; to send the preprocessed data sets into the improved detection model for iterative training, to calculate the loss through forward propagation of the model in each iteration, and to perform gradient descent through back propagation to complete the training of the model after obtaining the optimal model parameters; to detect foreign objects in millimeter wave images and obtain detection results, thereby targeting the problems existing in the existing methods in the field of target detection in millimeter wave images and utilizing the characteristics of knowledge distillation, using Resnet-50 as the backbone network, FPN as the detector of the neck network as the reference network for contraband detection, using a typical teacher-student model and combining the elastic response knowledge distillation selection strategy, so that the model learns more valuable responses, introducing a parameter-free masking module, and applying the mask generated by it to the multi-scale feature map generated by FPN, so that the student model pays more attention to the prospects related to contraband. The Soft-NMS algorithm is introduced to improve the problem of model mistaken deletion when multiple candidate boxes overlap.

[0176] The present disclosure also provides a computer-readable storage medium having computer executable instructions stored thereon. When the computer executable instructions are executed by a processor, the method for detecting foreign objects by a dynamic three-dimensional imaging radar based on adaptive knowledge distillation is mainly implemented, which at least includes:

[0177] Based on millimeter wave scanning images, data collection and preprocessing are performed;

[0178] Label and partition datasets;

[0179] The preprocessed data set is sent to the improved detection model for iterative training. In each iteration, the loss is calculated through the model forward propagation, and the gradient is descended through back propagation. After the optimal model parameters are obtained, the model training is completed.

[0180] Perform foreign body detection on the millimeter wave image to obtain the detection result.

[0181] The above embodiments are only exemplary embodiments of the present disclosure and are not intended to limit the present disclosure. The protection scope of the present disclosure is defined by the claims. Those skilled in the art may make various modifications or equivalent substitutions to the present disclosure within the essence and protection scope of the present disclosure, and such modifications or equivalent substitutions shall also be deemed to fall within the protection scope of the present disclosure.

Claims

1. A dynamic 3D imaging radar foreign body detection method based on adaptive knowledge distillation, including: Based on millimeter wave scanning images, data collection and preprocessing are performed; Label and partition datasets; The preprocessed data set is sent to the improved detection model for iterative training. In each iteration, the loss is calculated through the model forward propagation, and the gradient is descended through back propagation. After the optimal model parameters are obtained, the model training is completed. Perform foreign body detection on the millimeter wave image to obtain the detection result.

2. The method according to claim 1, wherein: Based on the millimeter wave scanning image, data acquisition and preprocessing are performed, including: Through the millimeter wave to be detected object omnidirectional multi-angle imaging processing, each rotation angle α collects β images, complete a complete scanning process will accumulate M = α × β images, the collected images are continuous in time, forming a set of time series, this time series is represented by X = {x1, x2…x t }(t≤T) represents.

3. The method according to claim 2, wherein: Model training includes teacher model training, which at least includes: Step S312: Input the dataset image into the Resnet-50 network, pass through a convolution layer with a convolution kernel size of 7×7, then pass through a BN layer for normalization, and finally pass through a maximum pooling layer to form a feature map group P1; Step S313: input P1 into a Conv Block to obtain a feature map group P2; Step S314: input P2 into an Identity Block to obtain a feature map group P3; Step S315: P3 passes through one Identity Block to obtain feature map group P4, P4 passes through one Conv Block to obtain feature map group P5, P5 passes through three Identity Blocks to obtain feature map group P6, P6 passes through one Conv Block to obtain feature map group P7, P7 passes through five Identity Blocks to obtain feature map group P8, P8 passes through one Conv Block to obtain feature map group P9, P9 passes through two Identity Blocks to obtain feature map group P 10 ; Step S316: P 10 After the average pooling layer and the Flatten layer, the feature map group P is obtained by passing through the BN layer and the Softmax layer. 11 ; Step S317: P 11 Input into the FPN network, P is connected through top-down and lateral connections. 11 Fusion is performed to obtain a multi-scale feature map group P 12 ; Step S318: P 12 Input into the detection head, and design multi-scale anchor boxes. Configure a classification head for each anchor box to obtain the logit value and a regression head to obtain the predicted Top-1 value. Step S319: Use the Softmax function to convert the number of pairs C obtained from the classification head T Transformed into a probability distribution, the output probability distribution of the teacher model is expressed as: Where the subscript T represents the teacher model, and t represents the temperature factor used to soften the output probability distribution; Step S3110: Select important responses of the classification head using statistical features: Calculate the confidence score of each classification score response node obtained by the teacher model and find its average value μ c ' and standard deviation σ c ', and the elastic threshold T is obtained c ', traverse all nodes and select the response nodes with confidence scores greater than the threshold as distillation nodes, where T c 'The way to obtain it is: T c '=μ c '+λ1σ c ' (2) Finally, we get the category response C' of the teacher model. T ; Step S3111: Use the Softmax function to represent each edge of each bounding box e as a probability distribution, thus obtaining a probability matrix of the bounding box B: B T =[p1,p2,p3,p4]∈R n×4 (3) Step S3112: Use statistical distribution information to select important responses of the regression head: First, select the Top-1 value of each prediction distribution and calculate its average value μ' B and standard deviation σ' B , and the elastic threshold T′ is obtained B : T B '=μ' B +λ2σ' B (4) Step S3113: traverse all distributions, select candidate boxes whose confidence scores are greater than the threshold, and calculate the IoU value between the candidate boxes; i represents the score of the i-th candidate box, b M represents the candidate box with the highest probability score, b i represents the i-th candidate box, T′ B represents the screening threshold; IoU(b M ,b i ) represents the intersection-and-union ratio, and the response b M and b i If the overlap between IoU(b M ,b i ) is less than T′ B , then keep this candidate box and send it to the next round of comparison. If IoU(b M ,b i ) is greater than or equal to T′ B , then reduce b i Confidence level; Step S3114: Repeat step S3113 to finally obtain the regression response B of the teacher model from the selected bounding box T .

4. The method according to claim 3, wherein: Teacher model training also includes: Step S3115: Modify the training hyperparameters, set learning_rate, batch_size, max_iters, and keep other parameter settings as the network default values. Start training, and after repeating the preset number of rounds, a trained teacher detection network is obtained.

5. The method according to claim 4, wherein: Model training includes student model training, which at least includes: Step S322: Obtain feature map group P according to the process from step S311 to step S316 13 , and P 13 The input FPN is fused through top-down and lateral connections to obtain a multi-scale feature map group P 14 ; Step S323: Using the parameter-free masking module, define P 13 The high-level feature is F∈R C×H×W , where C, H, and W represent the number of channels, height, and width of F, respectively; Step S324: Use k to represent the foreign body target in the input image, and derive its vector representation by averaging the feature vectors of the region of interest of k Use (h,w) to enumerate the H×W spatial domain and a binary mask M k ∈{0,1} H×W To activate the region of interest of k, we get the final expression: Among them, M k (h,w) is 1 only when the kth foreign object covers (h,w), N k Represents the total number of eigenvectors in its region of interest: Step S325: Average the vector representations of all foreign object types and obtain the prototype F of each type of foreign object from each input image. P ∈R C By measuring the similarity between each type of foreign object prototype and all pixels embedded in the original feature F, a pseudo mask M is obtained. IM ∈R H×W ; Step S326: Planarize the spatial dimensions of F, that is, reshape F from C×H×W to HW×C; Step S327: In each type of foreign body prototype F P Perform matrix multiplication between and F to obtain an H×W similarity map, loop through all types of foreign objects, and combine the similarity map with element addition to obtain an H×W non-standard mask, and perform min-max normalization on this mask to obtain the final pseudo mask M IM ∈[0,1] H×W ; Step S328: Using threshold The pseudo mask M IM Segmented into M representing the foreground F and M for background B ,when When , all pixels are embedded as foreground. When M F and M B Acting on P 14 superior; Step S329: P 14 Send it to the detection head and design multi-scale anchor frames. Configure a classification head for each anchor frame to obtain the logit value and a regression head to obtain the predicted Top-1 value. Step S3210: Use the Softmax function to transform the logit number C S Transformed into a probability distribution, the output probability distribution of the student model is expressed as: Where the subscript S represents the student model, and t represents the temperature factor, which is used to soften the output probability distribution; Step S3211: Use the new foreign object and repeat step S3110 to obtain the category response C' of the teacher model T and the category response C' of the student model S , and C' T Passed to the student model, the incremental distillation loss in the classification head is: Where m represents one of the category responses; Step S3212: Repeat steps S3111 to S3114 to obtain B S , and extract the selected bounding box B from the teacher model T Then use KL-Divergence loss to pass it to the student model S: Step S3213: The incremental positioning response of the regression head is obtained as: Step S3214: The total loss of the model obtained from step S312 to step S3213 is: L all L1+α1L cls (C T ,C S )+α2L bbox (B T ,B S ) (12) Where α1 and α2 represent the balancing factors that balance the weights of different loss terms.

6. The method according to claim 5, wherein: Student model training also includes: Step S3215: Keep the training hyperparameters, learning_rate, batch_size, and max_iters of the student model consistent with those of the teacher model, start training, and after repeating the preset number of rounds, obtain a trained student target detection network.

7. The method according to claim 6, wherein: Perform foreign body detection on millimeter wave images and obtain detection results, including: The data set is input into the trained student target detection network, and the detection results are output through the model. The location of the foreign object is framed in the image and formed into a file. The file content contains the category of the foreign object and the coordinate information of the human body where it is located.

8. The method according to claim 1, wherein: Label and partition datasets, including: The collected images are classified according to the types of prohibited items, and each type is represented by Y={X1,X2…X i }(i≤N), where N represents the number of samples of a single type of contraband, thus forming the total sample data set D={Y1,Y2…T j }(j≤M), where M represents the total number of prohibited items; Divide into training set, validation set and test set according to preset ratio; Use rectangular boxes to label the training data set.

9. The method according to claim 1, wherein the pre-processing comprises: Use a mean filter to reduce the noise of the image; Use image sharpening to enhance the edge information of the image.

10. The method according to claim 1, wherein the detection is performed based on millimeter wave security inspection equipment, and the foreign matter at least includes prohibited items located at any part of the object to be detected.

Images