Contraband detection method based on multi-view images and deep learning

Abstract

The present invention provides a multi-view image contraband detection method based on NeRF and deep learning. The method comprises the following steps: first, collecting 2D security inspection images of an object to be detected at different viewing angles; inputting the multi-view images into a 2D feature extraction network for feature extraction to obtain 2D image features; inputting the multi-view images into a NeRF neural radiation field for 3D scene rendering, and mapping the 2D image features to the rendered 3D scene to obtain 3D voxel features; inputting the 3D voxel features into a 3D target detection network for target detection to obtain 3D target detection results; and projecting the 3D target detection results onto a 2D image plane to obtain 2D target detection results, thereby completing contraband detection. The method can reduce the effects of occlusion and viewing angle deviation, improve the accuracy of contraband recognition, and simultaneously enhance the detection capability of small contraband. Furthermore, the method has greater adaptability and robustness to changes in external conditions.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and image recognition, and more specifically, to a multi-view image contraband detection method based on NeRF and deep learning. Background Art

[0002] Airports are crucial hubs in the air transport industry, and their safety is directly linked to the safety and efficiency of air transport. Detecting prohibited items is crucial in air logistics, as they encompass a wide range of items, including but not limited to flammable and explosive items and sharp objects.

[0003] Currently, mainstream methods for detecting prohibited items include X-ray imaging, metal detection, millimeter-wave scanning, chemical detection, and deep learning. X-ray imaging can penetrate packaging and reveal the internal structure of items, effectively detecting metal and other high-density items. However, it is less effective for detecting non-metallic prohibited items, such as liquids and powders. Metal detection can effectively identify metal objects, such as guns and knives, but is unable to detect non-metallic prohibited items and is easily affected by the surrounding environment, resulting in a high false alarm rate. Millimeter-wave scanning can detect non-metallic items, including liquids and explosives. However, the equipment is expensive and the detection speed is relatively slow, which affects traffic efficiency. Chemical detection offers high sensitivity and can detect trace traces of specific chemicals, making it suitable for explosives detection. However, it cannot provide information on the shape or size of an object, has a limited detection range, and can only identify specific chemicals. Deep learning methods, using technologies such as deep learning, can identify a wide range of prohibited items. However, they can experience false positives in complex environments and are ineffective for heavily obscured objects.

[0004] With the rapid development of artificial intelligence (AI) technology, computer vision-based automatic contraband detection has become a research hotspot. Through intelligent analysis of surveillance video or image data, this technology can automatically identify and locate contraband, improving detection efficiency and accuracy. However, existing 2D image processing technologies based on a single or limited perspective still face challenges in complex environments. Issues such as occlusion by contraband, image blur, and lighting variations can all affect the accuracy of detection results. Furthermore, traditional methods struggle to accurately detect contraband that is small or whose shape and material resemble those of the surrounding environment.

[0005] Existing patent documents disclose a method, device, and medium for detecting contraband in security inspection images based on machine learning. The method includes: obtaining a preset number of X-ray security inspection images to form a dataset; the dataset includes X-ray security inspection images of various contraband items; inputting the dataset into a contraband segmentation model for model training to obtain a trained contraband segmentation model; the contraband segmentation model includes an encoder network, a global multi-scale module, and a decoder network; inputting the security inspection image to be detected into the trained contraband segmentation model to obtain a contraband detection result; the contraband detection result includes the contraband segmentation result. However, this prior art solution still relies on 2D images with a single or limited perspective for detection and still faces challenges in complex environments. Issues such as contraband occlusion, image blur, and lighting variations can affect the accuracy of detection results. Furthermore, this solution is particularly difficult to accurately detect contraband that is small in size or has a shape or material similar to the surrounding environment. Summary of the Invention

[0006] In order to overcome the defects of the above-mentioned existing technologies that detection is based only on images from a single perspective or a specific perspective, which have low detection accuracy in complex environments and are difficult to detect small contraband, the present invention provides a multi-perspective image contraband detection method based on NeRF and deep learning, which can effectively reduce the impact of occlusion and perspective deviation, improve the accuracy of contraband identification; at the same time, it also enhances the detection capability of small contraband; in addition, the present invention also has stronger adaptability and robustness to changes in external conditions.

[0007] In order to solve the above technical problems, the technical solutions of the present invention are as follows:

[0008] A multi-view image contraband detection method based on NeRF and deep learning, including:

[0009] S101: Using an X-ray machine to collect 2D security inspection images of the object to be inspected at different viewing angles to obtain multi-view images of the object to be inspected;

[0010] S102: Inputting the multi-view images of the object to be detected into a preset 2D feature extraction network to perform feature extraction to obtain 2D image features of the object to be detected;

[0011] S103: Inputting the multi-view images of the object to be detected into the NeRF neural radiation field for 3D scene rendering, and mapping the 2D image features of the object to be detected to the rendered 3D scene to obtain 3D voxel features of the object to be detected;

[0012] S104: Inputting the 3D voxel features of the object to be detected into a preset 3D object detection network to perform object detection and obtain a 3D object detection result;

[0013] S105: Project the 3D target detection result onto a 2D image plane to obtain a 2D target detection result, thereby completing contraband detection.

[0014] Preferably, in step S102, the preset 2D feature extraction network is specifically a ResNet neural network model;

[0015] The ResNet neural network model includes a ResNet50 deep residual network.

[0016] Preferably, the step S103 includes:

[0017] Inputting the multi-view images of the object to be detected into the NeRF neural radiation field, establishing a 3D coordinate system containing a number of voxels, and performing 3D scene rendering;

[0018] Get the camera intrinsic parameter matrix corresponding to the i-th perspective and the external parameter matrix , where i=1,2,3,…,T, T is the total number of viewing angles;

[0019] For the center point of each voxel of the object to be detected in the 3D coordinate system ,point 2D pixel coordinates under the i-th viewing angle projection for:

[0020]

[0021]

[0022] in, is the 2D pixel coordinate of the corresponding position in the 2D security inspection image of the i-th perspective of the object to be detected; is the 2D image feature corresponding to the 2D security inspection image of the object to be detected at the i-th perspective; is the camera intrinsic parameter matrix after scaling ;

[0023] point 3D projection features at the i-th view for:

[0024]

[0025] in, Indicates that the nearest neighbor difference method is used to find the position Search characteristics;

[0026] For points projected outside the boundaries of the 2D plane or behind the 2D plane , the projection under this perspective is invalid, set its ; Projections from other perspectives are all valid projections;

[0027] Get Point The 3D projection features at all viewing angles are aggregated using the weighted average method to obtain the points 3D average projection features :

[0028]

[0029] in, is the number of viewing angles of effective projection;

[0030] The 3D average projection features of each voxel of the object to be detected in the 3D coordinate system are obtained. After the 3D average projection features of all voxels are processed by G-MLP shared with the NeRF neural radiation field, they are fused with the rendered 3D scene to obtain the 3D voxel features of the object to be detected.

[0031] Preferably, in step S104, the preset 3D object detection network is specifically an ImVoxelNet neural network model.

[0032] Preferably, the 3D object detection result in step S104 includes: a 3D bounding box of the object to be detected and a contraband identification result;

[0033] The prohibited items identification result includes any one of: no prohibited items, controlled device prohibited items, fireworks prohibited items, dangerous goods prohibited items, fire prohibited items and battery prohibited items.

[0034] Preferably, the steps of obtaining the 2D feature extraction network preset in step S102 and the 3D object detection network preset in step S104 include:

[0035] S201: Using an X-ray machine to collect 2D images of several types of contraband at different viewing angles, obtaining a multi-view contraband image dataset and performing preprocessing;

[0036] S202: Establishing a 2D feature extraction network and a 3D object detection network;

[0037] S203: Inputting the preprocessed multi-view contraband image dataset into a 2D feature extraction network for feature extraction to obtain 2D image features corresponding to each contraband;

[0038] S204: Inputting the preprocessed multi-view contraband image dataset into the NeRF neural radiation field for 3D scene rendering, and mapping the 2D image features corresponding to each contraband item to the rendered 3D scene to obtain the 3D voxel features corresponding to each contraband item;

[0039] S205: Inputting the 3D voxel features corresponding to each prohibited item into a 3D object detection network to perform object detection, and obtaining a 3D detection result corresponding to each prohibited item;

[0040] S206: Projecting the 3D detection results corresponding to each prohibited item onto a 2D image plane to obtain the 2D detection results corresponding to each prohibited item;

[0041] S207: Setting a total loss function and calculating a total loss function value based on the 2D detection results corresponding to each contraband item;

[0042] S208: Determine whether the total loss function value meets the preset conditions. If not, update the parameters of the 2D feature extraction network and the 3D object detection network, and repeat steps S203 to S208. If so, obtain the trained 2D feature extraction network and the trained 3D object detection network, and save them as the preset 2D feature extraction network and the preset 3D object detection network respectively.

[0043] Preferably, the preprocessing in step S201 includes:

[0044] Performing data augmentation and normalization operations on the multi-view contraband image dataset in sequence to complete preprocessing;

[0045] The data enhancement operation includes any one or more of flip transformation, rotation transformation, crop transformation and scaling transformation.

[0046] Preferably, in step S207, the total loss function Specifically:

[0047]

[0048] in, is the classification loss function; is the total prediction error loss function; is the bounding box loss function; 、 and are the first, second and third hyperparameters respectively.

[0049] Preferably, after step S105, the following steps are further included:

[0050] Post-processing the 2D target detection results using a non-maximum suppression (NMS) algorithm;

[0051] After step S206, the following steps are further included:

[0052] The non-maximum suppression (NMS) algorithm is used to post-process the 2D detection results corresponding to each prohibited item.

[0053] The present invention also provides a multi-view image contraband detection system based on NeRF and deep learning, which applies the multi-view image contraband detection method based on NeRF and deep learning, including:

[0054] Multi-view image acquisition unit: used to use an X-ray machine to collect 2D security inspection images of the object to be inspected at different viewing angles, and obtain multi-view images of the object to be inspected;

[0055] 2D image feature extraction unit: used for inputting the multi-view images of the object to be detected into a preset 2D feature extraction network to perform feature extraction and obtain 2D image features of the object to be detected;

[0056] 3D voxel feature construction unit: used to input the multi-view images of the object to be detected into the NeRF neural radiation field to perform 3D scene rendering, and map the 2D image features of the object to be detected to the rendered 3D scene to obtain the 3D voxel features of the object to be detected;

[0057] 3D target detection unit: used to input the 3D voxel features of the object to be detected into a preset 3D target detection network to perform target detection and obtain 3D target detection results;

[0058] Projection mapping unit: used to project the 3D target detection results onto a 2D image plane to obtain 2D target detection results and complete contraband detection.

[0059] Compared with the prior art, the beneficial effects of the technical solution of the present invention are:

[0060] The present invention provides a multi-view image contraband detection method based on NeRF and deep learning. First, an X-ray machine is used to collect 2D security inspection images of an object to be detected at different viewing angles to obtain multi-view images of the object to be detected; the multi-view images of the object to be detected are input into a preset 2D feature extraction network for feature extraction to obtain 2D image features of the object to be detected; the multi-view images of the object to be detected are input into a NeRF neural radiation field for 3D scene rendering, and the 2D image features of the object to be detected are mapped to the rendered 3D scene to obtain 3D voxel features of the object to be detected; the 3D voxel features of the object to be detected are input into a preset 3D target detection network for target detection to obtain 3D target detection results; the 3D target detection results are projected and mapped onto a 2D image plane to obtain 2D target detection results, thereby completing contraband detection.

[0061] The present invention has the following beneficial effects:

[0062] 1) By reconstructing the 3D voxel features of an object from 2D images acquired from multiple perspectives, more comprehensive object information can be obtained. Compared with traditional 2D image detection methods based on a single perspective, this method can effectively reduce the impact of occlusion and perspective deviation, thereby improving the accuracy of contraband identification;

[0063] 2) Traditional deep learning detection methods often perform poorly when dealing with small targets. However, through 3D reconstruction and feature extraction, this method can better highlight the spatial characteristics of small targets, thereby enhancing the detection capability of small contraband.

[0064] 3) The present invention combines multi-view image fusion and 3D reconstruction technology, making the method more adaptable and robust to external conditions such as lighting changes and environmental interference. BRIEF DESCRIPTION OF THE DRAWINGS

[0065] Figure 1 This is a flow chart of a multi-view image contraband detection method based on NeRF and deep learning provided in Example 1.

[0066] Figure 2 This is a flow chart of a multi-view image contraband detection method based on NeRF and deep learning provided in Example 2.

[0067] Figure 3 This is an architectural diagram of the method for detecting contraband using multi-view images, as provided in Example 2.

[0068] Figure 4 This is a structural diagram of the ImVoxelNet neural network model provided in Example 2.

[0069] Figure 5 This is a structural diagram of a multi-view image contraband detection system based on NeRF and deep learning provided in Example 3. DETAILED DESCRIPTION

[0070] The accompanying drawings are for illustrative purposes only and are not to be construed as limiting this patent;

[0071] In order to better illustrate this embodiment, some parts in the drawings may be omitted, enlarged, or reduced, and do not represent the actual product size;

[0072] It is understandable to those skilled in the art that some well-known structures and descriptions thereof may be omitted in the drawings.

[0073] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0074] Example 1

[0075] like Figure 1 As shown, this embodiment provides a multi-view image contraband detection method based on NeRF and deep learning, including:

[0076] S101: Using an X-ray machine to collect 2D security inspection images of the object to be inspected at different viewing angles to obtain multi-view images of the object to be inspected;

[0077] S102: Inputting the multi-view images of the object to be detected into a preset 2D feature extraction network to perform feature extraction to obtain 2D image features of the object to be detected;

[0078] S103: Inputting the multi-view images of the object to be detected into the NeRF neural radiation field for 3D scene rendering, and mapping the 2D image features of the object to be detected to the rendered 3D scene to obtain 3D voxel features of the object to be detected;

[0079] S104: Inputting the 3D voxel features of the object to be detected into a preset 3D object detection network to perform object detection and obtain a 3D object detection result;

[0080] S105: Project the 3D target detection result onto a 2D image plane to obtain a 2D target detection result, thereby completing contraband detection.

[0081] During the specific implementation process, an X-ray machine is first used to collect 2D security inspection images of the object to be detected at different viewing angles to obtain multi-view images of the object to be detected; the multi-view images of the object to be detected are input into a preset 2D feature extraction network for feature extraction to obtain 2D image features of the object to be detected; the multi-view images of the object to be detected are input into the NeRF neural radiation field for 3D scene rendering, and the 2D image features of the object to be detected are mapped to the rendered 3D scene to obtain 3D voxel features of the object to be detected; the 3D voxel features of the object to be detected are input into a preset 3D target detection network for target detection to obtain 3D target detection results; the 3D target detection results are projected and mapped onto the 2D image plane to obtain 2D target detection results, completing the detection of contraband; then, security personnel can directly view and analyze the detected contraband in the 2D image to make further processing decisions;

[0082] This method can significantly improve the efficiency and accuracy of contraband detection. By combining 3D reconstruction and target detection technology, it can not only handle occlusion problems in complex scenes, but also accurately identify contraband from multiple angles, greatly improving the efficiency of security inspections. In addition, the application of this technology also reduces the need for manual inspections and improves the degree of automation of the entire contraband detection process.

[0083] Example 2

[0084] like Figure 2 As shown, this embodiment provides a multi-view image contraband detection method based on NeRF and deep learning, including:

[0085] S101: Using an X-ray machine to collect 2D security inspection images of the object to be inspected at different viewing angles to obtain multi-view images of the object to be inspected;

[0086] S102: Inputting the multi-view images of the object to be detected into a preset 2D feature extraction network to perform feature extraction to obtain 2D image features of the object to be detected;

[0087] S103: Inputting the multi-view images of the object to be detected into the NeRF neural radiation field for 3D scene rendering, and mapping the 2D image features of the object to be detected to the rendered 3D scene to obtain 3D voxel features of the object to be detected;

[0088] S104: Inputting the 3D voxel features of the object to be detected into a preset 3D object detection network to perform object detection and obtain a 3D object detection result;

[0089] S105: Projecting the 3D target detection result onto a 2D image plane to obtain a 2D target detection result, and post-processing the 2D target detection result using a non-maximum suppression (NMS) algorithm to complete contraband detection;

[0090] In step S102, the preset 2D feature extraction network is specifically a ResNet neural network model;

[0091] The ResNet neural network model includes a ResNet50 deep residual network;

[0092] The step S103 includes:

[0093] Inputting the multi-view images of the object to be detected into the NeRF neural radiation field, establishing a 3D coordinate system containing a number of voxels, and performing 3D scene rendering;

[0094] Get the camera intrinsic parameter matrix corresponding to the i-th perspective and the external parameter matrix , where i=1,2,3,…,T, T is the total number of viewing angles;

[0095] For the center point of each voxel of the object to be detected in the 3D coordinate system ,point 2D pixel coordinates under the i-th viewing angle projection for:

[0096]

[0097]

[0098] in, is the 2D pixel coordinate of the corresponding position in the 2D security inspection image of the i-th perspective of the object to be detected; is the 2D image feature corresponding to the 2D security inspection image of the object to be detected at the i-th perspective; is the camera intrinsic parameter matrix after scaling ;

[0099] point 3D projection features at the i-th view for:

[0100]

[0101] in, Indicates that the nearest neighbor difference method is used to find the position Search characteristics;

[0102] For points projected outside the boundaries of the 2D plane or behind the 2D plane , the projection under this perspective is invalid, set its ; Projections from other perspectives are all valid projections;

[0103] Get Point The 3D projection features at all viewing angles are aggregated using the weighted average method to obtain the points 3D average projection features :

[0104]

[0105] in, is the number of viewing angles of effective projection;

[0106] Obtain the 3D average projection features of each voxel of the object to be detected in the 3D coordinate system, process the 3D average projection features of all voxels through the G-MLP shared with the NeRF neural radiation field, and then fuse them with the rendered 3D scene to obtain the 3D voxel features of the object to be detected;

[0107] In step S104, the preset 3D object detection network is specifically an ImVoxelNet neural network model;

[0108] The 3D object detection result in step S104 includes: a 3D bounding box of the object to be detected and a contraband recognition result;

[0109] The prohibited items identification result includes: no prohibited items, prohibited items such as controlled devices, prohibited items such as fireworks, prohibited items such as dangerous goods, prohibited items such as fire, and prohibited items such as batteries;

[0110] The steps of obtaining the 2D feature extraction network preset in step S102 and the 3D object detection network preset in step S104 include:

[0111] S201: Using an X-ray machine to collect 2D images of several types of contraband at different viewing angles, obtaining a multi-view contraband image dataset and performing preprocessing;

[0112] S202: Establishing a 2D feature extraction network and a 3D object detection network;

[0113] S203: Inputting the preprocessed multi-view contraband image dataset into a 2D feature extraction network for feature extraction to obtain 2D image features corresponding to each contraband;

[0114] S204: Inputting the preprocessed multi-view contraband image dataset into the NeRF neural radiation field for 3D scene rendering, and mapping the 2D image features corresponding to each contraband item to the rendered 3D scene to obtain the 3D voxel features corresponding to each contraband item;

[0115] S205: Inputting the 3D voxel features corresponding to each prohibited item into a 3D object detection network to perform object detection, and obtaining a 3D detection result corresponding to each prohibited item;

[0116] S206: Projecting the 3D detection results corresponding to each prohibited item onto a 2D image plane to obtain the 2D detection results corresponding to each prohibited item, and post-processing the 2D detection results corresponding to each prohibited item using a non-maximum suppression (NMS) algorithm;

[0117] S207: Setting a total loss function and calculating a total loss function value based on the 2D detection results corresponding to each contraband item;

[0118] S208: Determine whether the total loss function value meets the preset conditions. If not, update the parameters of the 2D feature extraction network and the 3D object detection network, and repeat steps S203 to S208. If so, obtain the trained 2D feature extraction network and the trained 3D object detection network, and save them as the preset 2D feature extraction network and the preset 3D object detection network, respectively.

[0119] The pre-processing in step S201 includes:

[0120] Performing data augmentation and normalization operations on the multi-view contraband image dataset in sequence to complete preprocessing;

[0121] The data enhancement operation includes: any one or more of flip transformation, rotation transformation, crop transformation and scaling transformation;

[0122] In step S207, the total loss function Specifically:

[0123]

[0124] in, is the classification loss function; is the total prediction error loss function; is the bounding box loss function; 、 and are the first, second and third hyperparameters respectively.

[0125] During the implementation process, a multi-view image dataset of prohibited items was first created. X-ray machines at the airport logistics center were used to obtain multi-view 2D images of each prohibited item. These images covered multiple viewpoints of the target prohibited item, ensuring that the prohibited item could be observed from all directions. Data augmentation and normalization preprocessing were then performed. Data augmentation involves scaling, cropping, and normalizing the image to meet network input requirements. This step aims to eliminate scale inconsistencies between images and ensure the accuracy of subsequent processing.

[0126] In the preprocessing process, the open source software labelme was used for labeling. There are five major categories of labels: controlled devices, fireworks products, dangerous goods, fire sources and batteries. The original sample data set is recorded as ;

[0127] The original sample image dataset is divided into several batches with a batch size of 8, and data enhancement and normalization are performed in batches. In the data enhancement process, a transformation function R is defined. T as follows:

[0128]

[0129] In the above formula, random represents a random selection function. There are four types of enhancements that can be selected: flip represents flip transformation, rotate represents rotation transformation, crop represents crop transformation, and zoom represents zoom transformation.

[0130] After each batch of data sets completes data enhancement, each sample image in the batch The random function will randomly select two types of enhancement transformations in turn. Since the transformation function does not change the angle of view, after each picture passes through the data enhancement function, the output will be 2N with a resolution of Enhanced datasets;

[0131] like Figure 3 The architecture diagram shown in the figure inputs multi-view contraband images into the ResNet network to extract high-order features, using a network with a depth of 50, namely the ResNet50 deep residual network;

[0132] ResNet50 is a deep residual network that solves the problem of vanishing or exploding gradients during deep network training by introducing a residual learning mechanism. The network contains multiple residual modules, each of which consists of multiple convolutional layers, batch normalization layers, and ReLU activation layers, which can effectively extract deep features of images.

[0133] During the pre-training process, the number of resolution channels is first set to The enhanced dataset is input into the untrained initial encoder, which is responsible for extracting features from the sample image. The whole process can be expressed as

[0134]

[0135] In the above formula, the function of Extract(.) includes obtaining the image representation of the sample data through the encoder and converting The image is converted to dimension The encoder used in the pre-training process includes a deep learning network module consisting of two parts: feature extraction and residual module. The ResNet50 network contains 49 convolution layers and 1 fully connected layer, for a total of 50 layers.

[0136] Specifically, the feature extraction module uses convolution operation to extract local features of the image. The specific operation uses a convolution kernel of For the input image The image is processed to a resolution of Tensor ;

[0137]

[0138] In the above formula, Indicates that the convolution kernel size of the convolution layer is 7 and the convolution kernel stride is 2;

[0139] Then it goes through a batch normalization layer and an activation function layer in sequence. Finally, in order to further reduce the image resolution, a convolution layer with a convolution kernel of 1 is used. The process can be expressed as follows;

[0140]

[0141]

[0142] In the above formula, BN2d represents the batch normalization layer, Represents the activation function, the ReLU function is used here;

[0143] The residual module contains two paths: the shortcut connection and the residual connection path. The shortcut connection path includes the weight layer, namely the convolution layer, batch normalization layer, and activation layer mentioned above, which is responsible for learning the mapping relationship between input and output; the shortcut path provides a way to directly connect the input to the output, helping the network learn the residual between input and output. The formula is as follows:

[0144]

[0145] in It is a combination of the above convolutional layer, batch normalization layer and activation layer;

[0146] Then, the collected multi-view 2D images are used to reconstruct the 3D voxel features of the contraband through NeRF;

[0147] Input RGB to the 2D image backbone and represent the image as ,in is a dimension identifier; inputting the multi-view image of the object to be detected into the NeRF neural radiation field, establishing a 3D coordinate system containing a number of voxels, and performing 3D scene rendering;

[0148] Get the camera intrinsic parameter matrix corresponding to the i-th perspective and the external parameter matrix , where i=1,2,3,…,T, T is the total number of viewing angles;

[0149] For the center point of each voxel of the object to be detected in the 3D coordinate system ,point 2D pixel coordinates under the i-th viewing angle projection for:

[0150]

[0151]

[0152] in, is the 2D pixel coordinate of the corresponding position in the 2D security inspection image of the i-th perspective of the object to be detected; is the 2D image feature corresponding to the 2D security inspection image of the i-th perspective of the object to be detected; considering the downsampling of the feature map, is the camera intrinsic parameter matrix after scaling ;

[0153] point 3D projection features at the i-th view for:

[0154]

[0155] in, Indicates that the nearest neighbor difference method is used to find the position Search characteristics;

[0156] For points projected outside the boundaries of the 2D plane or behind the 2D plane , the projection under this perspective is invalid, set its ; The projections at other perspectives are all valid projections; intuitively, this is equivalent to going from the origin of perspective i through pixels Shoot a ray and for all voxels on the ray, convert the image features Dispersed voxels in ;

[0157] Get Point The 3D projection features at all viewing angles are aggregated using the weighted average method to obtain the points 3D average projection features :

[0158]

[0159] in, is the number of viewing angles of effective projection;

[0160] Obtain the 3D average projection features of each voxel of the object to be detected in the 3D coordinate system, process the 3D average projection features of all voxels through the G-MLP shared with the NeRF neural radiation field, and then fuse them with the rendered 3D scene to obtain the 3D voxel features of the object to be detected;

[0161] A shared MLP (G-MLP) is used between NeRF and the above-mentioned feature matching. This embodiment sets up two branches, one for 3D detection and the other for reconstructing 3D scenes. Sharing GMLP actually shares the 3D spatial features generated by the two branches for 2D images, thereby obtaining richer 3D features. This design makes it possible for the gradient to flow from the NeRF branch back to the detection branch, thereby enriching the 3D spatial features for better target detection. G-MLP predicts a density field, which is converted into an opacity field. This opacity field is multiplied with the volume features to reduce the influence of blank space, thereby generating geometrically perceived volume features for detection.

[0162] NeRF is a volume-based rendering technique that assumes that (where o is the observation point, d is the direction vector, and t is the distance parameter) to calculate the color of each point , volume density ; The color along the light can be obtained by the integral rendering formula, which is as follows:

[0163]

[0164] in, is the cumulative transparency from the observation point to position t (the probability that the light is not blocked);

[0165] In this embodiment, a shared G-MLP is designed to connect the NeRF-rendered 3D scene and the 3D feature matching. This allows the 3D spatial features generated by the rendering to be used during training to enrich the 3D features obtained from the 2D feature mapping, thereby achieving better detection.

[0166] Then use the ImVoxelNet network to process the extracted features, such as Figure 4 The figure shows the structure of the ImVoxelNet network. ImVoxelNet is used to process voxelized 3D space to identify and locate target objects. A deep convolutional neural network is used to learn the complex features of the 3D space and perform target detection. The detection formula is as follows:

[0167]

[0168] in Represents the detection result, including the target category and 3D bounding box parameters, is the ImVoxelNet detection network, is the input voxel feature;

[0169] Then, after 3D bounding box regression, based on the network output, 3D bounding box regression is performed on each detected object to determine its exact position and size in 3D space; the bounding box regression formula is as follows:

[0170]

[0171] in, Represents the 3D bounding box parameters after regression, including the center point coordinates, length, width, height and direction angle. R is the regression model. is the original detection result output by the network;

[0172] Post-processing operations are performed on the obtained detection results, including non-maximum suppression (NMS), confidence threshold screening, and bounding box adjustment. Non-maximum suppression is performed on the above-obtained detection result B to remove duplicate detections and optimize the final detection result. The formula is as follows:

[0173]

[0174] in, is the final set of 3D bounding boxes after NMS, It is the Iou threshold, which is used to determine whether to remove overlapping bounding boxes. The Iou principle is as follows: suppose there are two bounding boxes, one is the predicted bounding box , and the other is the true bounding box , where gt represents the true sign; we need to calculate the intersection between the two and The overlapping part, and then calculate the union of the two, including and The minimum rectangular area of ​​the entire area, minus the intersection, is calculated as follows:

[0175]

[0176] Get it The result is used to measure the degree of overlap between each pair of bounding boxes; by setting an IoU threshold , NMS will remove the bounding box with the highest confidence level exceeding All other bounding boxes of , thereby reducing duplicate detections and retaining the best detection results;

[0177] The obtained 3D voxel features are then input into the 3D object detection model. This embodiment uses the same detection head as ImVoxelNet to identify and locate contraband;

[0178] The 3D object detection results are then mapped back to the 2D image plane through a projection transformation. This step utilizes the camera's intrinsic and extrinsic parameter matrices to convert the 3D detected contraband bounding box into coordinates in the 2D image.

[0179] The 3D to 2D projection transformation uses the camera's intrinsic and extrinsic matrices to map points in the 3D world coordinate system to the 2D image plane. Given a 3D point's world coordinates , its projection coordinates on the 2D image plane It can be calculated by the following formula:

[0180]

[0181] Where K is the internal parameter matrix in the form of ,in , is the focal length of the camera in x and y, and are the coordinates of the center of the image.

[0182] is the external parameter matrix, which is composed of the rotation matrix It consists of a translation vector t, which describes the transformation from the world coordinate system to the camera coordinate system;

[0183] The loss function is calculated based on the target box and the ground-truth box obtained in the above steps, and a multi-task loss function is used to supervise the training of the two models. The total loss function formula in this embodiment is:

[0184]

[0185] in, It is a classification loss function used to measure the gap between the predicted category and the true category; is the total prediction error loss function, which is defined as the sum of the absolute errors between the predicted value and the true value; It is the bounding box loss function, which is used to evaluate the overlap between the predicted box and the real box in the detection task; 、 and are the first, second, and third hyperparameters, respectively;

[0186] The model parameters are updated according to the total loss function value. After the training is completed, the trained 2D feature extraction network and 3D object detection network are obtained for subsequent security inspection and contraband detection;

[0187] The steps for detecting prohibited items during security inspection are as follows:

[0188] First, an X-ray machine is used to capture 2D security inspection images of the object to be inspected (luggage or cargo) at different viewing angles to obtain multi-view images of the object to be inspected.

[0189] Input the multi-view images of the object to be detected into the trained 2D feature extraction network for feature extraction to obtain the 2D image features of the object to be detected;

[0190] Inputting the multi-view images of the object to be detected into the NeRF neural radiation field for 3D scene rendering, and mapping the 2D image features of the object to be detected to the rendered 3D scene to obtain the 3D voxel features of the object to be detected;

[0191] Input the 3D voxel features of the object to be detected into the trained 3D object detection network to perform object detection and obtain the 3D object detection results;

[0192] The 3D object detection results are projected onto the 2D image plane to obtain the 2D object detection results. The final 2D object detection results undergo a post-processing step (NMS) to optimize and confirm the final detection results.

[0193] Security personnel can then view and analyze the detected contraband directly in the 2D image to make further handling decisions;

[0194] In addition, after detecting prohibited items, an alarm function can be set to automatically mark and highlight the location of the prohibited items on the system interface, and at the same time issue an alarm to notify security personnel for further processing;

[0195] At the same time, when the alarm is issued, a report containing detailed information on the prohibited items detection results can be further generated. The report content includes the type of prohibited items, location, detection confidence, etc., for security personnel to refer to and make decisions;

[0196] In addition, by collecting feedback from security personnel on the test results, the deep learning model can be regularly updated and optimized to continuously improve the accuracy of contraband detection and the overall performance of the system;

[0197] To meet the demand for high efficiency and high accuracy in contraband detection in large-scale logistics scenarios, this method proposes a method for constructing 3D voxel features based on multi-view 2D images. These 3D features are used to more accurately detect 2D contraband targets. This method will significantly reduce the labor cost of identifying contraband in airport logistics and eliminate many complex operations. Moreover, this method is generalizable and can be easily transferred and adapted to other application scenarios such as subway checkpoints and high-speed rail security checks.

[0198] This method reconstructs the 3D voxel features of objects from 2D images acquired from multiple perspectives, which can obtain more comprehensive object information. Compared with traditional 2D image detection methods based on a single perspective, this method can effectively reduce the impact of occlusion and perspective deviation, thereby improving the accuracy of contraband identification. Secondly, this method can better highlight the spatial characteristics of small targets, thereby enhancing the detection ability of small contraband. In addition, this method combines multi-perspective image fusion and 3D reconstruction technology, making it more adaptable and robust to external conditions such as lighting changes and environmental interference.

[0199] Example 3

[0200] like Figure 5 As shown, this embodiment provides a multi-view image contraband detection system based on NeRF and deep learning, applying the multi-view image contraband detection method based on NeRF and deep learning described in Example 1 or 2, including:

[0201] Multi-view image acquisition unit 301: used to use an X-ray machine to collect 2D security inspection images of the object to be inspected at different viewing angles, thereby obtaining multi-view images of the object to be inspected;

[0202] 2D image feature extraction unit 302: used for inputting the multi-view images of the object to be detected into a preset 2D feature extraction network to perform feature extraction and obtain 2D image features of the object to be detected;

[0203] 3D voxel feature construction unit 303: used to input the multi-view images of the object to be detected into the NeRF neural radiation field to perform 3D scene rendering, and map the 2D image features of the object to be detected to the rendered 3D scene to obtain the 3D voxel features of the object to be detected;

[0204] 3D object detection unit 304: configured to input the 3D voxel features of the object to be detected into a preset 3D object detection network to perform object detection and obtain a 3D object detection result;

[0205] Projection mapping unit 305: used to project the 3D object detection result onto a 2D image plane to obtain the 2D object detection result and complete the contraband detection.

[0206] In the specific implementation process, first, the multi-view image acquisition unit 301 uses an X-ray machine to collect 2D security inspection images of the object to be detected at different viewing angles to obtain multi-view images of the object to be detected;

[0207] The 2D image feature extraction unit 302 inputs the multi-view images of the object to be detected into a preset 2D feature extraction network to perform feature extraction and obtain the 2D image features of the object to be detected;

[0208] The 3D voxel feature construction unit 303 inputs the multi-view images of the object to be detected into the NeRF neural radiation field to perform 3D scene rendering, and maps the 2D image features of the object to be detected to the rendered 3D scene to obtain the 3D voxel features of the object to be detected;

[0209] The 3D object detection unit 304 inputs the 3D voxel features of the object to be detected into a preset 3D object detection network to perform object detection and obtain a 3D object detection result;

[0210] Finally, the projection mapping unit 305 projects the 3D object detection results onto the 2D image plane to obtain the 2D object detection results and complete the contraband detection. Then, security personnel can directly view and analyze the detected contraband in the 2D image to make further processing decisions.

[0211] This system can significantly improve the efficiency and accuracy of contraband detection. By combining 3D reconstruction and target detection technology, it can not only handle occlusion problems in complex scenes, but also accurately identify contraband from multiple angles, greatly improving the efficiency of security inspections. In addition, the application of this technology also reduces the need for manual inspections and improves the degree of automation of the entire contraband detection process.

[0212] The same or similar reference numerals correspond to the same or similar components;

[0213] The terms used in the drawings to describe positional relationships are for illustrative purposes only and should not be construed as limiting this patent;

[0214] Obviously, the above embodiments of the present invention are merely examples for the purpose of clearly illustrating the present invention, and are not intended to limit the embodiments of the present invention. Those skilled in the art will appreciate that other variations or modifications can be made based on the above description. It is not necessary and impossible to enumerate all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the claims of the present invention.

Claims

1. A multi-view image contraband detection method based on NeRF and deep learning, characterized by: The following steps are involved: S101: Using an X-ray machine to collect 2D security inspection images of the object to be inspected at different viewing angles to obtain multi-view images of the object to be inspected; S102: Inputting the multi-view images of the object to be detected into a preset 2D feature extraction network to perform feature extraction to obtain 2D image features of the object to be detected; S103: Inputting the multi-view images of the object to be detected into the NeRF neural radiation field to perform 3D scene rendering, and mapping the 2D image features of the object to be detected to the rendered 3D scene to obtain 3D voxel features of the object to be detected, including: Inputting the multi-view images of the object to be detected into the NeRF neural radiation field, establishing a 3D coordinate system containing a number of voxels, and performing 3D scene rendering; Get the camera intrinsic parameter matrix corresponding to the i-th perspective and the external parameter matrix , where i=1,2,3,…,T, T is the total number of viewing angles; For the center point of each voxel of the object to be detected in the 3D coordinate system ,point 2D pixel coordinates under the i-th viewing angle projection for: in, is the 2D pixel coordinate of the corresponding position in the 2D security inspection image of the i-th perspective of the object to be detected; is the 2D image feature corresponding to the 2D security inspection image of the object to be detected at the i-th perspective; is the camera intrinsic parameter matrix after scaling ; point 3D projection features at the i-th view for: in, Indicates that the nearest neighbor difference method is used to find the position Search characteristics; For points projected outside the boundaries of the 2D plane or behind the 2D plane , the projection under this perspective is invalid, set its ; Projections from other perspectives are all valid projections; Get Point The 3D projection features at all viewing angles are aggregated using the weighted average method to obtain the points 3D average projection features : in, is the number of viewing angles of effective projection; Obtain the 3D average projection features of each voxel of the object to be detected in the 3D coordinate system, process the 3D average projection features of all voxels through the G-MLP shared with the NeRF neural radiation field, and then fuse them with the rendered 3D scene to obtain the 3D voxel features of the object to be detected; S104: Inputting the 3D voxel features of the object to be detected into a preset 3D object detection network to perform object detection and obtain a 3D object detection result; S105: Project the 3D target detection result onto a 2D image plane to obtain a 2D target detection result, thereby completing contraband detection.

2. The multi-view image contraband detection method based on NeRF and deep learning according to claim 1 is characterized in that: In step S102, the preset 2D feature extraction network is specifically a ResNet neural network model; The ResNet neural network model includes a ResNet50 deep residual network.

3. The multi-view image contraband detection method based on NeRF and deep learning according to claim 1 is characterized in that: In step S104, the preset 3D object detection network is specifically an ImVoxelNet neural network model.

4. The multi-view image contraband detection method based on NeRF and deep learning according to claim 1, characterized in that: The 3D object detection result in step S104 includes: a 3D bounding box of the object to be detected and a contraband recognition result; The prohibited items identification result includes any one of: no prohibited items, controlled device prohibited items, fireworks prohibited items, dangerous goods prohibited items, fire prohibited items and battery prohibited items.

5. The multi-view image contraband detection method based on NeRF and deep learning according to claim 1, characterized in that: The steps of obtaining the 2D feature extraction network preset in step S102 and the 3D object detection network preset in step S104 include: S201: Using an X-ray machine to collect 2D images of several types of contraband at different viewing angles, obtaining a multi-view contraband image dataset and performing preprocessing; S202: Establishing a 2D feature extraction network and a 3D object detection network; S203: Inputting the preprocessed multi-view contraband image dataset into a 2D feature extraction network for feature extraction to obtain 2D image features corresponding to each contraband; S204: Inputting the preprocessed multi-view contraband image dataset into the NeRF neural radiation field for 3D scene rendering, and mapping the 2D image features corresponding to each contraband item to the rendered 3D scene to obtain the 3D voxel features corresponding to each contraband item; S205: Inputting the 3D voxel features corresponding to each prohibited item into a 3D object detection network to perform object detection, and obtaining a 3D detection result corresponding to each prohibited item; S206: Projecting the 3D detection results corresponding to each prohibited item onto a 2D image plane to obtain the 2D detection results corresponding to each prohibited item; S207: Setting a total loss function and calculating a total loss function value based on the 2D detection results corresponding to each contraband item; S208: Determine whether the total loss function value meets the preset conditions. If not, update the parameters of the 2D feature extraction network and the 3D object detection network, and repeat steps S203 to S208. If so, obtain the trained 2D feature extraction network and the trained 3D object detection network, and save them as the preset 2D feature extraction network and the preset 3D object detection network respectively.

6. The multi-view image contraband detection method based on NeRF and deep learning according to claim 5, characterized in that: The pre-processing in step S201 includes: Performing data augmentation and normalization operations on the multi-view contraband image dataset in sequence to complete preprocessing; The data enhancement operation includes any one or more of flip transformation, rotation transformation, crop transformation and scaling transformation.

7. The multi-view image contraband detection method based on NeRF and deep learning according to claim 5, characterized in that: In step S207, the total loss function is specifically: in, is the classification loss function; is the total prediction error loss function; is the bounding box loss function; 、 and are the first, second and third hyperparameters respectively.

8. The multi-view image contraband detection method based on NeRF and deep learning according to claim 1 or 5, characterized in that: After step S105, the following steps are further included: Post-processing the 2D target detection results using a non-maximum suppression (NMS) algorithm; After step S206, the following steps are further included: The non-maximum suppression (NMS) algorithm is used to post-process the 2D detection results corresponding to each prohibited item.

9. A multi-view image contraband detection system based on NeRF and deep learning, applying the multi-view image contraband detection method based on NeRF and deep learning as described in any one of claims 1 to 8, characterized in that: include: Multi-view image acquisition unit: used to use an X-ray machine to collect 2D security inspection images of the object to be inspected at different viewing angles, and obtain multi-view images of the object to be inspected; 2D image feature extraction unit: used for inputting the multi-view images of the object to be detected into a preset 2D feature extraction network to perform feature extraction and obtain 2D image features of the object to be detected; 3D voxel feature construction unit: used to input the multi-view images of the object to be detected into the NeRF neural radiation field to perform 3D scene rendering, and map the 2D image features of the object to be detected to the rendered 3D scene to obtain the 3D voxel features of the object to be detected; 3D target detection unit: used to input the 3D voxel features of the object to be detected into a preset 3D target detection network to perform target detection and obtain 3D target detection results; Projection mapping unit: used to project the 3D target detection results onto a 2D image plane to obtain 2D target detection results and complete contraband detection.

Images